Enabling Heartbeat in App Services for .NET and .NET Core

d3r3kk commented 6 years ago

Heartbeat feature for App Services

We would like the heartbeat feature to be extended to serve App Services instrumented with Application Insights.

Heartbeat Content

Heartbeat data already includes some default properties, and for Azure App Services we'd like to add a few more default properties:

asSiteName: The site name reported by Azure App Services.
asStamp: The website "Home Stampname" reported by Azure App Services.
asHost: The site host name as reported by Azure App Services.

These properties will help us to aid our customers discover the specific infrastructure their application is running under.

Implementation Options

Thus far I've identified two options for implementing this change. In either case, the process will be to create a heartbeat ITelemetryModule that will get added to the AppInsights pipeline, acquire the default field values, and add them to the heartbeat upon initialization.

Option 1:

Create an ITelemetryModule in the Microsoft.ApplicationInsights.Web SDK (this repo)
Create a similar ITelemetryModule in the Microsoft.ApplicationInsights.AspNetCore SDK

Option 2:

Create an ITelemetryModule in the Microsoft.ApplicationInsights.Web SDK (this repo)
Alter the Microsoft.ApplicationInsights.Web project to support .NET Standard 1.6
Add a dependency on the Microsoft.ApplicationInsights.Web project in the Microsoft.ApplicationInsights.AspNetCore SDK

NOTE: I am hoping to have this feature in for 2.6.0-beta3, so taking the approach of implementing Option 1, and then Option 2 is on the table here.

Adding Heartbeats for App Services Automatically

ApplicationInsights.config.install.xdt would be employed to add the created ITelemetryModule to the ApplicationInsights.config file, in the case of .NET Full Framework applications. This is already available in the Microsoft.ApplicationInsights.Web nuget configuration and would be trivial.

For .NET Core applications, however, we would have to alter code in Microsoft.ApplicationInsights.AspNetCore to follow the pattern there today regarding AdaptiveSamplingTelemetryProcessor. That is, we would by default add the heartbeat ITelemetryModule but add a member to ApplicationInsightsServiceOptions that would allow customers to turn it off. (Until we have a way of configuring the application via config, I believe this is the only facility we have available).

SergeyKanzhelev commented 6 years ago

Why do you think this feature belongs to the Web package? This feature belongs to Windows package that is already on it's way to be shared with asp.net core.

What is your plan of synchronizing identity information set by azure web site telemetry initializer on every telemetry item and this telemetry module?

d3r3kk commented 6 years ago

Why do you think this feature belongs in the Web package?

I think this feature belongs in the Web package because it is for Web apps! Perhaps my understanding of the various SDK packages we own is missing some detail, and relying on the naming convention isn't quite enough.

This feature belongs to Windows package that is already on it's way to be shared with asp.net core.

I see what your intent is, ok, that works. If we are using the WindowsServer SDK package to propagate Heartbeats then there is no issue here at all and I can get to work injecting it straight away. Renaming the current Heartbeat ITelemetryModule in WindowsServer SDK from the very specific AzureInstanceMetadataTelemetryModule to something more like HeartbeatProviderTelemetryModule might be prudent then!

What is your plan of synchronizing identity information set by azure web site telemetry initializer on every telemetry item and this telemetry module?

Interesting, I wasn't aware of the telemetry initializer adding this data, thanks for pointing it out. We already capture 2/3 of the fields in our AzureWebAppRoleEnvironmentTelemetryInitializer here. This is going to be redundant data when both modules are configured!

I'm not sure what you mean by synchronizing these two telemetry modules. Can the values change during runtime? I know it is possible that they could change, but it is a scenario we are going to need to support (how likely is a website name to change during runtime?).

d3r3kk commented 6 years ago

I could create an ITelemetryProcessor in AspNetCore that looks for the properties on HeartbeatState metrics, and if the Context information is different than the content of the HeartbeatState instance, reset the value in the HeartbeatProvider. I could use this same processor to remove redundancy in HeartbeatState telemetry items (remove the context). Not sure this is necessary for the low-load Heartbeat metric item count though.

SergeyKanzhelev commented 6 years ago

Web package is meta package that enables ASP.NET apps monitoring AND brings many more features like dependency collection, performance counters monitoring and windows package for the common features. So packaging some common code into Web package is not aligned with it's purpose.

Renaming the current Heartbeat ITelemetryModule in WindowsServer SDK from the very specific AzureInstanceMetadataTelemetryModule to something more like HeartbeatProviderTelemetryModule might be prudent then!

You can have two separate ones. So it will be easier to comment out one or another if customer knows where an app will be deployed to. Also it will make modules logic-less and coinfig-free. They will simply detect the environment and read corresponding settings.

SergeyKanzhelev commented 6 years ago

I'm not sure what you mean by synchronizing these two telemetry modules. Can the values change during runtime? I know it is possible that they could change, but it is a scenario we are going to need to support (how likely is a website name to change during runtime?).

Regarding the change - yes, they can. See corresponding issue tracked in this repository. It happens with the slots swap. Regarding synchronizing of values - two things needs validation:

values match between telemetry items context and heartbeat event. So perhaps let initializer populate heartbeat properties
there is a way to join telemetry item with the corresponding heartbeat event. This item is important for app to infra scenarios

d3r3kk commented 6 years ago

Thanks for getting back to me!

You can have two separate ones.

Agreed, this is the best way I think.

Regarding the change - yes, they can. See corresponding issue tracked in this repository. It happens with the slots swap.

Ah yes, the slot swap!

Ok, I'll have to put logic into the telemetry initializer to identify "HeartbeatState" metrics, test the value of the various properties, and update the heartbeat provider itself if I detect a difference.

I will have the AppServicesHeartbeatProviderTelemetryModule add the properties to the heartbeat if the environment is correct, and I'll update the AzureWebAppRoleEnvironmentTelemetryInitializer to verify the values and update them whenever necessary.

there is a way to join telemetry item with the corresponding heartbeat event.

I don't quite follow what you mean here, is there a simple way to join properties from one telemetry item to another?

SergeyKanzhelev commented 6 years ago

For reference - this is swap issue I was talking about: https://github.com/Microsoft/ApplicationInsights-dotnet-server/issues/648

d3r3kk commented 6 years ago

Cool. Thanks for linking the issue. (I understood the slot swap stuff, but am still not sure what was meant by 'join telemetry item with the corresponding heartbeat event').

d3r3kk commented 6 years ago

All App Services covered by .NET and .NET Core SDKs are now in place.

microsoft / ApplicationInsights-dotnet-server