Durable Tracing V2 issue when using Application Insights with Entra Authentication

[MahrRah]

Description

We have encountered a problem regarding the combination of Durable Tracing V2 and using ManagedIdentity with Application Insights.

We found that when disabling local authentication in our Application Insights (see Microsoft Entra authentication for Application Insights) and relying on the ManagedIdentity of the Azure Function Host to authenticate to Application Insights, we start losing telemetry data related to the V2 distributed tracing.

Looking into the code, it seems like the Durable Function Extension configures its own TelemetryClient which does not take the ManagedIdentity of the Function Host into account.

Expected behavior

When using Managed Identity there should be a way to set the Managed Identity credentials of the Azure Function Hosts when this TelemetryClient is created, similar to how this is handled in the azure-webjobs-sdk with the APPLICATIONINSIGHTS_AUTHENTICATION_STRING (see code here)

Actual behavior

This TelemetryClient only sets the APPINSIGHTS_INSTRUMENTATIONKEY and the APPLICATIONINSIGHTS_CONNECTION_STRING, but does not have a way to set credentials when creating the TelemetryConfiguration (see full code reference here or simplified snippet below)

Relevant source code snippets

private TelemetryConfiguration SetupTelemetryConfiguration()
        {
            TelemetryConfiguration config = TelemetryConfiguration.CreateDefault();
            if (this.OnSend != null)
            {
                config.TelemetryChannel = new NoOpTelemetryChannel { OnSend = this.OnSend };
            }

            string resolvedInstrumentationKey = this.nameResolver.Resolve("APPINSIGHTS_INSTRUMENTATIONKEY");
            string resolvedConnectionString = this.nameResolver.Resolve("APPLICATIONINSIGHTS_CONNECTION_STRING");

            bool instrumentationKeyProvided = !string.IsNullOrEmpty(resolvedInstrumentationKey);
            bool connectionStringProvided = !string.IsNullOrEmpty(resolvedConnectionString);

            if (instrumentationKeyProvided && connectionStringProvided)
            {
                this.endToEndTraceHelper.ExtensionWarningEvent(...);
            }

            if (!instrumentationKeyProvided && !connectionStringProvided)
            {
                this.endToEndTraceHelper.ExtensionWarningEvent(...);
            }

            if (instrumentationKeyProvided)
            {
                this.endToEndTraceHelper.ExtensionInformationalEvent(...);

#pragma warning disable CS0618 // Type or member is obsolete
                config.InstrumentationKey = resolvedInstrumentationKey;
#pragma warning restore CS0618 // Type or member is obsolete
            }

            if (connectionStringProvided)
            {
                this.endToEndTraceHelper.ExtensionInformationalEvent(...);

                config.ConnectionString = resolvedConnectionString;
            }

            return config;
        }

App Details

• Durable Functions extension version (e.g. v1.8.3): >=2.4.1
• Azure Functions runtime version (1.0 or 2.0): 2.0
• Programming language used: Python

tagging: @ransonjb

[cgillum]

FYI @AnatoliB and @lilyjma, this might be an important issue to prioritize as it is security compliance related, and we may start seeing more asks for this.

[bachuv]

I discussed this with @jviau offline and he recommended making the following fix:

• Rewrite ITelemetryActivator as an ITelemetryModule so we get TelemetryConfiguration without having to instantiate it ourselves. With this approach, TelemetryConfiguration will have the necessary auth information.

Jacob, feel free to add any details that I missed.

[jviau]

@bachuv, yes it is a very straight forward change of removing our own ITelemetryActivator contract and use ITelemetryModule instead. Register our implementation as a singleton:

Implementation of our module (using existing TelemetryActivator implementation):

/// <inheritdoc/>
public void Initialize(TelemetryConfiguration configuration)
{
    if (this.options.Tracing.Version == Options.DurableDistributedTracingVersion.V2)
    {
        DurableTelemetryModule module = new DurableTelemetryModule();
        module.Initialize(configuration);
        this.telemetryModule = module;
    }
    else
    {
        this.SetUpV1DistributedTracing();
        if (CorrelationSettings.Current.EnableDistributedTracing)
        {
            this.SetUpTelemetryClient(configuration);
            if (CorrelationSettings.Current.EnableDistributedTracing)
            {
                this.SetUpTelemetryCallbacks();
            }
        }
    }
}

Registration:

services.AddSingleton<ITelemetryModule, TelemetryActivator>();