SF Context Variables not applied to dependent service

[andrewdmoreno]

I am utilizing the Microsoft.ApplicationInsights.ServiceFabric.Native (2.1.1-beta1) package in my Service Fabric application mostly to great success. I have one particular service, however, that does not seem to be populating the SF context properties correctly.

The situation is essentially when Service A is making a remoting (V2) call to Service B which is a StatefulService.

On the Service A side, a dependency telemetry is tracked as expected with all of the service fabric context properties populated correctly.

On the Service B side a request is indeed tracked and is correlated to the same operation as the dependency in Service A, however, all of the additional context properties for Service B are not being populated. In fact, the only one populated is ServiceFabric.NodeName. Additionally, the cloud role name does not display the name of the service, but instead that of the code package. The cloud role instance also displays the hostname of the node rather than the service instance.

In terms of config and initialization, I am setting the call context as instructed:

        protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
        {
            FabricTelemetryInitializerExtension.SetServiceCallContext(Context);
            return this.CreateServiceRemotingReplicaListeners();
        }

The service is using V2 remoting:

[assembly: FabricTransportServiceRemotingProvider(RemotingListener = RemotingListener.V2Listener, RemotingClient = RemotingClient.V2Client)]

And my application insights config looks as such:

<?xml version="1.0" encoding="utf-8"?>
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
  <TelemetryInitializers>
    <Add Type="Microsoft.ApplicationInsights.DependencyCollector.HttpDependenciesParsingTelemetryInitializer, Microsoft.AI.DependencyCollector"/>
    <Add Type="Microsoft.ApplicationInsights.ServiceFabric.FabricTelemetryInitializer, Microsoft.AI.ServiceFabric"/>
    <Add Type="Microsoft.ApplicationInsights.ServiceFabric.CodePackageVersionTelemetryInitializer, Microsoft.AI.ServiceFabric.Native"/>
  </TelemetryInitializers>
  <TelemetryModules>
    <Add Type="Microsoft.ApplicationInsights.DependencyCollector.DependencyTrackingTelemetryModule, Microsoft.AI.DependencyCollector">
      <ExcludeComponentCorrelationHttpHeadersOnDomains>
        <!-- 
        Requests to the following hostnames will not be modified by adding correlation headers. 
        This is only applicable if Profiler is installed via either StatusMonitor or Azure Extension.
        Add entries here to exclude additional hostnames.
        NOTE: this configuration will be lost upon NuGet upgrade.
        -->
        <Add>core.windows.net</Add>
        <Add>core.chinacloudapi.cn</Add>
        <Add>core.cloudapi.de</Add>
        <Add>core.usgovcloudapi.net</Add>
      </ExcludeComponentCorrelationHttpHeadersOnDomains>
    </Add>
    <Add Type="Microsoft.ApplicationInsights.ServiceFabric.Module.ServiceRemotingRequestTrackingTelemetryModule, Microsoft.AI.ServiceFabric.Native"/>
    <Add Type="Microsoft.ApplicationInsights.ServiceFabric.Module.ServiceRemotingDependencyTrackingTelemetryModule, Microsoft.AI.ServiceFabric.Native"/>
  </TelemetryModules>
</ApplicationInsights>

Any thoughts as to why the context fields are not being populated as expected? Am I doing anything wrong?

[brahmnes]

Hi Andrew,

Unfortunately the SetServiceCallContext method is not as reliable as it should be. Basically it relies on the CallContext to track the ServiceContext object, and this context can be absent or different depending on the execution flow. For example, if it enters native code and back to managed code, the context can be lost. In absence of the context, FabricTelemetryInitializer attempts to fall back onto the environment variables for those values, which is probably what you were seeing.

There isn't a good solution to this problem yet, one workaround that should be quite reliable is setting the context directly in the initializer associated with the active TelemetryConfiguration at the very beginning. It's not elegant but it should help.

Here is an example:

    public StatelessBackendService(StatelessServiceContext context)
        : base(context)
    {
        var telemetryConfig = TelemetryConfiguration.Active;
        // Replace the fabric telemetry initializer, if there is one, with one that has the rich context
        for (int i = 0; i < telemetryConfig.TelemetryInitializers.Count; i++)
        {
            if (telemetryConfig.TelemetryInitializers[i] is FabricTelemetryInitializer)
            {
                telemetryConfig.TelemetryInitializers[i] = FabricTelemetryInitializerExtension.CreateFabricTelemetryInitializer(context);
                break;
            }
        }
    }

[andrewdmoreno]

@brahmnes Thanks I will give that a try. Should that be in place of the line currently here or in addition to?

        protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
        {
            FabricTelemetryInitializerExtension.SetServiceCallContext(Context);
            return this.CreateServiceRemotingReplicaListeners();
        }

[brahmnes]

In your context, it is service B that is getting the right context, so you want do that only in the constructor for service B.

Having said that, I forgot to mention that this workaround is ok only if the process is not shared by multiple service instances. The reason is TelemetryConfiguration.Active is a singleton.

[andrewdmoreno]

Did you mean Service A is getting the right context? Service A is the one that has it's values populated correctly.

[brahmnes]

Sorry I mistyped my message above. I meant service B that is NOT getting the right context. For the service that is not getting the right context, its constructor needs to have its telemetry initializer attached with the context explicitly.

[brahmnes]

Hi @andrewdmoreno I am closing this issue, please reopen if the workarond doesn't work for you.

[andrewdmoreno]

@brahmnes - Sorry for the late reply. Yes, this work around does indeed work for the meantime.