Throws circular dependency exception when used together with Microsoft.ApplicationInsights.Kubernetes

[pihalve]

I'm creating a dockerized ASP.NET Core solution to be hosted in Kubernetes, and want to use ApplicationInsights. So I added 'Microsoft.ApplicationInsights.AspNetCore 2.6.1' and 'Microsoft.ApplicationInsights.Kubernetes 1.0.2'.

Additionally, since I want also to log my own stuff to ApplicationInsights using ILogger, I added 'Microsoft.Extensions.Logging.ApplicationsInsights 2.9.0-beta3'. But when I deploy the solution to Kubernetes the pod crashes with this exception:

Unhandled Exception: System.InvalidOperationException: A circular dependency was detected for the service of type 'Microsoft.Extensions.Logging.ILoggerFactory'.
Microsoft.Extensions.Logging.ILogger<Microsoft.AspNetCore.Hosting.Internal.WebHost>(Microsoft.Extensions.Logging.Logger<Microsoft.AspNetCore.Hosting.Internal.WebHost>) -> Microsoft.Extensions.Logging.ILoggerFactory(Microsoft.Extensions.Logging.LoggerFactory) -> System.Collections.Generic.IEnumerable<Microsoft.Extensions.Logging.ILoggerProvider> -> Microsoft.Extensions.Logging.ILoggerProvider(Microsoft.Extensions.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider) -> Microsoft.Extensions.Options.IOptions<Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration>(Microsoft.Extensions.DependencyInjection.TelemetryConfigurationOptions) -> System.Collections.Generic.IEnumerable<Microsoft.Extensions.Options.IConfigureOptions<Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration>> -> Microsoft.Extensions.Options.IOptionsChangeTokenSource<Microsoft.Extensions.Logging.Console.ConsoleLoggerOptions>(Microsoft.Extensions.Logging.Configuration.LoggerProviderOptionsChangeTokenSource<Microsoft.Extensions.Logging.Console.ConsoleLoggerOptions, Microsoft.Extensions.Logging.Console.ConsoleLoggerProvider>) -> Microsoft.Extensions.Options.IConfigureOptions<Microsoft.Extensions.Logging.Console.ConsoleLoggerOptions>(Microsoft.Extensions.Logging.Console.ConsoleLoggerOptionsSetup) -> Microsoft.Extensions.Options.IConfigureOptions<Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration>(Microsoft.Extensions.DependencyInjection.TelemetryConfigurationOptionsSetup) -> System.Collections.Generic.IEnumerable<Microsoft.ApplicationInsights.Extensibility.ITelemetryInitializer> -> Microsoft.ApplicationInsights.Extensibility.ITelemetryInitializer(Microsoft.ApplicationInsights.Kubernetes.KubernetesTelemetryInitializer) -> Microsoft.ApplicationInsights.Kubernetes.IK8sEnvironmentFactory(Microsoft.ApplicationInsights.Kubernetes.K8sEnvironmentFactory) -> Microsoft.Extensions.Options.IConfigureOptions<Microsoft.ApplicationInsights.AspNetCore.Extensions.ApplicationInsightsServiceOptions>(Microsoft.AspNetCore.Hosting.DefaultApplicationInsightsServiceConfigureOptions) -> Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientFactory -> Microsoft.Extensions.Logging.ILogger<Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientFactory>(Microsoft.Extensions.Logging.Logger<Microsoft.ApplicationInsights.Kubernetes.KubeHttpClientFactory>) -> Microsoft.Extensions.Logging.ILoggerFactory
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteChain.CheckCircularDependency(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.CreateCallSite(Type serviceType, CallSiteChain callSiteChain)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.CreateArgumentCallSites(Type serviceType, Type implementationType, CallSiteChain callSiteChain, ParameterInfo[] parameters, Boolean throwIfCallSiteNotFound)

In program.cs I added this after UseStartup:

.UseApplicationInsights()
.ConfigureLogging(logging =>
{
    logging.AddApplicationInsights();

    // Optional: Apply filters to configure LogLevel Trace or above is sent to
    // ApplicationInsights for all categories.
    logging.AddFilter<ApplicationInsightsLoggerProvider>("", LogLevel.Debug);

    // Additional filtering For category starting in "Microsoft",
    // only Warning or above will be sent to Application Insights.
    logging.AddFilter<ApplicationInsightsLoggerProvider>("Microsoft", LogLevel.Warning);
});

In startup.cs I added this in ConfigureServices:

services.AddApplicationInsightsKubernetesEnricher();

So it seems there's a bit of a compatibility issue. Or perhaps I'm doing something wrong.

[cijothomas]

@pihalve Can you remove the services.AddApplicationInsightsKubernetesEnricher(); and also remove 'Microsoft.ApplicationInsights.Kubernetes 1.0.2' and see if this issue occurs. This will help us narrow down the root cause.

Using Microsoft.Extensions.Logging.ApplicationInsights and Microsoft.ApplicationInsights.AspNetCore together shouldn't cause this issue and is documented here: https://github.com/Microsoft/ApplicationInsights-dotnet-logging/blob/develop/src/ILogger/Readme.md

[pihalve]

Removing services.AddApplicationInsightsKubernetesEnricher(); makes it work fine. In fact, just using Microsoft.ApplicationInsights.AspNetCore and Microsoft.ApplicationInsights.Kubernetes works fine, and just using Microsoft.ApplicationInsights.AspNetCore and Microsoft.Extensions.Logging.ApplicationsInsights works fine. But the combination of all three together causes the issue.

[cijothomas]

Ok thanks for confirming. We need to get some help from Microsoft.ApplicationInsights.Kubernetes owners.

[cijothomas]

let me find the best person to help

[cijothomas]

@karolz-ms Is this something which you might know?

[karolz-ms]

@xiaomi7732 is the expert here. @brahmnes did some investigation and figured out that when Ilogger is created, it goes to the AI logging provider, which then goes to the telemetry client, and then the telemetry configuration, and then the list of telemetry initializer, and then the kube telemetry initializer, and then our k8senvironmentfactory, and then our kube http client, which uses and requires an ILogger on construction 😕

So we need to break this circle somehow

[cijothomas]

Thanks for sharing this info Karol. Will start an internal mail with folks.

[joergjo]

This is also tracked here: https://github.com/Microsoft/ApplicationInsights-Kubernetes/issues/173

[joergjo]

Hm... version of 2.9.1 of Microsoft.Extensions.Logging.ApplicationInsights is now available on NuGet and the error is gone. What makes me wonder is that none of the changes in 2.9.1 or the base SDK seem to have anything to do with this issue at first glance .

Please ignore. The error is still here, I had my app accidentally configured so that App Insights was not being used.

[ferantivero]

@karolz-ms just in case, it's been already tackled from https://github.com/microsoft/ApplicationInsights-Kubernetes/pull/180

cc/ @cijothomas

[karolz-ms]

@ferantivero cool, thx