[BUG] flooding of "POST /v2.1/track" dependencies

[basilfx]

Library name and version

Azure.Monitor.OpenTelemetry.AspNetCore 1.2.0

Describe the bug

Using the Azure Monitor OpenTelemetry exporter results in flooded calls of POST /v2.1/track in Application Insights.

I believe this is very similar to https://github.com/Azure/azure-sdk-for-java/issues/41040, which got approved as a bug and solved.

<PackageReference Include="Azure.Monitor.OpenTelemetry.AspNetCore" Version="1.2.0" />
<PackageReference Include="OpenTelemetry" Version="1.9.0" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.0" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.9.0" />

Expected behavior

No dependencies should be logged that originate from the exporter itself, by default.

Actual behavior

Dependencies originating from the exporter are tracked.

Reproduction Steps

In a regular .NET 8 console/worker application:

public static Task Main(string[] args)
{
    return CreateHostBuilder(args).Build().RunAsync();
}

private static IHostBuilder CreateHostBuilder(string[] args)
{
    var builder = Host.CreateDefaultBuilder(args);

    return builder.ConfigureServices(ConfigureServices);
}
internal static void ConfigureServices(HostBuilderContext hostBuilder, IServiceCollection services)
{
    services
            .AddOpenTelemetry()
            .UseAzureMonitor();
}

Environment

.NET SDK: Version: 8.0.303 Commit: 29ab8e3268 Workload version: 8.0.300-manifests.c915c39d MSBuild version: 17.10.4+10fbfbf2e

Runtime Environment: OS Name: Mac OS X OS Version: 14.6 OS Platform: Darwin RID: osx-x64 Base Path: /usr/local/share/dotnet/sdk/8.0.303/

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @rajkumar-rangaraj @reyang @TimothyMothra @vishweshbankwar.

[TimothyMothra]

Hi @basilfx, Thanks for reporting this. This should absolutely never happen.

For context, we suppress exactly these messages in the Exporter, so I'm surprised to see you encounter this. https://github.com/Azure/azure-sdk-for-net/blob/496ca8d3e833e9144017bd1b794560281cba7cc7/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/AzureMonitorTraceExporter.cs#L36-L39

---

I tried the repro code you shared and I'm not seeing this same behavior.

Next Steps

1. Can you please collect some more information? Go to your logs, and query the dependencies table. I'm interested in the sdkVersion value for these "applicationinsights.azure.com" telemetry items.

2. Can you share a minimal repro? Either attach a zip to this issue, or create a new repo on github and share the link. If you can provide a sample app that reproduces this, I can debug and investigate.

[cijothomas]

Also can you check if the classic ApplicationInsights SDK/LiveMetrics is used along with OTel distro.

[basilfx]

@TimothyMothra

Thank you for your answer. I tried to isolate this problem, but I am not able to replicate it. I don't know what triggers it, so my attempt to create a minimal example failed (made a simple worker with a background service that emits metrics using System.Diagnostics.Metrics, but that did not do it).

I did some digging, and I might have found some clues. First of all, the SDK Version is dotnet8.0.7:otel1.9.0:ext1.3.0-d.

If I cross-reference the dependency (that's what we see) with customMetrics, then I see the following (note that we use sampling):

These metrics are not to be found by me, and not enabled explicitly. The metrics are from the same cloud role instance (container app, same replica).

@cijothomas I checked all dependencies, and there is no Application Insights assembly included

[TimothyMothra]

Hi @basilfx The SDK version tells me that you're on the latest versions of .NET, OTel, and our Azure Monitor exporter. Let me discuss with my team to see what else we can investigate.

Other questions

• Is it at all possible that OpenTelemetry is being initialized more than once within your process?
• Where is your application hosted? (ie: on-prem, Azure Functions, etc)

Workaround

In the meantime, you can filter the dependencies to our service so your logs are useful.

builder.Services.Configure<HttpClientTraceInstrumentationOptions>(options =>
{
    options.FilterHttpRequestMessage = (httpRequestMessage) =>
    {

        if (httpRequestMessage.RequestUri?.Host.Equals("westeurope-3.in.applicationinsights.azure.com") ?? false)
        {
            return false;
        }

        return true;
    };
});

[TimothyMothra]

It's been a few weeks without any response, so I'm closing this issue. If you have any follow up questions, please feel free to reopen the issue or open a new issue to continue the conversation.