search

open-telemetry / opentelemetry-dotnet

The OpenTelemetry .NET Client
https://opentelemetry.io
Apache License 2.0
3.12k stars 745 forks source link

Metrics not being Sent #5324

Closed cliedeman closed 5 months ago

cliedeman commented 5 months ago

Bug Report

List of all OpenTelemetry NuGet packages and version that you are using (e.g. OpenTelemetry 1.0.2):

Package versions

    <ItemGroup Label="OpenTelemetry">
        <PackageVersion Include="OpenTelemetry" Version="1.6.0"/>
        <PackageVersion Include="OpenTelemetry.Api" Version="1.6.0"/>
        <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.6.0"/>
        <PackageVersion Include="OpenTelemetry.SemanticConventions" Version="1.0.0-rc9.9"/>
        <!-- Contrib -->
        <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.5.1-beta.1"/>
        <PackageVersion Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.5.1-beta.1"/>
        <PackageVersion Include="OpenTelemetry.Instrumentation.EntityFrameworkCore" Version="1.0.0-beta.7"/>
        <PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="1.5.1"/>
        <PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.5.1-beta.1"/>
        <!-- Exporters -->
        <PackageVersion Include="OpenTelemetry.Exporter.Jaeger" Version="1.5.1"/>
        <PackageVersion Include="Azure.Monitor.OpenTelemetry.Exporter" Version="1.1.0"/>
        <PackageVersion Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.6.0-rc.1"/>
    </ItemGroup>

I have tried various combinations of the telemetry libraries but no combination works. My best guess is that it is related to the dotnet 8 upgrade but thats just a guess

net8.0

Appinsights telemetry not being sent

Log Output


If you are seeing this message, it means that the OpenTelemetry SDK has successfully created the log file used to write self-diagnostic logs. This file will be appended with logs as they appear. If you do not see any logs following this line, it means no logs of the configured LogLevel is occurring. You may change the LogLevel to show lower log levels, so that logs of lower severities will be shown.
2024-02-07T08:00:58.1637346Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:01:08.1884118Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:01:13.1972416Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:01:18.2061763Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:01:33.2661762Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:01:38.2789219Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:01:43.2869439Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:01:48.2892755Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:01:53.3022665Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:02:03.3285186Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:02:08.3352276Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:02:13.3565442Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:02:18.3532808Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:02:23.3662601Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:02:33.3772502Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:02:38.3893408Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
2024-02-07T08:02:53.4198957Z:{0} - {1}{FailedToTransmit}{Error code is 206: Telemetry is dropped. Instrumentation Key: [REDACTED], Configured Endpoint: https://northeurope-3.in.applicationinsights.azure.com/, Actual Endpoint: northeurope-3.in.applicationinsights.azure.com}
...
cliedeman commented 5 months ago

I figured out the issue. My metrics are being sampled along with my traces and the metrics are being drowned out.

I have sampling set to 12.5%.

For now I have disable tracing and will switch to sdk sampling as a workaround or possibly split the appinsights resource for tracing and metrics.

https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling-classic-api#brief-summary

Some telemetry types, including performance metrics and custom metrics, are always kept regardless of whether sampling is enabled or not.

cijothomas commented 5 months ago

Please use AzureMonitor exporter repo for issues specific to AzureMonitor exporter.

Also - there is nothing called "sampling" for Metrics.

rajkumar-rangaraj commented 5 months ago

Please open your issue here for Azure Monitor Exporter - https://github.com/Azure/azure-sdk-for-net/issues