search

DataDog / dd-trace-dotnet

.NET Client Library for Datadog APM
https://docs.datadoghq.com/tracing/
Apache License 2.0
413 stars 132 forks source link

Unknown service created when using Hangfire instrumentation #5650

Open Hawxy opened 1 month ago

Hawxy commented 1 month ago

Describe the bug We use the Hangfire OTEL instrumentation (located here) to add root spans to the running jobs.

OTEL is configured like so to add the .NET Activity instrumentation but the export is delegated to DD Trace's automated OTEL support:

using var tracer = Sdk
    .CreateTracerProviderBuilder()
    .AddHangfireInstrumentation()
    .Build();

However this results in the hangfire root spans showing up under their own service "unknown_service:dotnet" in the DD console. All other spans from the service are correctly aggregated.

image

The span metadata does have the correct service name: image

This application is running in App Services with the tracer baked into the image:

COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
COPY --from=datadog/dd-lib-dotnet-init /datadog-init/monitoring-home/ /dd_tracer/dotnet/
ENV DD_SERVICE=(hidden)-integration-api
ENV DD_LOGS_INJECTION=true
ENV DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS='Serilog'
ENV DD_TRACE_OTEL_ENABLED=true
ENV DD_DATA_STREAMS_ENABLED=true
ENV DD_TRACE_SPAN_ATTRIBUTE_SCHEMA='v1'

Runtime environment:

bouwkast commented 1 month ago

Thanks @Hawxy, I was able to replicate this and will work on getting a better fix implemented for the Tracer. There does seem to be a potential workaround though by setting the environment variable OTEL_SERVICE_NAME to the same value as DD_SERVICE.

Note that we are currently working on supporting remapping OpenTelemetry environment variables to their Datadog equivalent ones: https://github.com/DataDog/dd-trace-dotnet/pull/5661 I don't think this would have an impact here though, but just wanted to raise it.

But I'll work on getting a fix to address it without needing to do that