Opentelemetry instrumentation separates http and db spans from the trace

[InterstellarStella]

Package

Sentry.Extensions.Logging

.NET Flavor

.NET

.NET Version

8.0.8

OS

Windows

SDK Version

4.12.0

Self-Hosted Sentry Version

No response

Steps to Reproduce

using Microsoft.ApplicationInsights.DependencyCollector;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System.Text.Json;
using Microsoft.Azure.Functions.Worker.OpenTelemetry;
using OpenTelemetry.Trace;
using Sentry.Azure.Functions.Worker;
using Sentry.OpenTelemetry;
using Sentry.Profiling;

DependencyTrackingTelemetryModule depModule = new DependencyTrackingTelemetryModule();
depModule.Initialize(TelemetryConfiguration.Active);

var host = new HostBuilder()
    .ConfigureServices(s =>
    {
        s.AddHttpClient();
        s.Configure<JsonSerializerOptions>(JsonOptions.ConfigureDefaultSerializerOptions);
        s.AddOpenTelemetry()
            .UseFunctionsWorkerDefaults()
            .WithTracing(tracerProviderBuilder =>
                    tracerProviderBuilder
                        .AddAspNetCoreInstrumentation()
                        .AddHttpClientInstrumentation()
                        .AddSentry()
            );
    })
    .ConfigureFunctionsWorkerDefaults((host, builder) =>
    {
        builder.UseSentry(host, options =>
        {
            options.Dsn = "DSN_HERE";
            options.Debug = true;
            options.AutoSessionTracking = true;

            string? tracesSampleRateEnvVar = Environment.GetEnvironmentVariable("SENTRY_TRACES_SAMPLE_RATE");
            options.TracesSampleRate = tracesSampleRateEnvVar != null ? double.Parse(tracesSampleRateEnvVar) : 1.0;

            string? profilesSampleRateEnvVar = Environment.GetEnvironmentVariable("SENTRY_PROFILES_SAMPLE_RATE");
            options.ProfilesSampleRate = profilesSampleRateEnvVar != null ? double.Parse(profilesSampleRateEnvVar) : 1.0;
            options.AddIntegration(new ProfilingIntegration(
                TimeSpan.FromMilliseconds(500)
            ));
            options.Environment = Environment.GetEnvironmentVariable("ENVIRONMENT_NAME") ?? "Unknown";
            options.CaptureFailedRequests = true;
            options.FailedRequestStatusCodes.Add((400, 499));
            options.UseOpenTelemetry();
        });
    }).Build();
host.Run();

Expected Result

Traces show all expected spans, including function, db, and http.

Actual Result

The trace only shows the function. Some, if not all, of the db and http spans can be seen with "Missing Trace Root".

Screenshots and links can be found in the shadow ticket.

┆Issue is synchronized with this Jira Improvement by Unito

[jamescrosswell]

Hey @InterstellarStella,

It's hard to say what's going on based on the information in this issue. I see:

Screenshots and links can be found in the shadow ticket.

I don't have access to JIRA though. Is there any way to DM the additional information to me on Discord maybe? The screenshots would definitely be handy.

EDIT: @bruno-garcia passed these along - all good.

[jamescrosswell]

From what I understand, the customer's technical stack here is:

• Azure Functions
• Entity Framework
• Postgres

The OpenTelemetry trace builder is being initialised with:

            .WithTracing(tracerProviderBuilder =>
                    tracerProviderBuilder
                        .AddAspNetCoreInstrumentation()
                        .AddHttpClientInstrumentation()
                        .AddSentry()
            );

The AddAspNetCoreInstrumentation and AddHttpClientInstrumentation extension methods will take care of adding sources (i.e. listeners) for Microsoft ASP.NET Core and HttpClient telemetry instrumentation.

It looks like the customer might be relying on our DiagnosticSource integration for EntityFramework, however... and I could see issues with parenting traces if some of the spans are created by OpenTelemetry and others are created by Sentry... particularly if there are also OpenTelemetry spans created for the same (e.g. via an NpgSql OpenTelemetry integration) due to the way our SpanProcessor works.

To test this theory, could you ask the customer to try disabling the Sentry DiagnosticSource integration as a first step:

            options.DisableDiagnosticSourceIntegration();

That would hopefully result in a bunch of traces that look much better... albeit without any instrumentation for the DB operations.

They could then try to get traces back for the DB operations using the EntityFrameworkCore Instrumentation for OpenTelemetry .NET.

[InterstellarStella]

Thanks a lot for looking into this @jamescrosswell I have notified the customer about your suggestion and waiting for an update.

[InterstellarStella]

@jamescrosswell I have not received a response so we could close this for now.