getsentry / sentry-dotnet

Sentry SDK for .NET
https://docs.sentry.io/platforms/dotnet
MIT License
595 stars 207 forks source link

Sentry.OpenTelemetry: The Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to initialize the Sentry SDK. #2738

Closed sinanbozkus closed 11 months ago

sinanbozkus commented 1 year ago

Package

Other

.NET Flavor

.NET

.NET Version

7.0.12

OS

Linux

SDK Version

7.0.0

Self-Hosted Sentry Version

No response

Steps to Reproduce

When I was using "Sentry.OpenTelemetry" 3.39.1 or older everything was ok. After updating 3.40.0 or 3.40.1 I got this error.

The Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to initialize the Sentry SDK.

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
            tracerProviderBuilder
                .AddSource("Development-Application.Api")
                .ConfigureResource(resource => resource.AddService("Development-Application.Api"))
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddSentry() // <-- Configure OpenTelemetry to send trace information to Sentry
    );

builder.WebHost.UseSentry(o =>
{
    o.Dsn = "XXXXXXXXX";
    o.TracesSampleRate = 1.0;
    o.UseOpenTelemetry();
});

Expected Result

No error

Actual Result

The Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to initialize the Sentry SDK.initialized

jamescrosswell commented 1 year ago

Thanks for the report @sinanbozkus !

Are you able to try switching those statments, so the Sentry registration comes first?

builder.WebHost.UseSentry(o =>
{
    o.Dsn = "XXXXXXXXX";
    o.TracesSampleRate = 1.0;
    o.UseOpenTelemetry();
});

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
            tracerProviderBuilder
                .AddSource("Development-Application.Api")
                .ConfigureResource(resource => resource.AddService("Development-Application.Api"))
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddSentry() // <-- Configure OpenTelemetry to send trace information to Sentry
    );
sinanbozkus commented 1 year ago

, so the Sentry registration comes first

services.AddOpenTelemetry()
            .WithTracing(builder =>
                {
                    builder.AddSentry();
                    builder.AddSource("Development-Application.Api"))
                        .ConfigureResource(resource => resource.AddService("Development-Application.Api"));
                    // builder.AddSentry(); // or here
                    builder.AddAspNetCoreInstrumentation();
                    builder.AddHttpClientInstrumentation();
                    builder.AddConsoleExporter();
                }
            );

I've tried like this but I got the same error.

jamescrosswell commented 1 year ago

Hi @sinanbozkus , can you try as I suggested (builder.WebHost.UseSentry comes first then builder.Services.AddOpenTelemetry)?

sinanbozkus commented 1 year ago

I've tried, it's the same.

jamescrosswell commented 1 year ago

Hm, the error itself is coming from here: https://github.com/getsentry/sentry-dotnet/blob/ebbb5001febc37fde119df35ae48f259a719e86c/src/Sentry.OpenTelemetry/SentrySpanProcessor.cs#L40-L47

Typically that would indicate that SentrySdk.CurrentHub had not been initialized correctly.

You could try adding o.Debug = true to your UseSentry... then check the debug output to see if there are any warnings or errors that might indicate what's going wrong with initialization.

If not, would it be possible to get you to share a minimal reproducible example in a GitHub repo that we could clone?

raulidavid commented 1 year ago

I have the same problem the Sentry SDK has not been initialised. To use Sentry with OpenTelemetry tracing you need to initialize the Sentry SDK.

Follow the instructions @jamescrosswell

jamescrosswell commented 1 year ago

@raulidavid I'm unable to reproduce this... the Sample app provided with Sentry works fine for me.

Are you able to provide a copy of what you're seeing in the Debug window when you run the application with Debug = true in your SentryOptions?

raulidavid commented 1 year ago

Version 3.40.1 Error `

<PackageVersion Include="Sentry.AspNetCore" Version="3.40.1" /> ERROR
<PackageVersion Include="Sentry.OpenTelemetry" Version="3.40.1" /> ERROR`
jamescrosswell commented 1 year ago

Version 3.40.1 Error <PackageVersion Include="Sentry.AspNetCore" Version="3.39.1" /> <PackageVersion Include="Sentry.OpenTelemetry" Version="3.39.1" /> <PackageVersion Include="Sentry.AspNetCore" Version="3.40.1" /> ERROR <PackageVersion Include="Sentry.OpenTelemetry" Version="3.40.1" /> ERROR

@raulidavid I think we're maybe not on the same wavelength... I mean the output that you see in the Debug console in your IDE. For example, this is what I see when I run the application in debug mode with SentryOptions.Debug = true:

/usr/local/share/dotnet/dotnet /Users/jamescrosswell/code/sentry-dotnet/samples/Sentry.Samples.OpenTelemetry.AspNetCore/bin/Debug/net7.0/Sentry.Samples.OpenTelemetry.AspNetCore.dll
  Debug: Logging enabled with ConsoleDiagnosticLogger and min level: Debug
  Debug: Initializing Hub for Dsn: 'https://b887218a80114d26a9b1a51c5f88e0b4@o447951.ingest.sentry.io/6601807'.
  Debug: Using 'GzipBufferedRequestBodyHandler' body compression strategy with level Optimal.
  Debug: Starting BackgroundWorker.
  Debug: BackgroundWorker Started.
  Debug: New scope pushed.
  Debug: Registering integration: 'AutoSessionTrackingIntegration'.
  Debug: Registering integration: 'AppDomainUnhandledExceptionIntegration'.
  Debug: Registering integration: 'AppDomainProcessExitIntegration'.
  Debug: Registering integration: 'UnobservedTaskExceptionIntegration'.
  Debug: Registering integration: 'SentryDiagnosticListenerIntegration'.
  Debug: New scope pushed.
   Info: Replacing current logger with: 'MelDiagnosticLogger'.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: https://localhost:5092
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/jamescrosswell/code/sentry-dotnet/samples/Sentry.Samples.OpenTelemetry.AspNetCore
info: Sentry.ISentryClient[0]
      Sentry trace header is null. Creating new Sentry Propagation Context.
info: Sentry.ISentryClient[0]
      Envelope queued up: 'dda2b958555848daa8874b20301f6dc3'
info: Sentry.ISentryClient[0]
      Envelope 'dda2b958555848daa8874b20301f6dc3' successfully sent.
leonid-lm commented 1 year ago

Here is an example of such behavior, I'm limited to using the "Microsoft.Extensions.Hosting.Host" to ensure that I can configure the custom tracing context, that will clean up health check routes from all traces. Unfortunately, I get the same error. That's why my setup is another one than in your sample. If you can give me a hint on how could I make a customization for tracing context using WebApplication.CreateBuilder(args), I would be really glad... Thanks for your support.

using System;
using System.IO;
using System.Reflection;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Profile.Host.Config;
using Sentry;
using Sentry.OpenTelemetry;
using Serilog;
using Hosting = Microsoft.Extensions.Hosting.Host;

namespace Sample.Host
{
    public static class Program
    {
        private const string EnvVariableName = "ASPNETCORE_ENVIRONMENT";

        public static void Main(string[] args)
        {
            var envName = Environment.GetEnvironmentVariable(EnvVariableName);
            var settingsFileName = !string.IsNullOrEmpty(envName)
                ? $"appsettings.{envName}.json"
                : $"appsettings.json";
            IConfiguration configuration = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile(settingsFileName)
                .AddEnvironmentVariables()
                .Build();

            Log.Logger = new LoggerConfiguration()
                .ReadFrom.Configuration(configuration)
                .CreateLogger();

            BuildWebHost(args).Build().Run();
        }

        public static IHostBuilder BuildWebHost(string[] args)
        {
            var builder = Hosting.CreateDefaultBuilder(args);

            builder.ConfigureServices(t => t.AddOpenTelemetry()
                    .WithTracing(g => g
                    .AddSource(Startup.ServiceName)
                    .ConfigureResource(resource => resource.AddService(serviceName: Startup.ServiceName,
                        serviceVersion: Assembly.GetExecutingAssembly().GetName().Version?.ToString()))
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddConsoleExporter()
                    .AddSentry())
                .WithMetrics(m => m
                    .AddHttpClientInstrumentation()
                    .AddAspNetCoreInstrumentation()
                    .AddConsoleExporter()));

            builder.ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseSentry(options =>
                    {
                        options.TracesSampler = context =>
                        {
                            if (context.CustomSamplingContext.TryGetValue("__HttpPath", out var value))
                            {
                                string path = (string)value;
                                if (path!.Equals(Startup.HealthPath) || path.Equals(Startup.ReadinessPath) ||
                                    path.Equals(Startup.LivenessPath))
                                    return 0.0;
                            }

                            return options.TracesSampleRate;
                        };
                        options.UseOpenTelemetry();
                        options.AddEntityFramework();
                    });
                    webBuilder.UseStartup<Startup>();
                })
                .UseSerilog();

            return builder;
        }
    }
}
leonid-lm commented 1 year ago

Even when I migrated my code to the .Net 6.0 style. It still doesn't work (even with WebApplication.CreateBuilder(args)).

jamescrosswell commented 1 year ago

@leonid-lm again, would it be possible to get a copy of the debug output so that we can get some visibility into what is and what isn't working during SDK initialization? See my reply to @raulidavid above.

shokkueibu commented 1 year ago

I was having this issue and in my case it was an existing line in my project clearing the logging providers that was also clearing the SentryOptions. Moving the UseSentry call below the logging configuration did the trick.

  (...)
  .ConfigureLogging(logging =>
  {
    logging.ClearProviders();
    logging.SetMinimumLevel(LogLevel.Trace);
  })
  .UseSentry(options =>
  {
    options.UseOpenTelemetry();
  })
  (...)
leonid-lm commented 11 months ago

I was able to identify the issue's root cause. For some reason existing initialization of the sentry goes into conflict with Serilog initialization. The current issue is easily reproducible if you just add the following code:

var builder = WebApplication.CreateBuilder(args);

var startup = new Startup(builder.Configuration);
startup.ConfigureServices(builder.Services);
builder.Services.AddOpenTelemetry()
    .WithTracing(traceBuilder =>
    {
        traceBuilder
            .AddSource(Startup.ServiceName)
            .ConfigureResource(resource => resource.AddService(serviceName: Startup.ServiceName,
                serviceVersion: Assembly.GetExecutingAssembly().GetName().Version?.ToString()))
            .AddAspNetCoreInstrumentation(option => option.RecordException = true)
            .AddHttpClientInstrumentation(option => option.RecordException = true)
            .AddSentry();
    });

builder.Host.UseSerilog((context, configuration) =>
    configuration.ReadFrom.Configuration(context.Configuration));

builder.WebHost.UseSentry(options =>
{
    options.Debug = builder.Environment.IsDevelopment();
    options.TracesSampler = context =>
    {
        if (context.CustomSamplingContext.TryGetValue("__HttpPath", out var value))
        {
            string path = (string)value;
            if (path!.Equals(Startup.HealthPath) || path.Equals(Startup.ReadinessPath) ||
                path.Equals(Startup.LivenessPath))
                return 0.0;
        }
        return options.TracesSampleRate;
    };
    options.AddEntityFramework();
     options.UseOpenTelemetry();
});

var app = builder.Build();
startup.Configure(app, app.Environment);

app.Run();

If you try to remove the following lines, it works:

builder.Host.UseSerilog((context, configuration) =>
configuration.ReadFrom.Configuration(context.Configuration));

So, if you would like to have the Sentry traces enabled, you need to get rid of Serilog.

jamescrosswell commented 11 months ago

If you try to remove the following lines, it works:

builder.Host.UseSerilog((context, configuration) =>
configuration.ReadFrom.Configuration(context.Configuration));

So, if you would like to have the Sentry traces enabled, you need to get rid of Serilog.

It's a bit tricky to tell without seeing the config that is being passed to Serilog, but I suspect what's happening is that the call to UseSentry, under the hood, eventually resolves to this: https://github.com/getsentry/sentry-dotnet/blob/aa25db90780528abbe032df120255def4a249976/src/Sentry.AspNetCore/SentryWebHostBuilderExtensions.cs#L78-L83

Sentry's initialization is being delayed when the Microsoft Extension Logging backend has been replaced (by Serilog) and so when AddSentry is called on the trace builder, Sentry hasn't yet been initialized.

@leonid-lm are you able to share a sanitized version of the relevant bits of your serilog/sentry config?

knd775 commented 11 months ago

...are you able to share a sanitized version of the relevant bits of your serilog/sentry config?

I'm having the same issue, and I've removed all Serilog config except just builder.Host.UseSerilog();

Here's my sentry config:

static void ConfigureSentry(WebApplicationBuilder builder)
{
    builder.Logging.AddOpenTelemetry(b =>
    {
        b.IncludeFormattedMessage = true;
        b.IncludeScopes = true;
        b.ParseStateValues = true;
        b.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("xxxxxx"));
    });

    builder.Services
        .AddOpenTelemetry()
        .WithTracing(b =>
        {
            b.AddSource("xxxxxx");
            b.ConfigureResource(r => r.AddService("xxxxxx"));
            b.AddAspNetCoreInstrumentation();
            b.AddHttpClientInstrumentation();
            b.AddHotChocolateInstrumentation();
            b.AddSentry(); // Removing this line "fixes" the issue
        }
    );

    builder.WebHost.UseSentry(o =>
    {
        o.Dsn = builder.Configuration.GetSetting("Sentry:Dsn");
        o.TracesSampleRate = 1.0;
        o.Environment = builder.Configuration.GetSetting("Sentry:Environment");
        o.UseOpenTelemetry();
        o.SetBeforeSendTransaction((sentryTransaction, hint) =>
        {
            // Filter out requests to BCP
            if (MatchBCPRegex().IsMatch(sentryTransaction.Name))
                return null;

            return sentryTransaction;
        });
    });
}
jamescrosswell commented 11 months ago

I'm having the same issue, and I've removed all Serilog config except just builder.Host.UseSerilog();

OK, progress. I'm able to replicate the same by adding this to Sentry.Samples.OpenTelemetry.AspNetCore.csproj

builder.Host.UseSerilog();

Now all that remains is to fix the problem 😜

leonid-lm commented 11 months ago

Thanks a lot... I had a really simple config for sentry and serilog:

  "Sentry": {
    "Dsn": "<LINK>",
    "SendDefaultPii": true,
    "IncludeActivityData": true,
    "AutoSessionTracking": true,
    "AttachStackTrace": true,
    "EnableTracing": true,
    "Debug": false,
    "DiagnosticLevel": "Error",
    "TracesSampleRate": 1.0,
    "Environment": "Integration"
  },
"Serilog": {
    "Using":  [ "Serilog.Sinks.Console", "Serilog.Sinks.Seq" ],
    "LevelSwitches": { "$controlSwitch": "Information" },
    "MinimumLevel": {
      "ControlledBy": "$controlSwitch",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning",
        "Hangfire": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "theme": "Serilog.Sinks.SystemConsole.Themes.AnsiConsoleTheme::Code, Serilog.Sinks.Console",
          "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} <s:{SourceContext}>{NewLine}{Exception}"
        }
      },
      {
        "Name": "Seq",
        "Args": {
          "serverUrl": "http://seq.logs.svc.cluster.local:5341",
          "apiKey": "<KEY>"
        }
      },
      {
        "Name": "Sentry",
        "Args": {
          "minimumBreadcrumbLevel": "Debug",
          "minimumEventLevel": "Error"
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId", "WithEventType" ],
    "Properties": {
      "Application": "app-api",
      "Environment": "Integration"
    }
  },
jamescrosswell commented 11 months ago

@knd775 and @leonid-lm are you able to try with version 3.41.3?

knd775 commented 11 months ago

are you able to try with version 3.41.3?

Working for me! Thank you!

jamescrosswell commented 11 months ago

Great, I'll mark this as closed then. Any other issues I think we open a new ticket 🙂