perlun
commented
1 day ago I got another idea when writing the issue: enabling the console exporter for tracing only, and see what it emits. Here's the outcome:
On the deployed environment
No output whatsoever (when looking with docker logs -f <container>), apart from some Microsoft.AspNetCore and Startup.cs-based logging on app startup.
In the local environment
In addition to the above logging (and some other more detailed logging in general, enabled via appsettings.Development.json), activity tracing output is properly logged to the console:
Activity.TraceId: 9e1617832d938d791854f973cb6d2d25
Activity.SpanId: 613b6f5b1fd87116
Activity.TraceFlags: Recorded
Activity.ActivitySourceName: Microsoft.AspNetCore
Activity.DisplayName: GET api/path/to/resource-1
Activity.Kind: Server
Activity.StartTime: 2024-10-23T06:44:37.3324209Z
Activity.Duration: 00:00:00.0167533
Activity.Tags:
[...]Will update the issue title accordingly. The problem seems to be with collection of tracing activity data in general; it's not isolated to the OTLP exporter per se.
Kielek
cijothomas
Package
OpenTelemetry
Package Version
Runtime Version
net8.0
Description
Hi,
We have been debugging a really weird problem for quite some time now, so I thought I'd reach out and ask for help. The problem is with our ASP.NET Core application not producing any tracing-related data, despite being enabled in our
Startupclass.What's even more weird: the exact same application works fine when running locally (both in Docker and on the Linux host). In the deployed environment, it runs inside Docker and this is where it doesn't work.
I am not necessarily saying this is a bug in OpenTelemetry, but something in our application or elsewhere is causing the OpenTelemetry-based instrumentation to malfunction.
Steps to Reproduce
Unfortunately, I don't currently have a minimal reproducible example; in an isolated ASP.NET application based on the "getting started" example (https://opentelemetry.io/docs/languages/net/getting-started/), everything works as intended. The problem is isolated to our production code.
Setup code
We set up OpenTelemetry in a method called from
ConfigureServicesin ourStartupclass:Expected Result
Tracing data from ASP.NET being sent to the configured OTLP exporter.
Actual Result
No tracing data emitted whatsoever.
Additional Context
NuGet package references
In addition to the package versions listed above, we also tested with
1.10.0-beta.1+1.9.0ofOpenTelemetry.Instrumentation.AspNetCore, with no difference.What we have tested
Set up the OpenTelemetry Collector using these instructions, for easy(er) debugging. Presume the name of the container is
otel-collector.Run the application (inside Docker). Provoke some HTTP requests that produces tracing-related data. Check the logs of the OpenTelemetry Collector using this command:
docker logs otel-collector 2>&1 | grep data_type. On the environments where this doesn't work, the command outputs data roughly like this. As can be seen, notraces-related data is emitted from the application. (logswas also missing at one point but I think this was because of a misconfiguration in our app)The
OTEL_DIAGNOSTICS.json-generated diagnosticsWhen it works (on my local machine), the log looks roughly like this. The "Activity stopped" events contains the path to the route being traced (
GET api/path/to/resource-1etc).When it doesn't work, it looks like this. Note how the
Activity stoppedentries are lacking the URL paths.More details
I've debugged this to the best of my ability, and I suspected that the
if (this.IsEnabled(EventLevel.Verbose, EventKeywords.All))call here returned false: https://github.com/open-telemetry/opentelemetry-dotnet/blob/5dff99f8a0b26ff75248d5f2e478b9c3c42f5678/src/OpenTelemetry/Internal/OpenTelemetrySdkEventSource.cs#L61-L65Because of limitations in my IDE, I was unable to place a breakpoint in 3rd party code when attaching to the process running inside the Docker container on the machine where we saw the problem, so I couldn't confirm this. Also, now when writing this, I am thinking: if the event source is somehow disabled, would we even get any
Activity startedevents being logged at all? :thinking:I am very much at the end of the road here; we don't know how to debug this further. Any ideas/suggestions are greatly appreciated. :pray: