search

open-telemetry / opentelemetry-dotnet-contrib

This repository contains set of components extending functionality of the OpenTelemetry .NET SDK. Instrumentation libraries, exporters, and other components can find their home here.
https://opentelemetry.io
Apache License 2.0
458 stars 271 forks source link

Not all ASP.NET requests are exported #1783

Open pardahlman opened 3 years ago

pardahlman commented 3 years ago

Bug Report

List of NuGet packages and version:

Runtime version:

Symptom

We are running an ASP.NET (classic) application with the telemetry module provided by Microsoft in Microsoft.AspNet.TelemetryCorrelation 1.0.8. HTTP requests and ASP.NET instrumentation are enabled and exported to Stackdriver through a local build of the contrib project with updated dependecy to underlying open-telemetry package to 1.0.0-rc1.1. For many requests, this works as expected and trace data is ingested by Cloud Trace. However, some requests are never properly instrumented. One may suspect the local build of Stackdriver as the culprit, but I believe that the issue lays somewhere in the ASP.NET instrumentation library. After debugging the code I have made the following observations:

When everything works:

When traces are not exported:

When backtracking why OnStopActivity is not called, I noticed that OpenTelemetry.Instrumentation.DiagnosticSourceListener's OnNext method bails out here when called with Microsoft.AspNet.HttpReqIn.Stop kvp because Activity.Current is null.

What is the expected behavior?

All HTTP requests to the ASP.NET should be instrumented and exported

What is the actual behavior?

Some http requests are not instrumented and exported

Reproduce

Unfortunately, I have not been able to create a small repro app. I will continue to investigate this issue and see if I can get back with easier repro-steps.

akselsson commented 3 years ago

Do note that we noticed this behaviour on a ServiceStack application, running the last BSD-licensed version of ServiceStack.

Here's an attempt to reproduce the problem: https://github.com/akselsson/OpenTelemetryServiceStackInvestigation

EDIT: I updated the example with example of what happens if a servicestack handler contains code that starts an activity (e.g. a http client call)

cijothomas commented 3 years ago

If the activity.current is null during OnEnd callback, then it could be a bug outside of this repo. (Like AspNetTelemetryCorrelation module.)

Is this only occurring with ServiceStack?

akselsson commented 3 years ago

Is this only occurring with ServiceStack?

As far as we know, yes. The HttpReqIn activity is finished in another plain asp.net mvc site that we host.

tijlreynhout commented 11 months ago

We are having the same problem when forwarding a http request from one .net wepabi (Server A) to another .net webapi (Server B). When inspecting the Activity.Current in an Owin middleware on Server B we also see "Microsoft.AspNet.HttpReqIn". Opentelemtry is configured for Server B but the trace for this request is not exported. If we call Server B directly, tracing works as expected. What is the status of this ticket?

nipunsahnieptura commented 1 month ago

We are using OTEL to enable monitoring for our .NET Application and are experiencing the same issue where the traces are generated with limited information and not all activities are recorded.

I suspect, the issue is similar to what has been described here, is there any update on this issue ?