census-instrumentation / opencensus-csharp

Distributed tracing and stats collecting framework
https://opencensus.io
Apache License 2.0
139 stars 32 forks source link

Capture custom request/response properties #130

Closed lmolkova closed 5 years ago

lmolkova commented 5 years ago

The scenario:

Azure APIs return server request is in HTTP response headers and we want to put it on the span created to trace this http call. We may argue the validity of this scenario and this might change in the future with W3C adoption, current support processes rely on this.

Also, this scenario of enhancing telemetry based on arbitrary request properties is popular among ApplicationInsights users.

While HttpClient instrumentation with DiagnosticSource allows it easily (you have the whole response object), Http listener in opencensus does not capture anything custom and does not provide extensibility points.

We have several options here

  1. Keep it as is. Ask users/library owners to implement Http listeners on their own which just enhance current span. This approach requires deep knowledge of DiagnosticSource, error-prone and extremely inefficient. It also requires special configurations done by users. This is a viable workaround for now.

  2. Provide hooks through DependenciesCollectorOptions like

    class DependenciesCollectorOptions
    {
    //...
    public  Action<HttpRequestMessage request, HttpResponseMessage response, Span span> EnhanceSpan;
    }
  3. Provide a way to disable automatic instrumentation for certain calls that require special treatment and ask users/libraries to implement instrumentation themselves with OpenCensus or DiagnosticSource.

    • Provide configuration (e.g. for certain HTTP Url do not instrument). The problem of this approach is that by default 2 spans are collected and headers are injected twice. Only when a proper configuration is applied, we have something working. It's also quite inefficient and hard to configure
    • It might be possible to auto-detect the presence of another instrumentation, but it would require setting some additional tags on the Activity OR attributes on Span OR both because so far we can have any of them :( So we'll need to implement another AsyncLocal which first instrumentation will lock and the next one would not run is detects this 'lock'.

Any of above requires someone to reimplement the whole HTTP instrumentation to do something as simple as reading one custom header. And custom instrumentations would be error prone and likely would not produce high-quality data.

So I believe (1) is workaround (2) is a possible solution, but we might want to spend time designing better API

SergeyKanzhelev commented 5 years ago

Moved to OpenTelemetry