eaze / tracing-honeycomb

Fork of https://github.com/inanna-malick/tracing-honeycomb
MIT License
17 stars 17 forks source link

Stacking of telemetry layers is not possible (tracing-distributed) #18

Open Infrasonics opened 2 years ago

Infrasonics commented 2 years ago

It is currently not possible to register more than one TelemetryLayer with tracing-subscriber. The problem is the inclusion of the timestamp SpanInitAt, since it leads to two spans with the same value which is not allowed by tracing-subscriber and causes a panic at runtime.

Side note: the visitor, as holder of the field values, is included as well, so this leads to the same problem when using the same visitor with different layers, yet is easier to work around.

My use case is to use separate backends for different kinds of traces.

Dropping the timestamp resolves it, yet it might lead to a different behavior for users of tracing-honeycomb. I made a simple sketch in #17 (WIP) so once it works, someone familiar with what Honeycomb expects could check whether things still work properly.

Please let me know what you think about the situation in general.

Infrasonics commented 2 years ago

I've been playing around a bit more and with my limited tracing knowledge I don't see an easy way to add the span initialization time and the duration in another place without changing the interface. For extra information and options to discuss in case, here is what I found so far:

  1. Putting the timestamp information into another layer. From the documentation of tracing-subscriber::registry::ExtensionMut:

    /// Note that extensions are not /// Layer-specific—they are span-specific. This means that /// other layers can access and mutate extensions that /// a different Layer recorded. For example, an application might /// have a layer that records execution timings, alongside a layer /// that reports spans and events to a distributed /// tracing system that requires timestamps for spans. /// Ideally, if one layer records a timestamp x, the other layer /// should be able to reuse timestamp x.

  2. Adding the time information to the values of the span instead of its extensions.
  3. Putting this kind of change behind a feature where either compatibility with honeycomb or stacking is possible.