tokio-rs / tracing-opentelemetry

MIT License
233 stars 81 forks source link

Populate attributes from span tree? #88

Open x1a0 opened 9 months ago

x1a0 commented 9 months ago

Feature Request

Motivation

In tracing, I think it's common that the metadata about one trace are spread in different spans. Different layers in an application create spans, and tag the span with metadata they are responsible.

When a metric event is emitted, I'd like to be able to use those metadata that could be found in the span tree. Either flatten and keep all attributes in the tree and attach them to the metric event, or, even better, being able to somehow filter the attributes.

ymgyt commented 9 months ago

how about this ?

  1. Add an option to MetricsLayer to enable collection of attributes from span field.
  2. Add a closure to allow filtering of fields to be added to the metrics attribute from
MetrycsLayer::new(provider)
  .with_collect_span(true)
  .with_collect_span_filter(| field: &tracing::field::Field| -> bool { field.name() == "my_interest" })

If span collect is enabled, MetricsLayer iterates span during metrics event processing and adds the field of span to metrics attribute.

x1a0 commented 9 months ago

@ymgyt Thanks for the quick reply. The proposal looks good 👍

I am trying to implement it locally but facing a problem:

MetricsLayer iterates span during metrics event processing and adds the field of span to metrics attribute

I cannot figure out how to directly read the fields of spans when processing an event. I have some code like:

        if let Some(scope) = ctx.event_scope(event) {
            for span in scope.from_root() {
                // how to visit span's fields here...?
            }
        }

So far by reading other libraries, my learning is that I need to store the fields in span extensions, then I can read from the extensions when processing an event - is this the only way?

ymgyt commented 9 months ago

@x1a0

So far by reading other libraries, my learning is that I need to store the fields in span extensions, then I can read from the extensions when processing an event - is this the only way?

Yes, as I understand it, if you want to refer to the span's field later, you need to set it in the extension of each span.

        if let Some(scope) = ctx.event_scope(event) {
            for span in scope.from_root() {
                let mut extension = span.extensions_mut();
                let Some(metrics_layer_ext) = extension.get_mut::<MetricsLayerExtension>() else {
                  continue;
                }
                // get metrics attribute from metrics_layer_ext 
                // and insert here https://github.com/tokio-rs/tracing-opentelemetry/blob/b2013d085cee229c83eb1558c3263e536186fb89/src/metrics.rs#L404
            }
        }

There might be other ways, but as far as I know, that is the only way.