metrics-rs / metrics

A metrics ecosystem for Rust.
MIT License
1.08k stars 148 forks source link

Unable to combine metrics-tracing-context and prometheus-exporter #407

Closed cBournhonesque closed 8 months ago

cBournhonesque commented 9 months ago

Hi, I'm trying to export metrics to prometheus, but also have 'scoped labels' using metrics-tracing-context to add any span variables to my metrics as labels.

However the metrics that I export don't seem to contain any additional label:

let subscriber = subscriber.with(MetricsLayer::new());
// create a prometheus exporter with tracing context support
let builder = metrics_exporter_prometheus::PrometheusBuilder::new();
let runtime = tokio::runtime::Builder::new_current_thread()
    .enable_all()
    .build()
    .unwrap();
let (recorder, exporter) = {
    let _g = runtime.enter();
    builder.build().unwrap()
};

// Add in tracing context
let traced_recorder = TracingContextLayer::all().layer(recorder);
thread::Builder::new()
    .spawn(move || runtime.block_on(exporter))
    .unwrap();
metrics::set_boxed_recorder(Box::new(traced_recorder));

then, if I do:

debug_span!("broadcast", user = "a").entered();
 metrics::increment_counter!("send_message_after_span");

I can see the span being entered in my logs, I can see the metrics being emitting; but there is no label user => "a" attached to the metric

tobz commented 9 months ago

Can you change your test example to something like:

let span = debug_span!(...);
let _entered = span.enter();
metrics::increment_counter!(...);
drop(_entered);

I have a sneaking suspicion that because you're (seemingly) not assigning the result of entered() to anything, it's being immediately dropped.

cBournhonesque commented 9 months ago

You were right; it works when I don't drop the guard immediately... Thanks! :)

Also I wonder if the process of combining tracing-context and prometheus could be made easier.

I tried to use

Stack::new(recorder)
  .push(TracingContextLayer::all())
  .install();
runtime.spawn(exporter);

instead of

let traced_recorder = TracingContextLayer::all().layer(recorder);
thread::Builder::new()
    .spawn(move || runtime.block_on(exporter))
    .unwrap();
metrics::set_boxed_recorder(Box::new(traced_recorder));

but it didn't work.

Also i basically had to copy-paste code from the prometheus-exporter even though my use-case seemed pretty common (adding some extra layer to the recorder before starting the export).

tobz commented 9 months ago

Huh, interesting.

What about it didn't work? At first blush, what you have seems fine, but I can't reasonably debug the problem without know what the problem is. :P

cBournhonesque commented 9 months ago

It just didn't seem to export any metrics

tobz commented 8 months ago

Closing this as there hasn't been any activity and there's no MCVE.