search

metrics-rs / metrics

A metrics ecosystem for Rust.
MIT License
1.13k stars 158 forks source link

Discarding old labeled metrics #328

Closed HurricanKai closed 2 years ago

HurricanKai commented 2 years ago

It's often useful to label metrics with somewhat temporary values, ie match ids in games. It makes it easier to investigate performance in more detail later. Unfortunately it seems metric-label combinations are kept around forever, leading to the exporting to slow down to a halt, in a scrape I've done just now on a test server the prometheus exporter had accumulated 11 million lines, and during export the whole system slowed down to a halt, with some requests even timing out.

Is there a way to delete old metrics somehow? Ideally they would be discarded after some seconds of going unused, but I can see that's difficult. Maybe providing a way to delete a metric by name & labels would be possible?

tobz commented 2 years ago

There's actually already a mechanism to do this, if you're using the Prometheus exporter: PrometheusBuilder::idle_timeout. It allows you to expire metrics that have been idle for a given amount of time, where idle is defined as having not been updated in that timeframe.

HurricanKai commented 2 years ago

Thanks, that should work!