search

beam-telemetry / telemetry_metrics_prometheus_core

Core Prometheus Telemetry.Metrics Reporter package for telemetry_metrics_prometheus
Apache License 2.0
35 stars 30 forks source link

resetting sum/counter metrics #62

Open rixmann opened 8 months ago

rixmann commented 8 months ago

Hello, we are switching to this library for metrics exposure with an application that used https://hex.pm/packages/prometheus beforehand. And by doing so we run into usage-scenarios that we can't handle at the moment (correct me please if i am wrong).

The biggest problem is when a sanity-check triggers which leads to sum/counter metrics getting re-initialized with absolute values. For sum-metrics we found an (ugly) workaround: we pull the metric value from the ets-table and then calculate the delta to our desired absolute value and emit an event that will have the desired effect. In case of counter metrics this approach is not possible (but its also fine to use sum with prometheus_type: counter in this case).

So we would really like to able to simply set an absolute value for sum and counter metrics.

Thanks for your attention to this topic, Ole

bryannaegele commented 8 months ago

I could be wrong but the prometheus spec does not allow setting counters to absolute values and can only be reset to 0, the given example being a server restart.

https://prometheus.io/docs/instrumenting/writing_clientlibs/#counter

For sums, there is no sum as an independent metric type in prometheus (only part of summary and histogram) so to support the TelemetryMetrics sum it's converted to a gauge (I believe) since it is allowed to decrement.

There is some functionality that is allowed or has MAY in the spec which this library doesn't implement.

For the functionality you're describing, you'd always need to use gauge but there are parts of TelemetryMetrics which don't support what you're wanting to do. You can to submit a PR that tries to tackle some of the things you want for the sum type but you would have to change your usage of counters to be sums going forward.