search

beam-telemetry / telemetry_metrics_prometheus_core

Core Prometheus Telemetry.Metrics Reporter package for telemetry_metrics_prometheus
Apache License 2.0
35 stars 30 forks source link

persistence when supervisor restarts / down #42

Closed guigaoliveira closed 3 years ago

guigaoliveira commented 3 years ago

Today this library uses :ets, would it be possible to adopt a way to persist value of metrics when supervisor restarts? Usually when deploying with kubernetes, pods restarts and consequently applications and resets metrics in :ets. I think we can use :ets and save metrics in DETS too, and read metrics from DETS when supervisor start.

bryannaegele commented 3 years ago

Prometheus is designed to lose everything between server restarts. All reporting tooling is conscientious of this, e.g. when a counter resets to zero, it will smooth things properly in a rate on that counter. If you're set to a 15s scrape, at most you're going to lose 15s worth of data on the container that is shutting down.

guigaoliveira commented 3 years ago

so Prometheus which resets a counter metric? I suspected it was this library just. Any tips to minimize this? or simply there is no way to avoid this reset?

bryannaegele commented 3 years ago

It's expected. Prometheus was designed for it - https://prometheus.io/docs/instrumenting/writing_clientlibs/#counter

If you want to avoid losing that 15s, you could change to the push model and implement that. Then you'd need to trap the sigterm when the server is told to shut down and do a final push.

guigaoliveira commented 3 years ago

makes sense for me, thanks!