Metric expiry/reset

[lobeck]

statsd will automatically reset counters when publishing to graphite and i'm looking for a similar feature in the exporter too. Did i miss the settings for this or isn't this available?

Currently we use counters which get reset on every publish, so we can use that values to detect inactivity or outages (when the metric is no longer updated). Technically it's a gauge with automatic expiry when not updated for a while.

Is there an option to expire or reset after a certain amount of time? Otherwise we'll need to look into the full prometheus instrumentation migration which was actually planned for later this year.

[SuperQ]

Counters which get reset on every publish seem more like increment events. The statsd_exporter should be able to take these events and turn them into "real" Prometheus counters.

You could simulate the lack of updates with Prometheus by using the changes() function.

ALERT MetricNotUpdating
  IF changes(counter_metric_total[10m]) < 1

This would alert if no updates have been published to Prometheus via the cumulative counter in the last 10 minutes.

[tonidas]

What about gauges @SuperQ? It would be nice if the exporter reset its metrics to prevent old data being considered new.

[SuperQ]

It is not possible to expire gauges as the statsd protocol does not have "this is expired" message that I'm aware of. Gauges represent "the last sampled value", so they're always "stale" depending on the timeframe of the value. It is not possible to correctly expire them without an explicit method in the statsd protocol.

[tonidas]

@SuperQ Thanks for the reply.

I'm considering to reboot the container, periodically, to detect the absence of new metrics, as they are not persisted.

[SuperQ]

It would be good to tie the lifecycle of the statsd_exporter to the lifecycle of the app writing to it. This way you get consistency with the metrics exported by the application writing to the exporter.