I've been testing opentelemetry SDK (python) to report counters, gauges to OTEL collector, and found out that when using PeriodicExportingMetricReader to periodically report the collected measurements, even though there is no values actually being reported from the observable gauges, the counters and gauges will keep reporting the last reported values (probably occurring within the reader itself?)
There are a couple of problems associated with this sort of behavior.
As for the gauge type of data, the keep reporting the last reported value with new timestamp may pose a confusion that the gauge measurement is still active, even though there's no point actually being reported from the source.
There is no way for the users to explicitly access the reader storage and remove the entry - during the lifecycle of the reader.
in a highly ephemeral environment - if the sources somehow change often, this may simply create large number of unnecessary telemetry time series that can pose for performance and storage.
Therefore, we may need some clear definition of how long we will keep the idle metrics (which stopped reporting regularly from the sources) to eventually disappear from the reader's metrics storage. As far as I'm aware, it doesn't look like there is any mention of this concept yet in opentelemetry metrics.
I've been testing opentelemetry SDK (python) to report counters, gauges to OTEL collector, and found out that when using
PeriodicExportingMetricReaderto periodically report the collected measurements, even though there is no values actually being reported from the observable gauges, the counters and gauges will keep reporting the last reported values (probably occurring within the reader itself?)There are a couple of problems associated with this sort of behavior.
Therefore, we may need some clear definition of how long we will keep the idle metrics (which stopped reporting regularly from the sources) to eventually disappear from the reader's metrics storage. As far as I'm aware, it doesn't look like there is any mention of this concept yet in opentelemetry metrics.
A good example is how statsd lets user configure such behavior in its configuration: https://github.com/statsd/statsd/blob/master/exampleConfig.js?MobileOptOut=1#L61
I hope this issue ticket would steer the development of opentelemetry into a better direction forward.