One of the issues of using this exporter outside of the sidecar model is that it might generate false values when using rate(), specially for counters and (conventional) histograms (which are also counters at the end). Due to how the function itself works and how data is incremented inside the exporter.
Lenghty explanation - click to expand
For example, an application could send 5 increments to the metric `myapp.http.requests.total` before the exporter is scraped for the first time, this will result in a scrape like:
```
myapp_http_requests_total{} 5
```
If the metric stays with that same value for a while, when using rate() function over it will result in 0 because the value never changed, when in reality it was not existent before so it went from 0 to 5 because the pod just gone up.
(Very) Recently, Prometheus started to accept zero-samples to tell it when a counter started to exist and therefore be able to properly calculate rates on series that are created like non-zero values.
Would be really nice to start preparing the exporter for this to work and we have more consistent metrics between Statsd and Prometheus. I would be willing to provide a PR in the near future if the maintainers think there is value in it.
matthiasr
commented
6 months ago
Summary
One of the issues of using this exporter outside of the sidecar model is that it might generate false values when using
rate(), specially for counters and (conventional) histograms (which are also counters at the end). Due to how the function itself works and how data is incremented inside the exporter.Lenghty explanation - click to expand
For example, an application could send 5 increments to the metric `myapp.http.requests.total` before the exporter is scraped for the first time, this will result in a scrape like: ``` myapp_http_requests_total{} 5 ``` If the metric stays with that same value for a while, when using rate() function over it will result in 0 because the value never changed, when in reality it was not existent before so it went from 0 to 5 because the pod just gone up.More on this problem, in the excellent talk from Bartek and Arthur.
(Very) Recently, Prometheus started to accept zero-samples to tell it when a counter started to exist and therefore be able to properly calculate rates on series that are created like non-zero values.
Would be really nice to start preparing the exporter for this to work and we have more consistent metrics between Statsd and Prometheus. I would be willing to provide a PR in the near future if the maintainers think there is value in it.