kamon-io / Kamon

Distributed Tracing, Metrics and Context Propagation for applications running on the JVM
https://kamon.io
Other
1.41k stars 328 forks source link

Investigate initializing metrics before using them with Prometheus #1049

Open SimunKaracic opened 3 years ago

SimunKaracic commented 3 years ago

Issue to track the following (raised on gitter by @imRentable):

Hi, I recently started using kamon-prometheus and noticed that a counter metric always yielded 0 when querying it via the increase or rate function of PromQL. 
The reason for this was that the corresponding Kamon counter has been initialised just right before incrementing it. 
Therefore, no initial counter value of 0 has been exported. 
So I did some research and stumbled across this part of the Prometheus documentation: https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics
It recommends to initialise all metrics before using them. 
I'd like to do this but it seems very tedious/unrealistic to do it manually by calling every metric with every possible label combination at the start of my application. 
So, I wonder, is there some utility or configuration for kamon-prometheus that initializes all the metrics (or rather series) automatically so that initial values are exported?
imRentable commented 3 years ago

Hi,

in the meantime I thought I avoided the aforementioned problem by just initialising all metrics with each expected label combination. Unfortunately, this didn't hold quite true. When a counter is initialised with labels by calling Kamon.counter(name).withTags an initial value of 0 is exported immediately, which is desired. However, if a histogram is initialised in a similar fashion, no metrics are exported which introduces the same issue and leads to potentially wrong metrics.

I could record a fake value to initialise a histogram, but this would mean that all my metrics would include this fake initial value. I think this is not a suitable approach since it messes with the resulting metrics.