search

micrometer-metrics / micrometer

An application observability facade for the most popular observability tools. Think SLF4J, but for observability.
https://micrometer.io
Apache License 2.0
4.49k stars 994 forks source link

OpenMetrics `_created` timestamp #2625

Open shakuzen opened 3 years ago

shakuzen commented 3 years ago

See the OpenMetrics specification. This is a follow-up to the initial tracking issue #2419.

We don't currently track or publish the created timestamp for each MetricFamily.

alshain commented 7 months ago

This seems to landing soon in Prometheus, so it'd be great to see it supported in Micrometer.

https://github.com/prometheus/prometheus/pull/13506

shakuzen commented 7 months ago

Thanks for the heads up. I've subscribed to updates on that PR. I'm not sure if we need to do anything in Micrometer for this anymore since we've upgrade to the Prometheus Java client 1.x (see #4406). The new client has an exporter configuration for created timestamps - see https://prometheus.github.io/client_java/config/config/#exporter-properties. I suspect that would work with Micrometer 1.13.0 (currently 1.13.0-RC1 is available from the milestone repository). Does that do what you need?

stefan-fast commented 1 week ago

I looked into this and tried using it. Indeed, it seems to be implemented in the lastest versions of the Prometheus Java Client and Micrometer is using these new versions with support for includeCreatedTimestamps. Unfortunately, the createdTimestampMillis parameter is always set to 0L in micrometer for supported metrics types, e.g. counters: https://github.com/micrometer-metrics/micrometer/blob/be70882669c690b28f8ec1c6143edc2fb33a7cb2/implementations/micrometer-registry-prometheus/src/main/java/io/micrometer/prometheusmetrics/PrometheusMeterRegistry.java#L221 (last parameter in that line). Therefore, the hasCreatedTimestamp() check in the Prometheus Java Client is always false: https://github.com/prometheus/client_java/blob/e17c0a10d5b55afc107b2e0114a433742b96a0ab/prometheus-metrics-exposition-formats/src/main/java/io/prometheus/metrics/expositionformats/PrometheusTextFormatWriter.java#L102 Because of that, the _created timestamp is never actually written yet. In summary, if I didn't miss anything, micrometer still needs to implment _created timestamp support.

Are there any plans to support it in the near future?

jonatan-ivanov commented 1 week ago

@stefan-fast Yes, that's what this issue is about: we don't track the created timestamp right now. We are going to do some planning for the next feature release, I will drop this into the set of topics we will discuss but it seems right now there is not a lot of users who needs this

Is there a certain use-case you need _created for or is the lack of it causes issues for you or is it a nice to have since the specs says it should be there?

stefan-fast commented 1 week ago

@jonatan-ivanov Thanks for taking this into the planning discussions!

Yes, basically this blog post by Arthur Sens sums up and explains the problem pretty well: https://www.arthursens.dev/posts/created-timestamps

I have counters that increase fast before the first scrape by prometheus. So the first increase from 0 to this first scrape value X is missing. Using increase or rate functions on this metric will then result in wrong values to be displayed. E.g. I did an increase over a longer timespan to get the total count of requests in that timeframe. Because the counter only increased before the first scrape and not after anymore, the increase function over the timespan resulted in 0 instead of the real correct value.

alshain commented 1 week ago

We ran into the same thing. Also, we have metrics with labels that depend on dynamic labels, such as the exception class.

So even if the application has been scraped for hours, these new metrics will appear with an initial count of 1, and due to the lack of a previous scrape and no _created timestamp, alerts won't trigger.