weaveworks / common

Libraries used in multiple Weave projects
Other
129 stars 92 forks source link

Enable Prometheus native histograms for request_duration_seconds #280

Closed beorn7 closed 1 year ago

beorn7 commented 1 year ago

This is an opt-in via setting Config.MetricsNativeHistogramFactor so that nobody will expose native histograms by accident. Even if native histograms are enabled, the conventional histograms are still maintained and presented as before to scrapers incapabale of scraping native histograms.

Native histograms were tested previously in the sparse-histograms branch without any problems. This commit obsoletes the sparse-histograms branch.

The histograms request_message_bytes and response_message_bytes are left as conventional histograms for now. Maybe they would work well as native histograms with a bucket factor of 2, as high resolution is not the aim here. But we can decide about that once we have gathered more experience with the native histograms for request_duration_seconds.

The settings for limiting the bucket count (i.e. NativeHistogramMaxBucketNumber = 100 and NativeHistogramMinResetDuration = 1h) are both conservative and most likely good enough for the expected use cases. We can make them configurable later, if users feel the need. The idea here is to not add too many knobs prematurely and thereby avoid confusion.

Also includes an update of prometheus/client_golang to v1.14.0 as that is the earliest release supporting native histograms.

beorn7 commented 1 year ago

@bboreham a number of people want to try out native histograms in production. With the change in this PR, they don't need to build from a branch anymore. For those that don't use native histograms, nothing should change.

beorn7 commented 1 year ago

Feedback addressed. PHAL.