prometheus-community / stackdriver_exporter

Google Stackdriver Prometheus exporter
Apache License 2.0
256 stars 97 forks source link

[Feature Request] Add support for native histogram #216

Open geNAZt opened 1 year ago

geNAZt commented 1 year ago

Hi,

we currently try to implement a SLO on the GCP metric 'loadbalancing.googleapis.com/https/backend_latencies'. This currently gets handled by the exporter in what is now called a conventional histogram in prometheus. This does not allow the usage of new histogram functions like https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_fraction. We would like to be able to use it so we can select latencies under 500ms with histogram_fraction(0, 500, rate(stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_backend_latencies_bucket[5m])) and above with this histogram_fraction(500, +Inf, rate(stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_backend_latencies_bucket[5m]))

If wanted i could also provide a PR with that feature.

Thanks

SuperQ commented 1 year ago

FYI, It's possible to get "latencies under 500ms" / apdex scores with classic Prometheus histograms. This is possible because the classic histogram buckets are <= cumulative. But of course you need to have a bucket defined for the fraction you want.

For example, this would give you the "less than 500ms" rate:

rate(stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_backend_latencies_bucket{le="0.5"}[5m])

But, supporting native histograms is something we should do. I welcome the contribution.

geNAZt commented 1 year ago

Sorry for the long response time. Thanks for the suggestion but this doesn't work since google exports that metric with a exponential bucket option and the le is all over the place thats why we got to the idea of using histogram_faction in the first place.

It doesn't seem like we can alter the bucket configuration for GCP predefined metrics (at least i couldn't find it) and the exporter doesn't have any "post processing" to rearrange bucket boundaries it seems

lasermoth commented 1 year ago

There is another issue here which touches on the need for bucket boundaries.

However I think native histogram support would be a desirable addition regardless.