search

prometheus-community / stackdriver_exporter

Google Stackdriver Prometheus exporter
Apache License 2.0
263 stars 99 forks source link

Apigee metrics from GCP - bucket type prometheus metrics value is not increasing monotonically #336

Open sundar-ka opened 6 months ago

sundar-ka commented 6 months ago

We've been introduced to this stackdriver_exporter recently and started using it to export metrics from GCP infra to our prometheus. Specifically the metrics of Apigee proxy provided as a managed service in GCP. We found a specific type of metric which got exported through the stackdriver exporter is not increasing monotonically in its value over the time. This metric type is used to bucket the range of values in different buckets which then can be used to calculate quantiles using histogram_quantile function with prometheus. Here is an example for metric which is used for calculating latency whose value didn't increase monotonically

test

Error got from prometheus while calculating histogram quantile using the above metric

test1

This is a helm based deployment in a kubernetes cluster.

Helm Chart version deployed - v4.5.0 Exporter app version deployed - v0.15.0

Expected: The metric _bucket value has to be increasing always like a counter type metric

Help needed . Thanks

kgeckhart commented 5 months ago

👋 can you try again with the latest release of v0.15.1? It contains a bugfix for this very issue.

sundar-ka commented 5 months ago

Thanks. I'll try it and get back

sundar-ka commented 5 months ago

@kgeckhart We've tried v0.15.1 version of the exporter, but still the values are not increasing monotonically. Getting same warning from grafana while calculating histogram quantile. We also use the current latest version of helm chart - v4.5.0

Warning below screenshot

Screenshot 2024-06-10 at 1 27 06 PM

Bucket metric screenshot below ( that doesn't increase monotonically )

Screenshot 2024-06-10 at 1 39 26 PM

kindly help here

kgeckhart commented 5 months ago

Can you check what stackdriver_exporter_build_info says regarding version and revision?

sundar-ka commented 5 months ago

Here is the version and revision details for the exporter from the metric stackdriver_exporter_build_info The version seems 0.15.1

test1
kgeckhart commented 5 months ago

🤔 troubleshooting this isn't easy at this point. I've checked for histograms in my environment which show this behavior and we have none. We also don't have any apigee metrics to go off of to see if it's a specific to something with the metric itself. Do you have other histograms with this behavior or just the apigee one?