kgeckhart
commented
2 months ago So unfortunately, you're trying to compare apples and oranges by looking at sum of the exporter vs GCP due to this being a delta metric. The exporter is doing an in-memory accumulation of delta values it has accumulated since startup. Just like with a pure prometheus counter you should apply a rate or an increase to look at the change over time vs the raw value. This is where the data is intended to match GCP as close as possible. You can read more about some caveats in https://github.com/prometheus-community/stackdriver_exporter?tab=readme-ov-file#what-to-know-about-aggregating-delta-metrics
Hello, today we've noticed that the data that is exported by this exporter doesn't match up with what GCP monitoring shows even if the queries are identical between both environments.
The metric in question is _stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_requestcount and in Grafana the total value shows to be about 60% less than that is shown in GCP.
The discrepancy is (logically) more noticeable in resources that have a lot more requests going through them. The full query used in both environments is:
sum by(response_code_class) (stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_request_count{target_proxy_name="_REDACTED_"})