Currently, each time a metrics endpoint is hit (with collect parameter e.g., /metrics?collect=pubsub.googleapis.com/topic), a new collector is created. This causes the monitoring metrics (stackdriver_monitoring_api_calls_total, stackdriver_monitoring_scrapes_total etc.) to reset on each scrape.
Solution
This PR:
Add collector caching in the handler using a map
Key the cache using project ID + filters combination
Reuse collectors to maintain metric state across scrapes
Testing
[x] Tested build locally, with one of our GCP project for different collector(s)
[x] Deployed in one of our cluster and tested with multiple collector (~ 110+ different metrics-type-prefixes)
[x] Verified metrics accumulate properly across scrapes for each collector
[x] Memory usage remains stable over time and is similar to previous version
Fix monitoring metrics
Problem
Currently, each time a metrics endpoint is hit (with
collect
parameter e.g.,/metrics?collect=pubsub.googleapis.com/topic
), a new collector is created. This causes the monitoring metrics (stackdriver_monitoring_api_calls_total
,stackdriver_monitoring_scrapes_total
etc.) to reset on each scrape.Solution
This PR:
Testing