search

Stackdriver / stackdriver-prometheus-sidecar

A sidecar for the Prometheus server that can send metrics to Stackdriver.
https://cloud.google.com/monitoring/kubernetes-engine/prometheus
Apache License 2.0
120 stars 43 forks source link

Any predefined filters to avoid metric duplication on GKE (and overcost from Stackdriver)? #227

Open sneko opened 4 years ago

sneko commented 4 years ago

Hi,

Just got a bill of $600 for 2,5GB of metrics on Stackdriver within a month... surprise! haha

After investigating it seems to come from external metrics, which are pushed by Prometheus. Since Google makes it free for all metrics from nodes and pods (since I'm GKE), and that I see my own Prometheus exports also those metrics, how could I filter all metrics Google is already managing?

Does anyone of you have a list for that?

Thank you,

AgrimPrasad commented 4 years ago

Yes, it's mentioned in the docs. Basically, you need to include only application metrics with the prometheus sidecar and exclude everything else. https://cloud.google.com/monitoring/kubernetes-engine/prometheus#managing-costs

For example, to import only those metrics generated by kubernetes-pods and kubernetes-service-endpoints, add the following --include statement when starting the stackdriver-prometheus-sidecar:

 --include={job=~"kubernetes-pods|kubernetes-service-endpoints"}