stackdriver sidecar + k8s deployment with several replicas

[lwsanty]

For example, there's a deployment, which pod has metrics and a prom sidecar container. Then I scale the deployment to some value >= 1.

What will eventually happen with Google Cloud Monitoring metrics? Will it become a mess of metrics from several replica pods?

If yes is there any way to additionally tag such metrics to distinct one pod from another?

[igorpeshansky]

I think there's some confusion here. The sidecar in this repository runs alongside the Prometheus server, not the user pods. Once you've deployed the Prometheus server with this sidecar, there's no need to run any sidecars in your other deployments to get Prometheus metrics.

The resulting metrics will distinguish the pods (rather, containers) that produced the metrics, via the k8s_container monitored resource.

[lwsanty]

@igorpeshansky thank you for a fast reply!

In this case I should probably consider the next approach: 1) I have a deployment with several replicas that contain metrics 2) there's also a k8s service that exposes metrics port 3) I run a deployment with prometheus server(having a job to scrape the service) and a stackdriver sidecar.

Is it correct or am I missing something?

[igorpeshansky]

I think this depends on your setup, but normally step 2 is not necessary. Prometheus does its own discovery and contacts the metrics ports on each pod directly.

This sidecar is designed for configuring an existing Prometheus server to write to Stackdriver. What I would suggest doing is following any of the myriad guides on the internet for deploying a Prometheus server on your Kubernetes cluster (for example, the one from the Prometheus docs). You will likely need to annotate your pods appropriately to get them scraped. Once you get that to work, attach the sidecar to the Prometheus server.

Please reopen this if you encounter issues with the sidecar (not covered by the documentation) during this process.