Stackdriver Prometheus Sidecar with Prometheus Operator Helm chart

[Montana]

Hey Folks Montana Mendy here,

I have setup Prometheus + Grafana on our GKE cluster using the stable/prometheus-operator helm chart. Now I want to export some metrics to Stackdriver because I have installed custom metrics Stackdriver adapter. I'm already using some Pub/Sub metrics from Stackdriver for autoscaling few deployments. Now I also want to use some Prometheus metrics (mainly nginx request rate) in the autoscaling of other deployments.

So, my first question: Can I use Prometheus adapter in parallel with Stackdriver adapter for autoscaling in the same cluster?

If not, I will need to install Stackdriver Prometheus Sidecar for exporting the Prometheus metrics to Stackdriver and then use them for autoscaling via Stackdriver adapter.

From the instructions here, it looks like I need to install Stackdriver sidecar on same pod on which Prometheus is running. I gave it a try. When I run the patch.sh script, I got the message back: statefulset.apps/prometheus-prom-operator-prometheus-o-prometheus patched but when I inspected the statefulset again, it didn't have the Stackdriver sidecar container in it. Since this statefulset is created by a Helm chart, I probably can't modify it directly. Is there a recommended way of doing this in Helm?

I thought I could in theory update the deployment by updating the helm template or something like kuebctl get statefulset prometheus-prom-operator-prometheus-o-prometheus -o yaml --export file.yaml, add the required fields then kubectl apply -f file.yaml.

I've found the volume that is mounted on this container on same path. In my case, it wasprometheus-prom-operator-prometheus-o-prometheus-db so I mounted the same volume on my Stackdriver sidecar container as well. Still no dice, I also ran: helm list (for obvious reasons) then ran: helm upgrade -f prom-config.yaml prom-operator stable/prometheus-operator.

Thanks, Montana

[kehao95]

Hi @Montana I just met the same challenge when trying to patch the Prometheus with Stackdriver sidecar. Then I found out that to patch prometheus managed by Prometheus-Operator, you need to update your Prometheus object(CRD). There is a field call containers for you to inject sidecars.

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusspec

Once you update the Prometheus yaml. The prometheus operator will update the statefulset accordingly, inject the sidecar for you.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
spec:
  ...
  containers:
  - args:
    - 
    image: gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar:0.8.0
    imagePullPolicy: Always
    name: sidecar
    ports:
    - containerPort: 9091
      name: sidecar
    volumeMounts:
    - mountPath: /prometheus
      name: prometheus-k8s-db

If you are using Helm chart, there should be a value for that. If you are using kube-promtheus you can update using jsonnet.

[Montana]

Hey @kehao95,

Thank you so much! This did the trick.

[WaterKnight1998]

@kehao95 @Montana I have tried using this too. However, I am just getting the metrics from the prometheus namespace.