grafana / helm-charts

Apache License 2.0
1.67k stars 2.29k forks source link

Multiple port pod with grafana annotations causes duplicate scrapes that leads to high DPM #3334

Closed ivan-figarolabs-com closed 1 month ago

ivan-figarolabs-com commented 1 month ago
metadata:
  annotations:
    ...
    "k8s.grafana.com/metrics.portNumber" = "9091"
    ...
spec:
  ...
  ports:
  - name: "http-9080"
    port: 9080
    target_port: 9080
    protocol: "TCP"
  - name: "http-9091"
    port: 9091
    target_port: 9091
    protocol: "TCP"

chart: k8s-monitoring-1.5.0 app version: 2.8.1

This causes to scrape 2x on port 9091, which makes the DPM 2x as well. The more ports we specify in the pod spec, the higher the DPM eventually gets.

ivan-figarolabs-com commented 1 month ago

https://github.com/grafana/helm-charts/compare/k8s-monitoring-1.5.0...k8s-monitoring-1.5.4

I don't see any fixes in the diff to the current latest version btw.

ivan-figarolabs-com commented 1 month ago

From support they mentioned this has been fixed in 1.5.2