Container dashboard showing doubled metrics

[matheusneder]

Hi!

I have a kubernetes cluster on which I recently installed prometheus, through prometheus operator, following official prometheus guide.

I connected Grafana Kubernetes App to kubernetes/prometheus then I noticed that the metrics exposed at K8s container dashboard for memory and cpu was been showing doubled.

After some debug, I discovered the bellow query for cpu usage:

sum(irate(container_cpu_usage_seconds_total{pod_name=~"my-pod"}[2m])) by (pod_name)

Executing the same query but removing aggregate (container_cpu_usage_seconds_total{pod_name=~"my-pod"}) I could see that there is an extra item that represents the sum of all containers in POD as in table bellow:

============================== ‣Element ‣Value

container_cpu_usage_seconds_total{container_name="POD",cpu="total",endpoint="http-metrics",id="/kubepods/burstable/pod1bb36063-bcf4-11e8-bba4-00505684c990/da0ebc2deef52414bc2f1cc8cb874956b6c3dc478ea8df0d2eb3106a605ba21c",image="k8s.gcr.io/pause:3.1",instance="10.17.14.237:10255",job="kubelet",name="k8s_POD_my-pod-b54bc7874-bpmhd_pocs_1bb36063-bcf4-11e8-bba4-00505684c990_0",namespace="pocs",pod_name="my-pod-b54bc7874-bpmhd",service="kubelet"} 0.012306132

container_cpu_usage_seconds_total{container_name="my-pod-rev2",cpu="total",endpoint="http-metrics",id="/kubepods/burstable/pod1bb36063-bcf4-11e8-bba4-00505684c990/f712e91d11913f01daa863fd3e5d59179afb40a7f1e8ff67cf8f24874d7fd44c",image="sha256:2f41732fe757bb8dc7739b5a536ea38ed3aa9c4bc8f6c3e3c342e4952b0e63e7",instance="10.17.14.237:10255",job="kubelet",name="k8s_my-pod-rev2_my-pod-b54bc7874-bpmhd_pocs_1bb36063-bcf4-11e8-bba4-00505684c990_1",namespace="pocs",pod_name="my-pod-b54bc7874-bpmhd",service="kubelet"} 107.36438691

container_cpu_usage_seconds_total{cpu="total",endpoint="http-metrics",id="/kubepods/burstable/pod1bb36063-bcf4-11e8-bba4-00505684c990",instance="10.17.14.237:10255",job="kubelet",namespace="pocs",pod_name="my-pod-b54bc7874-bpmhd",service="kubelet"} 145.636093587

So, this justify the doubled metrics.

The extra item with the sum of containers (the 3th item on the above table) don't contains the container_name attribute.

In order to avoid Grafana Kubernetes App showing doubled values, I edited queries on dashboard definition by adding container_name=~".+" to filter, so the query for cpu_usage now looks as:

sum(irate(container_cpu_usage_seconds_total{pod_name=~"my-pod", container_name=~".+"}[2m])) by (pod_name)

I'm wondering if there is a better way to handle this. Have you ever seen this scenario before?

[gabrielfsousa]

i have the same problem :(

[ghost]

WE TOOOOOO!

[pezzati]

ME TOO!!

[ChrisCooney]

AND ME!

[chanjarster]

AND ME