search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
16.96k stars 2.31k forks source link

Kublet embedded cAdvisor outputs empty container_name fields #2249

Open IvanovOleg opened 5 years ago

IvanovOleg commented 5 years ago

I am trying to get the container CPU usage information in grafana, but getting emty diagrams. Grafana has the next dashboard request: sum by (container_name) (rate(container_cpu_usage_seconds_total{job=\"kubelet\", namespace=\"$namespace\", image!=\"\",container_name!=\"POD\",pod_name=\"$pod\"}[1m])) Here is the part of the cAdvisor output: container_cpu_usage_seconds_total{container_name="",cpu="total",id="/kubepods/burstable/pod6c90ee67-83c5-11e9-94aa-000d3a218149",image="",name="",namespace="default",pod_name="cassandra-1"} 23630.333620239 As you can see, there is the empty container_name field. Can anyone help?

Environment:

dashpole commented 5 years ago

That looks like a metric for the pod's cgroup. container_name=="POD" is the "pause" container for the pod.

IvanovOleg commented 5 years ago

@dashpole cadvisor outputs container_name="". In the grafana dashboard there is a filter container_name!="POD", not container_name=="POD". In the prometeus I see no container_name field.

dashpole commented 5 years ago

Right, there are two seperate metrics. One for the "pause container", which has container_name=="POD", and one for the pod cgroup, which has container_name==""

IvanovOleg commented 5 years ago

@dashpole Looks like grafana is able to build a separate graph for each container in the pod, but as long as container_name="" it shows nothing. As far as I know container_name field shoud not be empty. cAdvisor outputs container_name="" for all entries.

bboreham commented 5 years ago

Do you have a nonstandard Docker root directory? If so see #1932

IvanovOleg commented 5 years ago

@bboreham I am using the containerd runtime. Maybe because of that I have no metadata.

daxwang commented 5 years ago

same promblem, the container_name in some node cadvisor metrics is not empty, but a lot is empty. is there any may to fix it?

IvanovOleg commented 5 years ago

@daxwang I've fixed that by updating containerd to the latest version

cgiraldo commented 4 years ago

@IvanovOleg which containerd version?

We are hitting the same problem in k3s v1.0.0 that uses containerd v1.3.0-k3s.4.

IvanovOleg commented 4 years ago

@cgiraldo It was containerd 1.2.7, CoreOs, kubernetes 1.14.6, and remote runtime configuration for kubelets that points to the containerd (no docker on hosts).

chenkaiyue commented 4 years ago

@daxwang I've fixed that by updating containerd to the latest version

@IvanovOleg Which containerd version you upgrade to?

IvanovOleg commented 4 years ago

@chenkaiyue 1.2.7

jordy1024 commented 4 years ago

Right, there are two seperate metrics. One for the "pause container", which has container_name=="POD", and one for the pod cgroup, which has container_name==""

@dashpole Excuse me, the container named POD is the famous pause container? Are there official documents related to it? The condition container_name!="POD" means that the metadata of the pause container is filtered out, right?

dashpole commented 4 years ago

Excuse me, the container named POD is the famous pause container? Are there official documents related to it? The condition container_name!="POD" means that the metadata of the pause container is filtered out, right?

Just to be precise, it filters out metric streams for the pause container, not metadata generally. Most people, if they want to graph the containers in their pod, don't want to see resource usage for the pause container, as it doesn't do much. The name of the pause container is an implementation detail of some container runtimes, but doesn't apply to all, and isn't guaranteed to stick around.

jordy1024 commented 4 years ago

@dashpole Thank you,Let me understand it again and consult you if you have any questions;

bboreham commented 3 years ago

There's a related discussion here with a bit more background: https://github.com/containerd/cri/issues/922