search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
16.6k stars 2.29k forks source link

Memory metrics like `container_memory_max_usage_bytes` are always zero / 0 #3515

Open Kiara0107 opened 2 months ago

Kiara0107 commented 2 months ago

Issue: Memory metrics like container_memory_max_usage_bytes are always zero / 0. I've read about cgroups 1 and 2 and stuff, but I don't get it. Docker stats does return memory usage, but metrics stay all zero. How can I fix that?

Versions:

Docker run command:

docker run -d -p 8080:8080 \
      --volume=/:/rootfs:ro \
      --volume=/var/run:/var/run:ro \
      --volume=/sys:/sys:ro \
      --volume=/var/lib/docker/:/var/lib/docker:ro \
      --volume=/dev/disk/:/dev/disk:ro \
      --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
      --restart unless-stopped \
      --network prometheusnet \
      --privileged \
      --device=/dev/kmsg \
      --name=cadvisor gcr.io/cadvisor/cadvisor-amd64:v0.49.1 \
      --v=4

Output docker stats

CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
1e071aa371da   cadvisor   24.54%    55.39MiB / 7.148GiB   0.76%     292kB / 12MB      0B / 0B           11
2a32e1eaea0b   consul     3.28%     137.8MiB / 7.148GiB   1.88%     27GB / 43.4GB     12.2MB / 538GB    14
fbf1ac058ce2   promtail   0.99%     194.6MiB / 7.148GiB   2.66%     46.2MB / 282MB    4.14MB / 1.06GB   11
ac9835ee4077   loki       1.25%     77.39MiB / 7.148GiB   1.06%     3.93GB / 6.47GB   4.3MB / 8.77GB    11
4129b134c79d   mimir      3.34%     264.9MiB / 7.148GiB   3.62%     270GB / 236GB     220MB / 43.5GB    16
cb62f55ffa53   nginx      0.00%     9.531MiB / 7.148GiB   0.13%     21.8GB / 21.6GB   5.02MB / 1.52GB   3

Logrecords with failed

~$ docker logs cadvisor 2>&1 | grep fail
I0412 09:11:28.923495       1 factory.go:219] Registration of the podman container factory failed: failed to validate Podman info: response not present: Get "http://d/v1.0.0/info": dial unix /var/run/podman/podman.sock: connect: no such file or directory
I0412 09:11:28.941581       1 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I0412 09:11:28.942663       1 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version
I0412 09:11:29.044418       1 factory.go:268] Error trying to work out if we can handle /system.slice/docker-1e071aa371da08766981cfdac104510532651a2a9fc926e68f97dbf5a72e2a75.scope: failed to load container: container "1e071aa371da08766981cfdac104510532651a2a9fc926e68f97dbf5a72e2a75" in namespace "k8s.io": not found
I0412 09:11:29.075149       1 factory.go:268] Error trying to work out if we can handle /system.slice/docker-ac9835ee40778125885fc65382bb3521f88f873f78b3ef15ae79a8caf4a2b12c.scope: failed to load container: container "ac9835ee40778125885fc65382bb3521f88f873f78b3ef15ae79a8caf4a2b12c" in namespace "k8s.io": not found
I0412 09:11:29.111873       1 factory.go:268] Error trying to work out if we can handle /system.slice/docker-4129b134c79d14c678a3b4cefe6cf7ed4631f5a504bcd309989a22d64cb6dca9.scope: failed to load container: container "4129b134c79d14c678a3b4cefe6cf7ed4631f5a504bcd309989a22d64cb6dca9" in namespace "k8s.io": not found
I0412 09:11:29.130872       1 factory.go:268] Error trying to work out if we can handle /system.slice/docker-2a32e1eaea0bf7ed3ca8a3e8509df7fbf8a0098d87a387bdba81dba2890e1fb9.scope: failed to load container: container "2a32e1eaea0bf7ed3ca8a3e8509df7fbf8a0098d87a387bdba81dba2890e1fb9" in namespace "k8s.io": not found
I0412 09:11:29.146755       1 factory.go:268] Error trying to work out if we can handle /system.slice/docker-fbf1ac058ce2b701469a7e05589098f55d96469290a82e6bd9e8f14dd393105f.scope: failed to load container: container "fbf1ac058ce2b701469a7e05589098f55d96469290a82e6bd9e8f14dd393105f" in namespace "k8s.io": not found
I0412 09:11:29.361679       1 factory.go:268] Error trying to work out if we can handle /system.slice/docker-cb62f55ffa53126982d784640527ff42ee3916972b30bbf8ccc2ea42d26e2888.scope: failed to load container: container "cb62f55ffa53126982d784640527ff42ee3916972b30bbf8ccc2ea42d26e2888" in namespace "k8s.io": not found