search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.23k stars 2.33k forks source link

cAdvisor is not starting #3612

Open jura43 opened 1 month ago

jura43 commented 1 month ago

I am trying to run cAdvisor in Docker container, but every time 2 minutes after creating container is up, but state is unhealthy.

sudo docker run \
  -idt \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --restart unless-stopped \
  --net=host \
  --name=cadvisor \
  gcr.io/cadvisor/cadvisor-amd64:v0.49.1 \
  -disable_metrics=disk,network,tcp,udp \
  -enable_load_reader=false \
  -housekeeping_interval=55s \
  -docker_only=true \
  -v=4
CONTAINER ID   IMAGE                                                                           COMMAND                  CREATED         STATUS                     PORTS       NAMES
ee1b9443812d   gcr.io/cadvisor/cadvisor-amd64:v0.49.1                                          "/usr/bin/cadvisor -…"   2 minutes ago   Up 2 minutes (unhealthy)               cadvisor
user@server1.local:~$ curl http://localhost:8080/metrics -vvv
*   Trying 127.0.0.1:8080...
* TCP_NODELAY set
* connect to 127.0.0.1 port 8080 failed: Connection refused
*   Trying 127.0.0.1:8080...
* TCP_NODELAY set
* connect to 127.0.0.1 port 8080 failed: Connection refused
* Failed to connect to localhost port 8080: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8080: Connection refused
user@server1.local:~$ ss -tulpn | grep 8080
user@server1.local:~$ docker -v
Docker version 20.10.5, build 55c4c88
user@server1.local:~$ uname -a
Linux trans1.itc 5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

This are all logs I am getting:

I1023 13:33:02.470580       1 cadvisor.go:123] enabled metrics: advtcp,app,cpu,cpuLoad,cpu_topology,cpuset,diskIO,hugetlb,memory,memory_numa,oom_event,percpu,perf_event,process,referenced_memory,resctrl,sched
I1023 13:33:02.470667       1 storagedriver.go:55] Caching stats in memory for 2m0s
I1023 13:33:02.470818       1 manager.go:169] cAdvisor running in container: "/docker/ee1b9443812db9d6ee29cb5944e25b141e78682a1ffe015a00219ab6f8981a40"
I1023 13:33:02.642212       1 fs.go:133] Filesystem UUIDs: map[07c73ab5-b553-456e-bb02-4656f6ae2fdf:/dev/dm-0 29088e98-7091-495b-8116-151e12ddec51:/dev/md127 3e457501-12a8-492b-bdb5-541cead4d734:/dev/md0 6ffc5d26-98d7-4130-9e36-43e9aa01120f:/dev/dm-3 9d0f01a5-ceda-447b-ab39-9074c08750b8:/dev/dm-2 d3617fa4-9a37-487a-8aaa-a2f9eea0c4db:/dev/dm-1]
I1023 13:33:02.642296       1 fs.go:134] Filesystem partitions: map[/dev:{mountpoint:/dev major:0 minor:75 fsType:tmpfs blockSize:0} /dev/mapper/server1.local-root:{mountpoint:/rootfs major:253 minor:0 fsType:ext4 blockSize:0} /dev/mapper/server1.local-var:{mountpoint:/var/lib/docker major:253 minor:1 fsType:ext4 blockSize:0} /dev/mapper/server1.local-var--log:{mountpoint:/rootfs/var/log major:253 minor:2 fsType:ext4 blockSize:0} /dev/md0:{mountpoint:/rootfs/boot major:9 minor:0 fsType:ext2 blockSize:0} /dev/md127:{mountpoint:/rootfs/media major:9 minor:127 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:78 fsType:tmpfs blockSize:0} /proc/acpi:{mountpoint:/proc/acpi major:0 minor:79 fsType:tmpfs blockSize:0} /proc/kcore:{mountpoint:/proc/kcore major:0 minor:75 fsType:tmpfs blockSize:0} /proc/keys:{mountpoint:/proc/keys major:0 minor:75 fsType:tmpfs blockSize:0} /proc/sched_debug:{mountpoint:/proc/sched_debug major:0 minor:75 fsType:tmpfs blockSize:0} /proc/scsi:{mountpoint:/proc/scsi major:0 minor:80 fsType:tmpfs blockSize:0} /proc/timer_list:{mountpoint:/proc/timer_list major:0 minor:75 fsType:tmpfs blockSize:0} /rootfs/dev/shm:{mountpoint:/rootfs/dev/shm major:0 minor:26 fsType:tmpfs blockSize:0} /rootfs/run:{mountpoint:/rootfs/run major:0 minor:25 fsType:tmpfs blockSize:0} /rootfs/run/lock:{mountpoint:/rootfs/run/lock major:0 minor:27 fsType:tmpfs blockSize:0} /rootfs/run/user/1000:{mountpoint:/rootfs/run/user/1000 major:0 minor:82 fsType:tmpfs blockSize:0} /rootfs/sys/fs/cgroup:{mountpoint:/rootfs/sys/fs/cgroup major:0 minor:28 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/containers/50abac060d19a1af3202de289bacd1afc74734b41df2df52e89c8687b0459d58/mounts/secrets:{mountpoint:/rootfs/var/lib/docker/containers/50abac060d19a1af3202de289bacd1afc74734b41df2df52e89c8687b0459d58/mounts/secrets major:0 minor:109 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/containers/851fc8be2c7a3b5ca4656c567b6636129a318813641947aecfc59368949327eb/mounts/secrets:{mountpoint:/rootfs/var/lib/docker/containers/851fc8be2c7a3b5ca4656c567b6636129a318813641947aecfc59368949327eb/mounts/secrets major:0 minor:121 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev:{mountpoint:/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev major:0 minor:75 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev/shm:{mountpoint:/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev/shm major:0 minor:78 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/sys/fs/cgroup:{mountpoint:/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/sys/fs/cgroup major:0 minor:77 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:25 fsType:tmpfs blockSize:0} /run/lock:{mountpoint:/run/lock major:0 minor:27 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:82 fsType:tmpfs blockSize:0} /sys/firmware:{mountpoint:/sys/firmware major:0 minor:81 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:28 fsType:tmpfs blockSize:0} /var/lib/docker/containers/50abac060d19a1af3202de289bacd1afc74734b41df2df52e89c8687b0459d58/mounts/secrets:{mountpoint:/var/lib/docker/containers/50abac060d19a1af3202de289bacd1afc74734b41df2df52e89c8687b0459d58/mounts/secrets major:0 minor:109 fsType:tmpfs blockSize:0} /var/lib/docker/containers/851fc8be2c7a3b5ca4656c567b6636129a318813641947aecfc59368949327eb/mounts/secrets:{mountpoint:/var/lib/docker/containers/851fc8be2c7a3b5ca4656c567b6636129a318813641947aecfc59368949327eb/mounts/secrets major:0 minor:121 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev major:0 minor:75 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev/shm major:0 minor:78 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/dev/shm:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/dev/shm major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/run:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/run major:0 minor:25 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/run/lock:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/run/lock major:0 minor:27 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/run/user/1000:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/run/user/1000 major:0 minor:82 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/sys/fs/cgroup:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/sys/fs/cgroup major:0 minor:28 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/containers/50abac060d19a1af3202de289bacd1afc74734b41df2df52e89c8687b0459d58/mounts/secrets:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/containers/50abac060d19a1af3202de289bacd1afc74734b41df2df52e89c8687b0459d58/mounts/secrets major:0 minor:109 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/containers/851fc8be2c7a3b5ca4656c567b6636129a318813641947aecfc59368949327eb/mounts/secrets:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/containers/851fc8be2c7a3b5ca4656c567b6636129a318813641947aecfc59368949327eb/mounts/secrets major:0 minor:121 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev major:0 minor:75 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/dev/shm major:0 minor:78 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/sys/fs/cgroup:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/sys/fs/cgroup major:0 minor:77 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/run:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/run major:0 minor:25 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/run/lock:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/run/lock major:0 minor:27 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/run/user/1000:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/run/user/1000 major:0 minor:82 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/sys/fs/cgroup:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/sys/fs/cgroup major:0 minor:28 fsType:tmpfs blockSize:0} 172.22.1.10:/media/input:{mountpoint:/rootfs/media/input major:0 minor:96 fsType:nfs4 blockSize:0} 172.22.1.10:/media/transcoder:{mountpoint:/rootfs/media/transcoder major:0 minor:96 fsType:nfs4 blockSize:0} nfsd:{mountpoint:/rootfs/proc/fs/nfsd major:0 minor:120 fsType:nfsd blockSize:0} overlay_0-107:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/383910581a2e4db6ebfc0382ffda040efedfe8dcd1085758a62b9dc241ce52ce/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/dec4560c75b4f0426aee2d5cdf0f0fe8da323b71d27c599a63c0ee79bca30112/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/1507f43bbb6b3f4f4b7d1d4eaddbbc1214586f988136b781dcc236222c13404c/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-52:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/5e312edd2ba284611aa651ce3801e5d313ee0bd88ed516a905e83a419dd70c8d/merged major:0 minor:52 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/3118521020b357b1f4bf98fd4a7744d98594d02f955e4a5e76b51ad3a8437b05/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/21cfc62cf07fc065175b274b95328007c9b9f725c43b9969e19be6e870ca8d0a/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/docker/overlay2/fafe331a5fce5bea4b6e7aae00ec06bb162b6a0ffe37a1d41d047f62c1206400/merged/rootfs/var/lib/docker/overlay2/0b3c31efacc2132099d436907e2e5ed1bd21792c0757a2e65a05b77496c238b1/merged major:0 minor:97 fsType:overlay blockSize:0}]

I also tried to run cAdvisor without container, but result is the same.

jura43 commented 1 month ago

With version v0.50.0 cAdvsior starts without issues.