Open gauravkarens opened 6 years ago
What problem are you experiencing? I don't see any unusual errors in the logs...
The UI is not opening on the 8080 port also can u suggest is there an API which can be used to stock the metric data coming from cadvisor
Not sure what you mean by "stock"ing metrics... How are you running cAdvisor? Just with docker, or in kubernetes as a daemonset, or using the one built into the kubelet?
I need to do an analysis of what resource utilization is associated with per container basis for a period of one day and to do that I will have to have record of the metric data produced by cadvisor so how do I do that
can you try running this curl localhost:8080/metrics
from the host and see if there is output?
That would help determine if it is a problem with cAdvisor, or a problem reaching cAdvisor from your browser.
@dashpole Hi, I cacth this problem too. The container not running and can't start the container. OS : Centos 7.4.1 docker version : 1.13.1 And Install docker by yum as default.
It likely isn't an issue with docker @Eric-LiuGang, please share your cadvisor log, and try curling the /metrics endpoint as I describe in my previous comment.
When I run cadvisor with
sudo docker run \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --publish=8080:8080 \ --detach=true \ --name=cadvisor \ google/cadvisor:latest
The container run up about 2 seconds and then stoped,so port 8080 is not up
The log as follows:
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: I0427 02:44:48.700521 1 storagedriver.go:50] Caching stats in memory for 2m0s
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: I0427 02:44:48.701273 1 manager.go:151] cAdvisor running in container: "/sys/fs/cgroup/cpuacct,cpu"
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: time="2018-04-27T10:44:48.723316610+08:00" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n"
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: time="2018-04-27T10:44:48.723412514+08:00" level=warning msg="failed to retrieve docker-init version"
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: time="2018-04-27T10:44:48.748355750+08:00" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n"
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: time="2018-04-27T10:44:48.748467364+08:00" level=warning msg="failed to retrieve docker-init version"
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: I0427 02:44:48.753382 1 fs.go:139] Filesystem UUIDs: map[]
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: I0427 02:44:48.753428 1 fs.go:140] Filesystem partitions: map[/dev/mapper/docker-253:0-50554515-850de0dbc61dcb167094b4cd7e0f60c4dec47c37cf2e2479fb245a92cfae9331:{mountpoint:/ major:253 minor:3 fsType:xfs blockSize:0} tmpfs:{mountpoint:/dev major:0 minor:39 fsType:tmpfs blockSize:0} /dev/mapper/cl-root:{mountpoint:/var/lib/docker major:253 minor:0 fsType:xfs blockSize:0} /dev/sda1:{mountpoint:/rootfs/boot major:8 minor:1 fsType:xfs blockSize:0} shm:{mountpoint:/dev/shm major:0 minor:36 fsType:tmpfs blockSize:0}]
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: I0427 02:44:48.760924 1 manager.go:225] Machine: {NumCores:4 CpuFrequency:2400085 MemoryCapacity:3974868992 HugePages:[{PageSize:2048 NumPages:0}] MachineID:5bb269792be0410d8e8cf4bcc561b3c5 SystemUUID:423735B7-EB95-DE79-C03D-EDA16581FDD0 BootID:8ae20ada-81dd-4d42-84c2-dbd650375df7 Filesystems:[{Device:shm DeviceMajor:0 DeviceMinor:36 Capacity:67108864 Type:vfs Inodes:485213 HasInodes:true} {Device:/dev/mapper/docker-253:0-50554515-850de0dbc61dcb167094b4cd7e0f60c4dec47c37cf2e2479fb245a92cfae9331 DeviceMajor:253 DeviceMinor:3 Capacity:10725883904 Type:vfs Inodes:5242368 HasInodes:true} {Device:tmpfs DeviceMajor:0 DeviceMinor:39 Capacity:1987432448 Type:vfs Inodes:485213 HasInodes:true} {Device:/dev/mapper/cl-root DeviceMajor:253 DeviceMinor:0 Capacity:18238930944 Type:vfs Inodes:8910848 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:1063256064 Type:vfsInodes:524288 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:18249416704 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:2147483648 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:10737418240 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 8:0:{Name:sda Major:8 Minor:0 Size:21474836480 Scheduler:deadline}] NetworkDevices:[{Name:ens160 MacAddress:00:50:56:b7:49:9c Speed:10000 Mtu:1500} {Name:ens192 MacAddress:00:50:56:b7:75:05 Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:4294500352 Cores:[{Id:0 Threads:[0] Caches:[]} {Id:1 Threads:[1] Caches:[]}] Caches:[{Size:12582912 Type:Unified Level:3}]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[]} {Id:1 Threads:[3] Caches:[]}] Caches:[{Size:12582912 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: I0427 02:44:48.762428 1 manager.go:231] Version: {KernelVersion:3.10.0-693.21.1.el7.x86_64 ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.13.1 DockerAPIVersion:1.26 CadvisorVersion:v0.28.3 CadvisorRevision:1e567c2}
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: time="2018-04-27T10:44:48.781672520+08:00" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n"
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: time="2018-04-27T10:44:48.781791536+08:00" level=warning msg="failed to retrieve docker-init version"
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: E0427 02:44:48.786038 1 factory.go:340] devicemapper filesystem stats will not be reported: usage of thin_ls is disabled to preserve iops
Apr 27 10:44:48 docker2.gang_test dockerd-current[6646]: I0427 02:44:48.787405 1 factory.go:356] Registering Docker factory
Apr 27 10:44:50 docker2.gang_test dockerd-current[6646]: I0427 02:44:50.788041 1 factory.go:54] Registering systemd factory
Apr 27 10:44:50 docker2.gang_test dockerd-current[6646]: I0427 02:44:50.788776 1 factory.go:86] Registering Raw factory
Apr 27 10:44:50 docker2.gang_test dockerd-current[6646]: I0427 02:44:50.789363 1 manager.go:1178] Started watching for new ooms in manager
Apr 27 10:44:50 docker2.gang_test dockerd-current[6646]: W0427 02:44:50.789410 1 manager.go:313] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory
Apr 27 10:44:50 docker2.gang_test dockerd-current[6646]: I0427 02:44:50.794910 1 manager.go:329] Starting recovery of all containers
Apr 27 10:44:50 docker2.gang_test dockerd-current[6646]: I0427 02:44:50.877151 1 manager.go:334] Recovery completed
Apr 27 10:44:50 docker2.gang_test dockerd-current[6646]: F0427 02:44:50.908107 1 cadvisor.go:156] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory
Apr 27 10:44:50 docker2.gang_test oci-systemd-hook[6916]: systemdhook
The docker service start status: /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false
Is “selinux enabled” OK?
I have a similar issue where cadvisor never completes the "Starting recovery of all containers" phase. According to the logs "Recovery completed" event is never emitted and the web UI port is never opened. I've strace'd cadvisor and there is an overwhelming number of futex errors all pointing to the same address.
futex(0x13ae8b0, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
The issue happens running in docker as well as via systemd on the host. It was working yesterday without any issues, but I suspect that it's something related to Docker. Unfortunately I cannot restart the docker daemon nor the host at this time to see if it fixes the issue. AFAICT there are no stuck or zombie containers. Cadvisor has been running for over three hours now with no change. I'm trying to replicate the issue on another system that I can restart if necessary.
Not sure why cadvisor fails to report its version number in the log, it was installed via apt, but maybe this will help:
Version: {KernelVersion:4.15.0-20-generic ContainerOsVersion:Ubuntu 18.04 LTS DockerVersion:17.12.1-ce DockerAPIVersion:1.35 CadvisorVersion: CadvisorRevision:}
apt-cache
says its cadvisor version 0.2.71
Pointing cadvisor at an invalid socket file for docker causes it to start immediately, though without docker statistics obviously. The futex errors remain so maybe theyre unrelated.
I0416 10:31:24.513018 1 storagedriver.go:50] Caching stats in memory for 2m0s I0416 10:31:24.513510 1 manager.go:151] cAdvisor running in container: "/sys/fs/cgroup/cpuacct,cpu" I0416 10:31:24.608594 1 fs.go:139] Filesystem UUIDs: map[] I0416 10:31:24.608634 1 fs.go:140] Filesystem partitions: map[/dev/sda6:{mountpoint:/rootfs/var major:8 minor:6 fsType:ext3 blockSize:0} /dev/sda1:{mountpoint:/rootfs/boot major:8 minor:1 fsType:ext3 blockSize:0} /dev/sda5:{mountpoint:/rootfs/tmp major:8 minor:5 fsType:ext3 blockSize:0} /dev/mapper/optvg-optlv:{mountpoint:/rootfs/opt major:253 minor:0 fsType:ext4 blockSize:0} shm:{mountpoint:/rootfs/opt/docker-latest/lib/containers/a949b8024c39b71fde0c74f6eb3fc44fa81c336a340fe80bb3a514924e43b01b/shm major:0 minor:80 fsType:tmpfs blockSize:0} overlay:{mountpoint:/ major:0 minor:178 fsType:overlay blockSize:0} tmpfs:{mountpoint:/dev major:0 minor:182 fsType:tmpfs blockSize:0} /dev/sda2:{mountpoint:/rootfs major:8 minor:2 fsType:ext3 blockSize:0}] I0416 10:31:24.618696 1 manager.go:225] Machine: {NumCores:4 CpuFrequency:2666761 MemoryCapacity:12331003904 HugePages:[{PageSize:2048 NumPages:0}] MachineID:776138ac4fe643cf9e2174648538e555 SystemUUID:4202D7A4-008A-D850-0214-03D3256FA251 BootID:a6f94e20-f8b7-4ac6-a822-505bfc698872 Filesystems:[{Device:/dev/sda6 DeviceMajor:8 DeviceMinor:6 Capacity:3103539200 Type:vfs Inodes:196608 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:499355648 Type:vfs Inodes:128016 HasInodes:true} {Device:/dev/sda5 DeviceMajor:8 DeviceMinor:5 Capacity:3103539200 Type:vfs Inodes:196608 HasInodes:true} {Device:/dev/mapper/optvg-optlv DeviceMajor:253 DeviceMinor:0 Capacity:63254081536 Type:vfs Inodes:3932160 HasInodes:true} {Device:shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:1505249 HasInodes:true} {Device:overlay DeviceMajor:0 DeviceMinor:178 Capacity:63254081536 Type:vfs Inodes:3932160 HasInodes:true} {Device:tmpfs DeviceMajor:0 DeviceMinor:182 Capacity:6165499904 Type:vfs Inodes:1505249 HasInodes:true} {Device:/dev/sda2 DeviceMajor:8 DeviceMinor:2 Capacity:9377800192 Type:vfs Inodes:589824 HasInodes:true}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:53687091200 Scheduler:deadline} 8:16:{Name:sdb Major:8 Minor:16 Size:64424509440 Scheduler:deadline} 253:0:{Name:dm-0 Major:253 Minor:0 Size:64399343616 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline}] NetworkDevices:[{Name:eth0 MacAddress:02:42:ac:11:00:0e Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:12884434944 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:25165824 Type:Unified Level:3}]} {Id:2 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:25165824 Type:Unified Level:3}]} {Id:4 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:25165824 Type:Unified Level:3}]} {Id:6 Memory:0 Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:25165824 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} I0416 10:31:24.621578 1 manager.go:231] Version: {KernelVersion:3.10.0-693.11.6.el7.x86_64 ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.13.1 DockerAPIVersion:1.26 CadvisorVersion:v0.28.3 CadvisorRevision:1e567c2} I0416 10:31:24.670378 1 factory.go:356] Registering Docker factory I0416 10:31:26.671372 1 factory.go:54] Registering systemd factory I0416 10:31:26.672593 1 factory.go:86] Registering Raw factory I0416 10:31:26.674437 1 manager.go:1178] Started watching for new ooms in manager W0416 10:31:26.674473 1 manager.go:313] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory I0416 10:31:26.684839 1 manager.go:329] Starting recovery of all containers I0416 10:31:26.685812 1 manager.go:334] Recovery completed I0416 10:31:26.690794 1 cadvisor.go:162] Starting cAdvisor version: v0.28.3-1e567c2 on port 8080
Please let me know how to fix this issue