search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.12k stars 2.32k forks source link

Unable to get cadvisor container running with ZFS dataset #1630

Open robbiewu007 opened 7 years ago

robbiewu007 commented 7 years ago

Run the container per guideline:

robbie@S001:~$ sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=5678:8080 --detach=true --name=cadvisor google/cadvisor:latest

logs output from container:

robbie@S001:~$ docker logs cadvisor I0404 13:28:08.025454 1 storagedriver.go:50] Caching stats in memory for 2m0s I0404 13:28:08.025722 1 manager.go:143] cAdvisor running in container: "/docker/a31e9eba8d7e9a6cda2300ef64fe667e389846d8c64a7253fd825c46ed68912b" W0404 13:28:08.044234 1 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused I0404 13:28:08.068203 1 fs.go:117] Filesystem partitions: map[mydata/projects:{mountpoint:/rootfs/mydata/projects major:0 minor:71 fsType:zfs blockSize:0} mydata/projects/docker:{mountpoint:/rootfs/mydata/projects/docker major:0 minor:72 fsType:zfs blockSize:0} mydata/projects/github:{mountpoint:/rootfs/mydata/projects/github major:0 minor:73 fsType:zfs blockSize:0} mydata/docker/gerrit:{mountpoint:/rootfs/mydata/docker/gerrit major:0 minor:45 fsType:zfs blockSize:0} mydata/docker/mysql/gerrit:{mountpoint:/rootfs/mydata/docker/mysql/gerrit major:0 minor:47 fsType:zfs blockSize:0} mydata/mirrors/aosp:{mountpoint:/rootfs/mydata/mirrors/aosp major:0 minor:55 fsType:zfs blockSize:0} mydata/mirrors/github/quantopian:{mountpoint:/rootfs/mydata/mirrors/github/quantopian major:0 minor:64 fsType:zfs blockSize:0} mydata/mirrors/github/shidenggui:{mountpoint:/rootfs/mydata/mirrors/github/shidenggui major:0 minor:65 fsType:zfs blockSize:0} mydata/mirrors/github:{mountpoint:/rootfs/mydata/mirrors/github major:0 minor:58 fsType:zfs blockSize:0} mydata/mirrors/github/dmlc:{mountpoint:/rootfs/mydata/mirrors/github/dmlc major:0 minor:61 fsType:zfs blockSize:0} mydata/docker_var:{mountpoint:/var/lib/docker/zfs major:0 minor:76 fsType:zfs blockSize:0} mydata:{mountpoint:/rootfs/mydata major:0 minor:37 fsType:zfs blockSize:0} mydata/docker/nginx/config:{mountpoint:/rootfs/mydata/docker/nginx/config major:0 minor:51 fsType:zfs blockSize:0} mydata/docker/nginx/www:{mountpoint:/rootfs/mydata/docker/nginx/www major:0 minor:53 fsType:zfs blockSize:0} mydata/docker/solr:{mountpoint:/rootfs/mydata/docker/solr major:0 minor:80 fsType:zfs blockSize:0} mydata/mirrors:{mountpoint:/rootfs/mydata/mirrors major:0 minor:54 fsType:zfs blockSize:0} mydata/projects/work:{mountpoint:/rootfs/mydata/projects/work major:0 minor:75 fsType:zfs blockSize:0} mydata/docker_var/0d66d360b0cb221187e49ef9809a1eb071320d617871902d83748d1bc7a3a577:{mountpoint:/rootfs/var/lib/docker/zfs/graph/0d66d360b0cb221187e49ef9809a1eb071320d617871902d83748d1bc7a3a577 major:0 minor:85 fsType:zfs blockSize:0} /dev/sdf1:{mountpoint:/rootfs major:8 minor:81 fsType:ext4 blockSize:0} mydata/dataset:{mountpoint:/rootfs/mydata/dataset major:0 minor:39 fsType:zfs blockSize:0} mydata/docker:{mountpoint:/rootfs/mydata/docker major:0 minor:44 fsType:zfs blockSize:0} mydata/mirrors/brillo:{mountpoint:/rootfs/mydata/mirrors/brillo major:0 minor:56 fsType:zfs blockSize:0} mydata/mirrors/lampweb.org:{mountpoint:/rootfs/mydata/mirrors/lampweb.org major:0 minor:70 fsType:zfs blockSize:0} mydata/dataset/Places2:{mountpoint:/rootfs/mydata/dataset/Places2 major:0 minor:41 fsType:zfs blockSize:0} mydata/dataset/mnist:{mountpoint:/rootfs/mydata/dataset/mnist major:0 minor:42 fsType:zfs blockSize:0} mydata/docker/nextcloud:{mountpoint:/rootfs/mydata/docker/nextcloud major:0 minor:49 fsType:zfs blockSize:0} mydata/mirrors/github/gbeced:{mountpoint:/rootfs/mydata/mirrors/github/gbeced major:0 minor:62 fsType:zfs blockSize:0} mydata/docker_var/f6eb18839ffe5dc004a52f35d71953c00f22c877d085c80b8474f6e5f423e014:{mountpoint:/rootfs/var/lib/docker/zfs/graph/f6eb18839ffe5dc004a52f35d71953c00f22c877d085c80b8474f6e5f423e014 major:0 minor:81 fsType:zfs blockSize:0} mydata/docker_var/4aa38bb915e241a4da9425d9426a7df139f7b8b49fae2f7b5114a9f3a161c04a:{mountpoint:/ major:0 minor:121 fsType:zfs blockSize:0} mydata/docker/nginx/logs:{mountpoint:/rootfs/mydata/docker/nginx/logs major:0 minor:52 fsType:zfs blockSize:0} mydata/projects/happy_hours:{mountpoint:/rootfs/mydata/projects/happy_hours major:0 minor:74 fsType:zfs blockSize:0} mydata/mirrors/github/lanbing510:{mountpoint:/rootfs/mydata/mirrors/github/lanbing510 major:0 minor:63 fsType:zfs blockSize:0} mydata/docker_var/2ade2ddddf88c150aa513fc523515d3e69a40c907449f65d326b735fd2ff37d3:{mountpoint:/rootfs/var/lib/docker/zfs/graph/2ade2ddddf88c150aa513fc523515d3e69a40c907449f65d326b735fd2ff37d3 major:0 minor:105 fsType:zfs blockSize:0} mydata/backups:{mountpoint:/rootfs/mydata/backups major:0 minor:38 fsType:zfs blockSize:0} mydata/dataset/ImageNet:{mountpoint:/rootfs/mydata/dataset/ImageNet major:0 minor:40 fsType:zfs blockSize:0} mydata/mirrors/codeaurora:{mountpoint:/rootfs/mydata/mirrors/codeaurora major:0 minor:57 fsType:zfs blockSize:0} mydata/mirrors/github/BVLC:{mountpoint:/rootfs/mydata/mirrors/github/BVLC major:0 minor:59 fsType:zfs blockSize:0} mydata/mirrors/github/Rockyzsu:{mountpoint:/rootfs/mydata/mirrors/github/Rockyzsu major:0 minor:60 fsType:zfs blockSize:0} mydata/docker_var/170b369bcf3389524e090c42cfc4d697e72e0c2836d16e26e69dddbefce6a1fa:{mountpoint:/rootfs/var/lib/docker/zfs/graph/170b369bcf3389524e090c42cfc4d697e72e0c2836d16e26e69dddbefce6a1fa major:0 minor:113 fsType:zfs blockSize:0} mydata/docker/mysql:{mountpoint:/rootfs/mydata/docker/mysql major:0 minor:46 fsType:zfs blockSize:0} mydata/docker/mysql/nextcloud:{mountpoint:/rootfs/mydata/docker/mysql/nextcloud major:0 minor:48 fsType:zfs blockSize:0} mydata/docker/nginx:{mountpoint:/rootfs/mydata/docker/nginx major:0 minor:50 fsType:zfs blockSize:0} mydata/mirrors/github/tensorflow:{mountpoint:/rootfs/mydata/mirrors/github/tensorflow major:0 minor:66 fsType:zfs blockSize:0} mydata/mirrors/github/waditu:{mountpoint:/rootfs/mydata/mirrors/github/waditu major:0 minor:69 fsType:zfs blockSize:0} mydata/dataset/mscoco:{mountpoint:/rootfs/mydata/dataset/mscoco major:0 minor:43 fsType:zfs blockSize:0} mydata/mirrors/github/torch:{mountpoint:/rootfs/mydata/mirrors/github/torch major:0 minor:67 fsType:zfs blockSize:0} mydata/mirrors/github/vnpy:{mountpoint:/rootfs/mydata/mirrors/github/vnpy major:0 minor:68 fsType:zfs blockSize:0} mydata/docker_var/c4b97d4540a3ce62858c5624517315fa3e57399c55769c0e889853fbda0644df:{mountpoint:/rootfs/var/lib/docker/zfs/graph/c4b97d4540a3ce62858c5624517315fa3e57399c55769c0e889853fbda0644df major:0 minor:97 fsType:zfs blockSize:0}] E0404 13:28:18.087336 1 fs.go:333] Stat fs failed. Error: exit status 1: "/usr/sbin/zfs zfs get -Hp all mydata/mirrors/github/shidenggui" => The /dev/zfs device is missing and must be created. Try running 'udevadm trigger' as root to create it.

Try to check the host ZFS dataset, verything is fine.

robbie@S001:~$ sudo zfs get -Hp all mydata/mirrors/github/shidenggui mydata/mirrors/github/shidenggui type filesystem - mydata/mirrors/github/shidenggui creation 1485160768 - mydata/mirrors/github/shidenggui used 134651904 - mydata/mirrors/github/shidenggui available 9450716110848 - mydata/mirrors/github/shidenggui referenced 134651904 - mydata/mirrors/github/shidenggui compressratio 1.01x - mydata/mirrors/github/shidenggui mounted yes - mydata/mirrors/github/shidenggui quota 0 default mydata/mirrors/github/shidenggui reservation 0 default mydata/mirrors/github/shidenggui recordsize 131072 default mydata/mirrors/github/shidenggui mountpoint /mydata/mirrors/github/shidenggui default mydata/mirrors/github/shidenggui sharenfs off default mydata/mirrors/github/shidenggui checksum on default mydata/mirrors/github/shidenggui compression lz4 inherited from mydata mydata/mirrors/github/shidenggui atime on default mydata/mirrors/github/shidenggui devices on default mydata/mirrors/github/shidenggui exec on default mydata/mirrors/github/shidenggui setuid on default mydata/mirrors/github/shidenggui readonly off default mydata/mirrors/github/shidenggui zoned off default mydata/mirrors/github/shidenggui snapdir hidden default mydata/mirrors/github/shidenggui aclinherit restricted default mydata/mirrors/github/shidenggui canmount on default mydata/mirrors/github/shidenggui xattr on default mydata/mirrors/github/shidenggui copies 1 default mydata/mirrors/github/shidenggui version 5 - mydata/mirrors/github/shidenggui utf8only off - mydata/mirrors/github/shidenggui normalization none - mydata/mirrors/github/shidenggui casesensitivity sensitive - mydata/mirrors/github/shidenggui vscan off default mydata/mirrors/github/shidenggui nbmand off default mydata/mirrors/github/shidenggui sharesmb off default mydata/mirrors/github/shidenggui refquota 0 default mydata/mirrors/github/shidenggui refreservation 0 default mydata/mirrors/github/shidenggui primarycache all default mydata/mirrors/github/shidenggui secondarycache all default mydata/mirrors/github/shidenggui usedbysnapshots 0 - mydata/mirrors/github/shidenggui usedbydataset 134651904 - mydata/mirrors/github/shidenggui usedbychildren 0 - mydata/mirrors/github/shidenggui usedbyrefreservation 0 - mydata/mirrors/github/shidenggui logbias latency default mydata/mirrors/github/shidenggui dedup off default mydata/mirrors/github/shidenggui mlslabel none default mydata/mirrors/github/shidenggui sync standard default mydata/mirrors/github/shidenggui refcompressratio 1.01x - mydata/mirrors/github/shidenggui written 134651904 - mydata/mirrors/github/shidenggui logicalused 129600000 - mydata/mirrors/github/shidenggui logicalreferenced 129600000 - mydata/mirrors/github/shidenggui filesystem_limit 18446744073709551615 default mydata/mirrors/github/shidenggui snapshot_limit 18446744073709551615 default mydata/mirrors/github/shidenggui filesystem_count 18446744073709551615 default mydata/mirrors/github/shidenggui snapshot_count 18446744073709551615 default mydata/mirrors/github/shidenggui snapdev hidden default mydata/mirrors/github/shidenggui acltype off default mydata/mirrors/github/shidenggui context none default mydata/mirrors/github/shidenggui fscontext none default mydata/mirrors/github/shidenggui defcontext none default mydata/mirrors/github/shidenggui rootcontext none default mydata/mirrors/github/shidenggui relatime on temporary mydata/mirrors/github/shidenggui redundant_metadata all default mydata/mirrors/github/shidenggui overlay off default

Docker inspect:

robbie@S001:~$ docker inspect cadvisor [ { "Id": "a31e9eba8d7e9a6cda2300ef64fe667e389846d8c64a7253fd825c46ed68912b", "Created": "2017-04-04T13:28:06.359313942Z", "Path": "/usr/bin/cadvisor", "Args": [ "-logtostderr" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 6692, "ExitCode": 0, "Error": "", "StartedAt": "2017-04-04T13:28:07.969987217Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:f9ba08bafdeaf8158b0c6dee1398bf023e7cb8f8505c0db0a65c1f7dd3dd8232", "ResolvConfPath": "/var/lib/docker/containers/a31e9eba8d7e9a6cda2300ef64fe667e389846d8c64a7253fd825c46ed68912b/resolv.conf", "HostnamePath": "/var/lib/docker/containers/a31e9eba8d7e9a6cda2300ef64fe667e389846d8c64a7253fd825c46ed68912b/hostname", "HostsPath": "/var/lib/docker/containers/a31e9eba8d7e9a6cda2300ef64fe667e389846d8c64a7253fd825c46ed68912b/hosts", "LogPath": "/var/lib/docker/containers/a31e9eba8d7e9a6cda2300ef64fe667e389846d8c64a7253fd825c46ed68912b/a31e9eba8d7e9a6cda2300ef64fe667e389846d8c64a7253fd825c46ed68912b-json.log", "Name": "/cadvisor", "RestartCount": 0, "Driver": "zfs", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/sys:/sys:ro", "/var/lib/docker/:/var/lib/docker:ro", "/:/rootfs:ro", "/var/run:/var/run:rw" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "default", "PortBindings": { "8080/tcp": [ { "HostIp": "", "HostPort": "5678" } ] }, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": -1, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0 }, "GraphDriver": { "Name": "zfs", "Data": null }, "Mounts": [ { "Source": "/sys", "Destination": "/sys", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Source": "/var/lib/docker", "Destination": "/var/lib/docker", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Source": "/", "Destination": "/rootfs", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Source": "/var/run", "Destination": "/var/run", "Mode": "rw", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "a31e9eba8d7e", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "ExposedPorts": { "8080/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GLIBC_VERSION=2.23-r3" ], "Cmd": null, "Image": "google/cadvisor:latest", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "/usr/bin/cadvisor", "-logtostderr" ], "OnBuild": null, "Labels": {} }, "NetworkSettings": { "Bridge": "", "SandboxID": "9dcf99da068b248bec4a01a23e40ce1e92ea1fc0b39a8d2315e61308f7a4d6ac", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": { "8080/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "5678" } ] }, "SandboxKey": "/var/run/docker/netns/9dcf99da068b", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "3235c71d2889a4820ed58962e892ca9970c4a333807d6098edeea459b2d4101c", "Gateway": "172.17.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "MacAddress": "02:42:ac:11:00:02", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "3ba8d5ab05b7c7ab629ac4f102e1d0038a39eaf40857b8907d83fd3a54e6b1ff", "EndpointID": "3235c71d2889a4820ed58962e892ca9970c4a333807d6098edeea459b2d4101c", "Gateway": "172.17.0.1", "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:42:ac:11:00:02" } } } } ]

Docker info:

robbie@S001:~$ docker info Containers: 7 Running: 6 Paused: 0 Stopped: 1 Images: 23 Server Version: 1.12.6 Storage Driver: zfs Zpool: mydata Zpool Health: ONLINE Parent Dataset: mydata/docker_var Space Used By Parent: 6596947968 Space Available: 9453336629248 Parent Quota: no Compression: lz4 Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge null host overlay Swarm: inactive Runtimes: runc Default Runtime: runc Security Options: apparmor seccomp Kernel Version: 4.4.0-71-generic Operating System: Ubuntu 16.04.2 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.63 GiB Name: S001 ID: 723V:EVCH:JMRX:PNM4:TNLF:MMRL:KHUQ:DATP:KPRP:DUKZ:TWHA:XUP4 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ WARNING: No swap limit support Insecure Registries: 127.0.0.0/8

dashpole commented 7 years ago

That is really strange... I have googled a little bit, and it may be possible that etc/mtag hasnt been symlinked to /proc/mounts, which is where cAdvisor is reading mounts from in the error message. See https://github.com/zfsonlinux/zfs/issues/4680 and https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/1607920. If you restart cAdvisor, do you get the same error?

robbiewu007 commented 7 years ago

Got the same error, after restarted cAdvisor.

dashpole commented 7 years ago

It is probably an issue of cAdvisor not being able to "see" a file it depends on. Try running the command inside a container with the same directories mounted. You could use: sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro -i -t alpine:3.4 /bin/sh and run zfs get -Hp all mydata/mirrors/github/shidenggui from inside the container. cAdvisor is built on alpine 3.4

robbiewu007 commented 7 years ago

robbie@S001:/mydata/projects/docker/mynextcloud$ sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro -i -t alpine:3.4 /bin/sh / # zfs get -Hp all mydata/mirrors/github/shidenggui /bin/sh: zfs: not found

imajes commented 7 years ago

Hey - i'm getting this as well. In my case, I know that /etc/mtab successfully symlinks to /proc/self/mounts as expected, but it's still barfing on any commands with zfs. For help, the container host is ubuntu yakkety.