search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
16.89k stars 2.31k forks source link

Zero stats for Docker containers network/blkio #822

Closed jimmidyson closed 9 years ago

jimmidyson commented 9 years ago

On Fedora 22 with docker-1.7.0-6.git74e7a7a.fc22.x86_64, all Docker containers report zero network traffic & blkio. Is there something that needs to be configured on the host to enable this reporting?

vmarmol commented 9 years ago

The network on seems weird, but blockio may be due to the scheduler used. Can you provide the ourpur of /validate on one of these machines? Do we know if they containers are detected as Docker containers by cAdvisor?

jimmidyson commented 9 years ago

@vmarmol The containers are shown under Docker containers so they're definitely being detected as Docker containers. FYI these containers are managed by Kubernetes, well OpenShift 3, so cadvisor is embedded in the Kubernetes/OpenShift binary. I've tried with latest cadvisor release (0.16) separately too & have the same problem so I believe it is related to cadvisor rather than Kubernetes/OpenShift's embedded version,

Here's the output of /validate:

cAdvisor version: 0.16.0

OS version: Fedora 22 (Twenty Two)

Kernel version: [Supported and recommended]
    Kernel version is 4.0.8-300.fc22.x86_64. Versions >= 2.6 are supported. 3.0+ are recommended.

Cgroup setup: [Supported and recommended]
    Available cgroups: map[devices:1 net_cls:1 perf_event:1 net_prio:1 cpuset:1 cpuacct:1 blkio:1 hugetlb:1 cpu:1 memory:1 freezer:1]
    Following cgroups are required: [cpu cpuacct]
    Following other cgroups are recommended: [memory blkio cpuset devices freezer]
    Hierarchical memory accounting enabled. Reported memory usage includes memory used by child containers.

Cgroup mount setup: [Supported and recommended]
    Cgroups are mounted at /sys/fs/cgroup.
    Cgroup mount directories: blkio cpu cpu,cpuacct cpuacct cpuset devices freezer hugetlb memory net_cls net_cls,net_prio net_prio perf_event systemd 
    Any cgroup mount point that is detectible and accessible is supported. /sys/fs/cgroup is recommended as a standard location.
    Cgroup mounts:
    cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
    cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0

Docker version: [Supported and recommended]
    Docker version is 1.7.0.fc22. Versions >= 1.0 are supported. 1.2+ are recommended.

Docker driver setup: [Supported and recommended]
    Docker exec driver is native-0.2. Storage driver is devicemapper.
    systemd is being used to create cgroups.
    Docker container state directory is at "/var/lib/docker/containers" and is accessible.

Block device setup: [Supported and recommended]
    At least one device supports 'cfq' I/O scheduler. Some disk stats can be reported.
     Disk "dm-6" Scheduler type "none".
     Disk "dm-8" Scheduler type "none".
     Disk "sda" Scheduler type "cfq".
     Disk "dm-10" Scheduler type "none".
     Disk "dm-2" Scheduler type "none".
     Disk "dm-3" Scheduler type "none".
     Disk "dm-5" Scheduler type "none".
     Disk "dm-7" Scheduler type "none".
     Disk "dm-9" Scheduler type "none".
     Disk "dm-0" Scheduler type "none".
     Disk "dm-1" Scheduler type "none".
     Disk "dm-4" Scheduler type "none".

Inotify watches: 
    /system.slice/dev-dm\x2d3.swap:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dev-dm\x2d3.swap
        /sys/fs/cgroup/memory/system.slice/dev-dm\x2d3.swap
        /sys/fs/cgroup/blkio/system.slice/dev-dm\x2d3.swap
    /system.slice/lvm2-lvmetad.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/lvm2-lvmetad.service
        /sys/fs/cgroup/memory/system.slice/lvm2-lvmetad.service
        /sys/fs/cgroup/blkio/system.slice/lvm2-lvmetad.service
    /system.slice/system-systemd\x2dfsck.slice:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/system-systemd\x2dfsck.slice
        /sys/fs/cgroup/memory/system.slice/system-systemd\x2dfsck.slice
        /sys/fs/cgroup/blkio/system.slice/system-systemd\x2dfsck.slice
    /system.slice/system-systemd\x2drfkill.slice:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/system-systemd\x2drfkill.slice
        /sys/fs/cgroup/memory/system.slice/system-systemd\x2drfkill.slice
        /sys/fs/cgroup/blkio/system.slice/system-systemd\x2drfkill.slice
    /system.slice/netcf-transaction.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/netcf-transaction.service
        /sys/fs/cgroup/memory/system.slice/netcf-transaction.service
        /sys/fs/cgroup/blkio/system.slice/netcf-transaction.service
    /system.slice/nfs-mountd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/nfs-mountd.service
        /sys/fs/cgroup/memory/system.slice/nfs-mountd.service
        /sys/fs/cgroup/blkio/system.slice/nfs-mountd.service
    /system.slice/run-user-995.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/run-user-995.mount
        /sys/fs/cgroup/memory/system.slice/run-user-995.mount
        /sys/fs/cgroup/blkio/system.slice/run-user-995.mount
    /:
        /sys/fs/cgroup/cpu,cpuacct
        /sys/fs/cgroup/cpuset
        /sys/fs/cgroup/memory
        /sys/fs/cgroup/blkio
    /system.slice/abrt-xorg.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/abrt-xorg.service
        /sys/fs/cgroup/memory/system.slice/abrt-xorg.service
        /sys/fs/cgroup/blkio/system.slice/abrt-xorg.service
    /system.slice/var-lib-nfs-rpc_pipefs.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/var-lib-nfs-rpc_pipefs.mount
        /sys/fs/cgroup/memory/system.slice/var-lib-nfs-rpc_pipefs.mount
        /sys/fs/cgroup/blkio/system.slice/var-lib-nfs-rpc_pipefs.mount
    /system.slice/avahi-daemon.service:
        /sys/fs/cgroup/blkio/system.slice/avahi-daemon.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/avahi-daemon.service
        /sys/fs/cgroup/memory/system.slice/avahi-daemon.service
    /system.slice/systemd-remount-fs.service:
        /sys/fs/cgroup/memory/system.slice/systemd-remount-fs.service
        /sys/fs/cgroup/blkio/system.slice/systemd-remount-fs.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-remount-fs.service
    /system.slice/systemd-udevd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-udevd.service
        /sys/fs/cgroup/memory/system.slice/systemd-udevd.service
        /sys/fs/cgroup/blkio/system.slice/systemd-udevd.service
    /system.slice/colord.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/colord.service
        /sys/fs/cgroup/memory/system.slice/colord.service
        /sys/fs/cgroup/blkio/system.slice/colord.service
    /system.slice/kmod-static-nodes.service:
        /sys/fs/cgroup/blkio/system.slice/kmod-static-nodes.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/kmod-static-nodes.service
        /sys/fs/cgroup/memory/system.slice/kmod-static-nodes.service
    /system.slice/docker.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/docker.service
        /sys/fs/cgroup/memory/system.slice/docker.service
        /sys/fs/cgroup/blkio/system.slice/docker.service
    /system.slice/-.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/-.mount
        /sys/fs/cgroup/memory/system.slice/-.mount
        /sys/fs/cgroup/blkio/system.slice/-.mount
    /system.slice/systemd-journal-flush.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-journal-flush.service
        /sys/fs/cgroup/memory/system.slice/systemd-journal-flush.service
        /sys/fs/cgroup/blkio/system.slice/systemd-journal-flush.service
    /system.slice/systemd-user-sessions.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-user-sessions.service
        /sys/fs/cgroup/memory/system.slice/systemd-user-sessions.service
        /sys/fs/cgroup/blkio/system.slice/systemd-user-sessions.service
    /system.slice/systemd-update-utmp.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-update-utmp.service
        /sys/fs/cgroup/memory/system.slice/systemd-update-utmp.service
        /sys/fs/cgroup/blkio/system.slice/systemd-update-utmp.service
    /system.slice/home.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/home.mount
        /sys/fs/cgroup/memory/system.slice/home.mount
        /sys/fs/cgroup/blkio/system.slice/home.mount
    /system.slice/dracut-shutdown.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dracut-shutdown.service
        /sys/fs/cgroup/memory/system.slice/dracut-shutdown.service
        /sys/fs/cgroup/blkio/system.slice/dracut-shutdown.service
    /system.slice/nfs-server.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/nfs-server.service
        /sys/fs/cgroup/memory/system.slice/nfs-server.service
        /sys/fs/cgroup/blkio/system.slice/nfs-server.service
    /system.slice/dev-disk-by\x2duuid-657dff8a\x2de602\x2d41fb\x2dac66\x2d845c2e038d2f.swap:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dev-disk-by\x2duuid-657dff8a\x2de602\x2d41fb\x2dac66\x2d845c2e038d2f.swap
        /sys/fs/cgroup/memory/system.slice/dev-disk-by\x2duuid-657dff8a\x2de602\x2d41fb\x2dac66\x2d845c2e038d2f.swap
        /sys/fs/cgroup/blkio/system.slice/dev-disk-by\x2duuid-657dff8a\x2de602\x2d41fb\x2dac66\x2d845c2e038d2f.swap
    /system.slice/nfs-config.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/nfs-config.service
        /sys/fs/cgroup/memory/system.slice/nfs-config.service
        /sys/fs/cgroup/blkio/system.slice/nfs-config.service
    /user.slice:
        /sys/fs/cgroup/blkio/user.slice
        /sys/fs/cgroup/cpu,cpuacct/user.slice
        /sys/fs/cgroup/memory/user.slice
    /system.slice/dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS1\x2da0c2c92c6cb04c4a90dd6f8198a9454d\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS1\x2da0c2c92c6cb04c4a90dd6f8198a9454d\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
        /sys/fs/cgroup/memory/system.slice/dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS1\x2da0c2c92c6cb04c4a90dd6f8198a9454d\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
        /sys/fs/cgroup/blkio/system.slice/dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS1\x2da0c2c92c6cb04c4a90dd6f8198a9454d\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
    /system.slice/systemd-udev-settle.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-udev-settle.service
        /sys/fs/cgroup/memory/system.slice/systemd-udev-settle.service
        /sys/fs/cgroup/blkio/system.slice/systemd-udev-settle.service
    /system.slice/rtkit-daemon.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/rtkit-daemon.service
        /sys/fs/cgroup/memory/system.slice/rtkit-daemon.service
        /sys/fs/cgroup/blkio/system.slice/rtkit-daemon.service
    /system.slice/ModemManager.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/ModemManager.service
        /sys/fs/cgroup/memory/system.slice/ModemManager.service
        /sys/fs/cgroup/blkio/system.slice/ModemManager.service
    /system.slice/gssproxy.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/gssproxy.service
        /sys/fs/cgroup/memory/system.slice/gssproxy.service
        /sys/fs/cgroup/blkio/system.slice/gssproxy.service
    /system.slice/dev-hugepages.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dev-hugepages.mount
        /sys/fs/cgroup/memory/system.slice/dev-hugepages.mount
        /sys/fs/cgroup/blkio/system.slice/dev-hugepages.mount
    /system.slice/abrt-ccpp.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/abrt-ccpp.service
        /sys/fs/cgroup/memory/system.slice/abrt-ccpp.service
        /sys/fs/cgroup/blkio/system.slice/abrt-ccpp.service
    /system.slice/var-lib-openshift-openshift.local.volumes-pods-bcb00f50\x2d2b05\x2d11e5\x2da2d4\x2d54ee7527188d-volumes-kubernetes.io\x7esecret-default\x2dtoken\x2da2sfj.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/var-lib-openshift-openshift.local.volumes-pods-bcb00f50\x2d2b05\x2d11e5\x2da2d4\x2d54ee7527188d-volumes-kubernetes.io\x7esecret-default\x2dtoken\x2da2sfj.mount
        /sys/fs/cgroup/memory/system.slice/var-lib-openshift-openshift.local.volumes-pods-bcb00f50\x2d2b05\x2d11e5\x2da2d4\x2d54ee7527188d-volumes-kubernetes.io\x7esecret-default\x2dtoken\x2da2sfj.mount
        /sys/fs/cgroup/blkio/system.slice/var-lib-openshift-openshift.local.volumes-pods-bcb00f50\x2d2b05\x2d11e5\x2da2d4\x2d54ee7527188d-volumes-kubernetes.io\x7esecret-default\x2dtoken\x2da2sfj.mount
    /system.slice/proc-fs-nfsd.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/proc-fs-nfsd.mount
        /sys/fs/cgroup/memory/system.slice/proc-fs-nfsd.mount
        /sys/fs/cgroup/blkio/system.slice/proc-fs-nfsd.mount
    /system.slice/plexmediaserver.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/plexmediaserver.service
        /sys/fs/cgroup/memory/system.slice/plexmediaserver.service
        /sys/fs/cgroup/blkio/system.slice/plexmediaserver.service
    /system.slice/rpc-statd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/rpc-statd.service
        /sys/fs/cgroup/memory/system.slice/rpc-statd.service
        /sys/fs/cgroup/blkio/system.slice/rpc-statd.service
    /system.slice/irqbalance.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/irqbalance.service
        /sys/fs/cgroup/memory/system.slice/irqbalance.service
        /sys/fs/cgroup/blkio/system.slice/irqbalance.service
    /system.slice/abrt-oops.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/abrt-oops.service
        /sys/fs/cgroup/memory/system.slice/abrt-oops.service
        /sys/fs/cgroup/blkio/system.slice/abrt-oops.service
    /system.slice/alsa-state.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/alsa-state.service
        /sys/fs/cgroup/memory/system.slice/alsa-state.service
        /sys/fs/cgroup/blkio/system.slice/alsa-state.service
    /system.slice/iscsi-shutdown.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/iscsi-shutdown.service
        /sys/fs/cgroup/memory/system.slice/iscsi-shutdown.service
        /sys/fs/cgroup/blkio/system.slice/iscsi-shutdown.service
    /system.slice/boot.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/boot.mount
        /sys/fs/cgroup/memory/system.slice/boot.mount
        /sys/fs/cgroup/blkio/system.slice/boot.mount
    /system.slice/sysstat.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/sysstat.service
        /sys/fs/cgroup/memory/system.slice/sysstat.service
        /sys/fs/cgroup/blkio/system.slice/sysstat.service
    /system.slice/upower.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/upower.service
        /sys/fs/cgroup/memory/system.slice/upower.service
        /sys/fs/cgroup/blkio/system.slice/upower.service
    /system.slice/system-lvm2\x2dpvscan.slice:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/system-lvm2\x2dpvscan.slice
        /sys/fs/cgroup/memory/system.slice/system-lvm2\x2dpvscan.slice
        /sys/fs/cgroup/blkio/system.slice/system-lvm2\x2dpvscan.slice
    /system.slice/dev-disk-by\x2did-dm\x2dname\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dev-disk-by\x2did-dm\x2dname\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
        /sys/fs/cgroup/memory/system.slice/dev-disk-by\x2did-dm\x2dname\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
        /sys/fs/cgroup/blkio/system.slice/dev-disk-by\x2did-dm\x2dname\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
    /system.slice/akmods-shutdown.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/akmods-shutdown.service
        /sys/fs/cgroup/memory/system.slice/akmods-shutdown.service
        /sys/fs/cgroup/blkio/system.slice/akmods-shutdown.service
    /system.slice/wpa_supplicant.service:
        /sys/fs/cgroup/memory/system.slice/wpa_supplicant.service
        /sys/fs/cgroup/blkio/system.slice/wpa_supplicant.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/wpa_supplicant.service
    /system.slice/sys-kernel-debug.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/sys-kernel-debug.mount
        /sys/fs/cgroup/memory/system.slice/sys-kernel-debug.mount
        /sys/fs/cgroup/blkio/system.slice/sys-kernel-debug.mount
    /system.slice/dev-mqueue.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dev-mqueue.mount
        /sys/fs/cgroup/memory/system.slice/dev-mqueue.mount
        /sys/fs/cgroup/blkio/system.slice/dev-mqueue.mount
    /kubelet:
        /sys/fs/cgroup/cpuset/kubelet
        /sys/fs/cgroup/memory/kubelet
        /sys/fs/cgroup/blkio/kubelet
        /sys/fs/cgroup/cpu,cpuacct/kubelet
    /system.slice/tmp.mount:
        /sys/fs/cgroup/blkio/system.slice/tmp.mount
        /sys/fs/cgroup/cpu,cpuacct/system.slice/tmp.mount
        /sys/fs/cgroup/memory/system.slice/tmp.mount
    /system.slice/dev-mapper-luks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dev-mapper-luks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
        /sys/fs/cgroup/memory/system.slice/dev-mapper-luks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
        /sys/fs/cgroup/blkio/system.slice/dev-mapper-luks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
    /machine.slice:
        /sys/fs/cgroup/cpu,cpuacct/machine.slice
        /sys/fs/cgroup/memory/machine.slice
        /sys/fs/cgroup/blkio/machine.slice
    /system.slice/system-getty.slice:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/system-getty.slice
        /sys/fs/cgroup/memory/system.slice/system-getty.slice
        /sys/fs/cgroup/blkio/system.slice/system-getty.slice
    /system.slice/fedora-readonly.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/fedora-readonly.service
        /sys/fs/cgroup/memory/system.slice/fedora-readonly.service
        /sys/fs/cgroup/blkio/system.slice/fedora-readonly.service
    /system.slice/jexec.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/jexec.service
        /sys/fs/cgroup/memory/system.slice/jexec.service
        /sys/fs/cgroup/blkio/system.slice/jexec.service
    /system.slice/lvm2-monitor.service:
        /sys/fs/cgroup/memory/system.slice/lvm2-monitor.service
        /sys/fs/cgroup/blkio/system.slice/lvm2-monitor.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/lvm2-monitor.service
    /system.slice/systemd-fsck-root.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-fsck-root.service
        /sys/fs/cgroup/memory/system.slice/systemd-fsck-root.service
        /sys/fs/cgroup/blkio/system.slice/systemd-fsck-root.service
    /system.slice/docker-5b881011f3e7efdae20491f386f50d4eac7355296ffcb5146a2b4d3c936a9430.scope:
        /sys/fs/cgroup/cpuset/system.slice/docker-5b881011f3e7efdae20491f386f50d4eac7355296ffcb5146a2b4d3c936a9430.scope
        /sys/fs/cgroup/memory/system.slice/docker-5b881011f3e7efdae20491f386f50d4eac7355296ffcb5146a2b4d3c936a9430.scope
        /sys/fs/cgroup/blkio/system.slice/docker-5b881011f3e7efdae20491f386f50d4eac7355296ffcb5146a2b4d3c936a9430.scope
        /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-5b881011f3e7efdae20491f386f50d4eac7355296ffcb5146a2b4d3c936a9430.scope
    /system.slice/run-user-1000.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/run-user-1000.mount
        /sys/fs/cgroup/memory/system.slice/run-user-1000.mount
        /sys/fs/cgroup/blkio/system.slice/run-user-1000.mount
    /system.slice/NetworkManager.service:
        /sys/fs/cgroup/memory/system.slice/NetworkManager.service
        /sys/fs/cgroup/blkio/system.slice/NetworkManager.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/NetworkManager.service
    /system.slice/nfs-idmapd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/nfs-idmapd.service
        /sys/fs/cgroup/memory/system.slice/nfs-idmapd.service
        /sys/fs/cgroup/blkio/system.slice/nfs-idmapd.service
    /system.slice/mcelog.service:
        /sys/fs/cgroup/blkio/system.slice/mcelog.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/mcelog.service
        /sys/fs/cgroup/memory/system.slice/mcelog.service
    /system.slice/system-systemd\x2dcryptsetup.slice:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/system-systemd\x2dcryptsetup.slice
        /sys/fs/cgroup/memory/system.slice/system-systemd\x2dcryptsetup.slice
        /sys/fs/cgroup/blkio/system.slice/system-systemd\x2dcryptsetup.slice
    /system.slice:
        /sys/fs/cgroup/cpu,cpuacct/system.slice
        /sys/fs/cgroup/cpuset/system.slice
        /sys/fs/cgroup/memory/system.slice
        /sys/fs/cgroup/blkio/system.slice
    /system.slice/abrtd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/abrtd.service
        /sys/fs/cgroup/memory/system.slice/abrtd.service
        /sys/fs/cgroup/blkio/system.slice/abrtd.service
    /system.slice/rngd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/rngd.service
        /sys/fs/cgroup/memory/system.slice/rngd.service
        /sys/fs/cgroup/blkio/system.slice/rngd.service
    /system.slice/livesys-late.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/livesys-late.service
        /sys/fs/cgroup/memory/system.slice/livesys-late.service
        /sys/fs/cgroup/blkio/system.slice/livesys-late.service
    /system.slice/systemd-journald.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-journald.service
        /sys/fs/cgroup/memory/system.slice/systemd-journald.service
        /sys/fs/cgroup/blkio/system.slice/systemd-journald.service
    /system.slice/docker-b2753c8b050cf66cfe3c1a21b28c15da4df85046c5500444447b1f72f86ab477.scope:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-b2753c8b050cf66cfe3c1a21b28c15da4df85046c5500444447b1f72f86ab477.scope
        /sys/fs/cgroup/cpuset/system.slice/docker-b2753c8b050cf66cfe3c1a21b28c15da4df85046c5500444447b1f72f86ab477.scope
        /sys/fs/cgroup/memory/system.slice/docker-b2753c8b050cf66cfe3c1a21b28c15da4df85046c5500444447b1f72f86ab477.scope
        /sys/fs/cgroup/blkio/system.slice/docker-b2753c8b050cf66cfe3c1a21b28c15da4df85046c5500444447b1f72f86ab477.scope
    /system.slice/libvirtd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/libvirtd.service
        /sys/fs/cgroup/memory/system.slice/libvirtd.service
        /sys/fs/cgroup/blkio/system.slice/libvirtd.service
    /system.slice/systemd-modules-load.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-modules-load.service
        /sys/fs/cgroup/memory/system.slice/systemd-modules-load.service
        /sys/fs/cgroup/blkio/system.slice/systemd-modules-load.service
    /docker-daemon:
        /sys/fs/cgroup/memory/docker-daemon
        /sys/fs/cgroup/blkio/docker-daemon
        /sys/fs/cgroup/cpu,cpuacct/docker-daemon
        /sys/fs/cgroup/cpuset/docker-daemon
    /system.slice/systemd-udev-trigger.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-udev-trigger.service
        /sys/fs/cgroup/memory/system.slice/systemd-udev-trigger.service
        /sys/fs/cgroup/blkio/system.slice/systemd-udev-trigger.service
    /system.slice/systemd-logind.service:
        /sys/fs/cgroup/memory/system.slice/systemd-logind.service
        /sys/fs/cgroup/blkio/system.slice/systemd-logind.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-logind.service
    /system.slice/systemd-sysctl.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-sysctl.service
        /sys/fs/cgroup/memory/system.slice/systemd-sysctl.service
        /sys/fs/cgroup/blkio/system.slice/systemd-sysctl.service
    /system.slice/fedora-import-state.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/fedora-import-state.service
        /sys/fs/cgroup/memory/system.slice/fedora-import-state.service
        /sys/fs/cgroup/blkio/system.slice/fedora-import-state.service
    /system.slice/proc-sys-fs-binfmt_misc.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/proc-sys-fs-binfmt_misc.mount
        /sys/fs/cgroup/memory/system.slice/proc-sys-fs-binfmt_misc.mount
        /sys/fs/cgroup/blkio/system.slice/proc-sys-fs-binfmt_misc.mount
    /system.slice/livesys.service:
        /sys/fs/cgroup/blkio/system.slice/livesys.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/livesys.service
        /sys/fs/cgroup/memory/system.slice/livesys.service
    /system.slice/acpid.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/acpid.service
        /sys/fs/cgroup/memory/system.slice/acpid.service
        /sys/fs/cgroup/blkio/system.slice/acpid.service
    /system.slice/atd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/atd.service
        /sys/fs/cgroup/memory/system.slice/atd.service
        /sys/fs/cgroup/blkio/system.slice/atd.service
    /system.slice/rsyslog.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/rsyslog.service
        /sys/fs/cgroup/memory/system.slice/rsyslog.service
        /sys/fs/cgroup/blkio/system.slice/rsyslog.service
    /system.slice/dbus.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/dbus.service
        /sys/fs/cgroup/memory/system.slice/dbus.service
        /sys/fs/cgroup/blkio/system.slice/dbus.service
    /system.slice/bluetooth.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/bluetooth.service
        /sys/fs/cgroup/memory/system.slice/bluetooth.service
        /sys/fs/cgroup/blkio/system.slice/bluetooth.service
    /system.slice/accounts-daemon.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/accounts-daemon.service
        /sys/fs/cgroup/memory/system.slice/accounts-daemon.service
        /sys/fs/cgroup/blkio/system.slice/accounts-daemon.service
    /system.slice/systemd-tmpfiles-setup.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-tmpfiles-setup.service
        /sys/fs/cgroup/memory/system.slice/systemd-tmpfiles-setup.service
        /sys/fs/cgroup/blkio/system.slice/systemd-tmpfiles-setup.service
    /system.slice/akmods.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/akmods.service
        /sys/fs/cgroup/memory/system.slice/akmods.service
        /sys/fs/cgroup/blkio/system.slice/akmods.service
    /system.slice/crond.service:
        /sys/fs/cgroup/memory/system.slice/crond.service
        /sys/fs/cgroup/blkio/system.slice/crond.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/crond.service
    /system.slice/fprintd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/fprintd.service
        /sys/fs/cgroup/memory/system.slice/fprintd.service
        /sys/fs/cgroup/blkio/system.slice/fprintd.service
    /system.slice/squid.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/squid.service
        /sys/fs/cgroup/memory/system.slice/squid.service
        /sys/fs/cgroup/blkio/system.slice/squid.service
    /system.slice/lightdm.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/lightdm.service
        /sys/fs/cgroup/memory/system.slice/lightdm.service
        /sys/fs/cgroup/blkio/system.slice/lightdm.service
    /system.slice/polkit.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/polkit.service
        /sys/fs/cgroup/memory/system.slice/polkit.service
        /sys/fs/cgroup/blkio/system.slice/polkit.service
    /system.slice/smartd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/smartd.service
        /sys/fs/cgroup/memory/system.slice/smartd.service
        /sys/fs/cgroup/blkio/system.slice/smartd.service
    /system.slice/firewalld.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/firewalld.service
        /sys/fs/cgroup/memory/system.slice/firewalld.service
        /sys/fs/cgroup/blkio/system.slice/firewalld.service
    /system.slice/auditd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/auditd.service
        /sys/fs/cgroup/memory/system.slice/auditd.service
        /sys/fs/cgroup/blkio/system.slice/auditd.service
    /system.slice/chronyd.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/chronyd.service
        /sys/fs/cgroup/memory/system.slice/chronyd.service
        /sys/fs/cgroup/blkio/system.slice/chronyd.service
    /system.slice/systemd-random-seed.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-random-seed.service
        /sys/fs/cgroup/memory/system.slice/systemd-random-seed.service
        /sys/fs/cgroup/blkio/system.slice/systemd-random-seed.service
    /system.slice/systemd-vconsole-setup.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-vconsole-setup.service
        /sys/fs/cgroup/memory/system.slice/systemd-vconsole-setup.service
        /sys/fs/cgroup/blkio/system.slice/systemd-vconsole-setup.service
    /system.slice/rpcbind.service:
        /sys/fs/cgroup/blkio/system.slice/rpcbind.service
        /sys/fs/cgroup/cpu,cpuacct/system.slice/rpcbind.service
        /sys/fs/cgroup/memory/system.slice/rpcbind.service
    /system.slice/cups.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/cups.service
        /sys/fs/cgroup/memory/system.slice/cups.service
        /sys/fs/cgroup/blkio/system.slice/cups.service
    /system.slice/sys-kernel-config.mount:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/sys-kernel-config.mount
        /sys/fs/cgroup/memory/system.slice/sys-kernel-config.mount
        /sys/fs/cgroup/blkio/system.slice/sys-kernel-config.mount
    /system.slice/systemd-tmpfiles-setup-dev.service:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/systemd-tmpfiles-setup-dev.service
        /sys/fs/cgroup/memory/system.slice/systemd-tmpfiles-setup-dev.service
        /sys/fs/cgroup/blkio/system.slice/systemd-tmpfiles-setup-dev.service
    /system.slice/system-systemd\x2dbacklight.slice:
        /sys/fs/cgroup/cpu,cpuacct/system.slice/system-systemd\x2dbacklight.slice
        /sys/fs/cgroup/memory/system.slice/system-systemd\x2dbacklight.slice
        /sys/fs/cgroup/blkio/system.slice/system-systemd\x2dbacklight.slice

Managed containers: 
    /system.slice/upower.service
    /system.slice/proc-fs-nfsd.mount
    /system.slice/system-systemd\x2dbacklight.slice
    /kubelet
    /system.slice/boot.mount
    /system.slice/gssproxy.service
    /system.slice/docker-b2753c8b050cf66cfe3c1a21b28c15da4df85046c5500444447b1f72f86ab477.scope
        Namespace: docker
        Aliases:
            k8s_grafana2.52da1e9_grafana2-opuv3_default_bcb00f50-2b05-11e5-a2d4-54ee7527188d_8454404f
            b2753c8b050cf66cfe3c1a21b28c15da4df85046c5500444447b1f72f86ab477
    /system.slice/systemd-udevd.service
    /system.slice/systemd-update-utmp.service
    /system.slice/sysstat.service
    /system.slice/ModemManager.service
    /system.slice/system-systemd\x2drfkill.slice
    /system.slice/system-lvm2\x2dpvscan.slice
    /system.slice/dracut-shutdown.service
    /system.slice/run-user-995.mount
    /system.slice/systemd-logind.service
    /system.slice/livesys.service
    /system.slice/run-user-1000.mount
    /system.slice/kmod-static-nodes.service
    /system.slice/jexec.service
    /system.slice/acpid.service
    /docker-daemon
    /system.slice/avahi-daemon.service
    /system.slice/fedora-readonly.service
    /system.slice/wpa_supplicant.service
    /system.slice/nfs-idmapd.service
    /system.slice/rsyslog.service
    /system.slice/atd.service
    /system.slice/systemd-sysctl.service
    /system.slice/rpc-statd.service
    /system.slice/dev-disk-by\x2did-dm\x2dname\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
    /system.slice/systemd-modules-load.service
    /system.slice/abrtd.service
    /system.slice/sys-kernel-config.mount
    /system.slice/colord.service
    /system.slice/rtkit-daemon.service
    /system.slice/proc-sys-fs-binfmt_misc.mount
    /system.slice/abrt-oops.service
    /system.slice/squid.service
    /system.slice/dev-mapper-luks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
    /system.slice/systemd-tmpfiles-setup-dev.service
    /system.slice/system-getty.slice
    /system.slice/dev-hugepages.mount
    /system.slice/systemd-user-sessions.service
    /system.slice/systemd-fsck-root.service
    /system.slice
    /system.slice/dev-disk-by\x2duuid-657dff8a\x2de602\x2d41fb\x2dac66\x2d845c2e038d2f.swap
    /system.slice/dev-dm\x2d3.swap
    /system.slice/lvm2-monitor.service
    /system.slice/rpcbind.service
    /system.slice/tmp.mount
    /system.slice/crond.service
    /system.slice/systemd-vconsole-setup.service
    /system.slice/systemd-tmpfiles-setup.service
    /system.slice/nfs-mountd.service
    /system.slice/firewalld.service
    /system.slice/var-lib-openshift-openshift.local.volumes-pods-bcb00f50\x2d2b05\x2d11e5\x2da2d4\x2d54ee7527188d-volumes-kubernetes.io\x7esecret-default\x2dtoken\x2da2sfj.mount
    /system.slice/abrt-xorg.service
    /system.slice/fprintd.service
    /system.slice/polkit.service
    /system.slice/alsa-state.service
    /system.slice/systemd-remount-fs.service
    /system.slice/var-lib-nfs-rpc_pipefs.mount
    /user.slice
    /system.slice/docker-5b881011f3e7efdae20491f386f50d4eac7355296ffcb5146a2b4d3c936a9430.scope
        Namespace: docker
        Aliases:
            k8s_POD.cad1e704_grafana2-opuv3_default_bcb00f50-2b05-11e5-a2d4-54ee7527188d_1649edfd
            5b881011f3e7efdae20491f386f50d4eac7355296ffcb5146a2b4d3c936a9430
    /system.slice/livesys-late.service
    /system.slice/NetworkManager.service
    /system.slice/home.mount
    /system.slice/system-systemd\x2dfsck.slice
    /system.slice/netcf-transaction.service
    /system.slice/auditd.service
    /system.slice/akmods.service
    /machine.slice
    /system.slice/systemd-udev-settle.service
    /system.slice/rngd.service
    /system.slice/systemd-udev-trigger.service
    /system.slice/sys-kernel-debug.mount
    /system.slice/dev-disk-by\x2did-dm\x2duuid\x2dCRYPT\x2dLUKS1\x2da0c2c92c6cb04c4a90dd6f8198a9454d\x2dluks\x2da0c2c92c\x2d6cb0\x2d4c4a\x2d90dd\x2d6f8198a9454d.swap
    /system.slice/chronyd.service
    /system.slice/dbus.service
    /system.slice/systemd-journal-flush.service
    /system.slice/akmods-shutdown.service
    /system.slice/nfs-server.service
    /system.slice/systemd-random-seed.service
    /system.slice/plexmediaserver.service
    /system.slice/lvm2-lvmetad.service
    /system.slice/systemd-journald.service
    /system.slice/libvirtd.service
    /system.slice/nfs-config.service
    /system.slice/irqbalance.service
    /
    /system.slice/dev-mqueue.mount
    /system.slice/system-systemd\x2dcryptsetup.slice
    /system.slice/lightdm.service
    /system.slice/cups.service
    /system.slice/bluetooth.service
    /system.slice/abrt-ccpp.service
    /system.slice/docker.service
    /system.slice/fedora-import-state.service
    /system.slice/smartd.service
    /system.slice/-.mount
    /system.slice/accounts-daemon.service
    /system.slice/iscsi-shutdown.service
    /system.slice/mcelog.service
vmarmol commented 9 years ago

It looks like only sda supports the cfq diskio scheduler which is the only thing we can get stats for (not sure how device mapper comes into play, but I suspect no stats for that). Since this is Kubernetes, the network namespace (and devices) are owned by a different container so only that one will show the stats. Is that the issue?

jimmidyson commented 9 years ago

Using docker stats I can see disk I/O, I believe via libcontainer. Not sure if this is different to how cadvisor gathers its metrics?

Even the infrastructure container in the pod shows no network traffic either - not sure why that is either.

jimmidyson commented 9 years ago

This appears to only be a problem on Fedora 22 - I've tested it with Kubernetes on GKE & OpenShift 3 on RHEL & network stats are correctly reported on the infrastructure containers in pods.

I wonder if there's some change introduced in fedora 22 that means network stats aren't reported in the same way? Could you point me to where I can look on the filesystem to see where cdvisor looks?

ZeroGraviti commented 9 years ago

I am seeing the same behavior (no blkio and zero values for network metrics) on cAdvisor when it is running in a docker container. Here are the top 10 lines (or so) of my /validate :

cAdvisor version: 0.16.0

OS version: Buildroot 2014.02

Kernel version: [Supported and recommended]
    Kernel version is 4.0.5-boot2docker. Versions >= 2.6 are supported. 3.0+ are recommended.

Cgroup setup: [Supported and recommended]
    Available cgroups: map[cpu:1 net_cls:1 perf_event:1 net_prio:1 cpuset:1 blkio:1 memory:1 devices:1 freezer:1 cpuacct:1]
    Following cgroups are required: [cpu cpuacct]
    Following other cgroups are recommended: [memory blkio cpuset devices freezer]
    Hierarchical memory accounting disabled. Memory usage does not include usage from child containers.

Please let me know if you want the full output ? To confirm I ran docker stats and --no-stream=true, with the following results which at least for network i/o show zero values for each of the containers. Sample output for one of the containers is :

CONTAINER           CPU %               MEM USAGE/LIMIT     MEM %          NET I/O
foobar1             1.54%               87.71 MB/1.044 GB   8.40%          0 B/0 B
lukaf commented 9 years ago

Hi,

Same results (no network statistics from containers) on Ubuntu 14.04 LTS, Docker 1.7.1 and cAdvisor 0.16.0. Statistics are shown using docker stats but not exported by cAdvisor.

pensive_thompson is a nginx container:

CONTAINER           CPU %               MEM USAGE/LIMIT     MEM %               NET I/O
pensive_thompson    0.03%               1.389 MB/1.041 GB   0.13%               267.7 kB/578.3 kB
$ curl localhost:8080/metrics -s | awk '/pensive_thompson/ && /network/'
container_network_receive_bytes_total{id="/docker/b29db2ffebdf165730b3585085caa459907cda32ffb02a0f2d2e2e86255a6bd7",name="pensive_thompson"} 0
container_network_receive_errors_total{id="/docker/b29db2ffebdf165730b3585085caa459907cda32ffb02a0f2d2e2e86255a6bd7",name="pensive_thompson"} 0
container_network_receive_packets_dropped_total{id="/docker/b29db2ffebdf165730b3585085caa459907cda32ffb02a0f2d2e2e86255a6bd7",name="pensive_thompson"} 0
container_network_receive_packets_total{id="/docker/b29db2ffebdf165730b3585085caa459907cda32ffb02a0f2d2e2e86255a6bd7",name="pensive_thompson"} 0
container_network_transmit_bytes_total{id="/docker/b29db2ffebdf165730b3585085caa459907cda32ffb02a0f2d2e2e86255a6bd7",name="pensive_thompson"} 0
container_network_transmit_errors_total{id="/docker/b29db2ffebdf165730b3585085caa459907cda32ffb02a0f2d2e2e86255a6bd7",name="pensive_thompson"} 0
container_network_transmit_packets_dropped_total{id="/docker/b29db2ffebdf165730b3585085caa459907cda32ffb02a0f2d2e2e86255a6bd7",name="pensive_thompson"} 0
container_network_transmit_packets_total{id="/docker/b29db2ffebdf165730b3585085caa459907cda32ffb02a0f2d2e2e86255a6bd7",name="pensive_thompson"} 0
jimmidyson commented 9 years ago

@vmarmol As this doesn't seem to be an isolated incident, is there anything more I can provide to help you identify the issue with this?

jimmidyson commented 9 years ago

@lukaf What distro is that running on?

@vmarmol Still seeing this problem now with docker 1.8.1 on fedora 22. Anything else I can do to help identify the issue?

jimmidyson commented 9 years ago

Looking through the code, network stats require the state.json file for each container to be readable. In my install I can't find that file for each container, but I assume that is because I'm using devicemapper with thin LVM for docker rather than loopback devicemapper or aufs? I have no idea how to be able to use this to get the state.json file from the docker-meta logical volume where I assume it is stored? Any idea?

jimmidyson commented 9 years ago

So more investigation... looks like this network stats are broken with Docker >= 1.7 since the introduction of libnetwork. I've tested this with vanilla binaries on Ubuntu to make sure it's not a Fedora specific thing.

@vmarmol What is the oldest version of Docker that cadvisor has to support? Does cadvisor currently change behaviour between versions? If not, what do you think about adding this in to use libnetwork similar to https://github.com/docker/docker/blob/master/daemon/stats.go#L77-L118 from Docker?

rjnagal commented 9 years ago

I think we can try to get stats from libnetwork if reading the state file fails. @jimmidyson would you like to try a PR for that? Thanks!

jimmidyson commented 9 years ago

@rjnagal I've had a brief look at that today... not as easy as it looks as you need to get the same libnetwork config as the docker daemon which I can't get a handle on. Will look again tomorrow.

jimmidyson commented 9 years ago

@rjnagal One problem of using libnetwork is it has a dependency to later version of docker when docker network plugins were introduced. I'm not quite sure how to proceed without updating the docker dependency which I'm sure you don't want to do. Any thoughts?

vishh commented 9 years ago

May be we can clone old libcontainer code for network stats in cadvisor and use that for older docker versions?

On Mon, Aug 24, 2015 at 11:47 AM, Jimmi Dyson notifications@github.com wrote:

@rjnagal https://github.com/rjnagal One problem of using libnetwork is it has a dependency to later version of docker when docker network plugins were introduced. I'm not quite sure how to proceed without updating the docker dependency which I'm sure you don't want to do. Any thoughts?

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/822#issuecomment-134335089.

lukaf commented 9 years ago

While testing 0.16.0.1 release which should include this changes, I've noticed network metrics collection works only if cadvisor is running outside container.

On Ubuntu 14.04.03 LTS with Docker 1.8.1. Downloaded 0.16.0.1 release and built using:

$ godep go build -a

Packaged in an image with Dockerfile:

FROM debian:8
COPY cadvisor-0.16.0.1 /cadvisor
ENTRYPOINT ["/cadvisor"]

Build and run:

$ docker build -t test/cadvisor .
$ docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor test/cadvisor

Metrics for network are empty:

$ curl localhost:8080/metrics -s | awk '/nginx/ && /network/'
container_network_receive_bytes_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_receive_errors_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_receive_packets_dropped_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_receive_packets_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_transmit_bytes_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_transmit_errors_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_transmit_packets_dropped_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_transmit_packets_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0

But if cadvisor is ran outside the container:

$ sudo ./cadvisor-0.16.0.1 &
[1] 11325
$ curl localhost:8080/metrics -s | awk '/nginx/ && /network/'
container_network_receive_bytes_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 752392
container_network_receive_errors_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_receive_packets_dropped_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_receive_packets_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 3204
container_network_transmit_bytes_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 353296
container_network_transmit_errors_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_transmit_packets_dropped_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 0
container_network_transmit_packets_total{id="/docker/2a0706ff57a311c3056d55ef0f9fc70f0526a3b5e8a2f0b5840f620553b47e0e",name="nginx"} 4538

Am I missing something?

Thanks!

jimmidyson commented 9 years ago

@lukaf Need to update documentation. When running inside a container it has to run inside the pid namespace of the host (--pid=host). So try:

docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --pid=host --name=cadvisor test/cadvisor
smarterclayton commented 9 years ago

If that's truly required that's going to be a problem for containerizing the Kubelet - do we want the kubelet in the parent PID namespace? Can you provide more info about why that is necessary (or tell me to read upthread if I just missed it)?

On Fri, Aug 28, 2015 at 11:08 AM, Jimmi Dyson notifications@github.com wrote:

@lukaf https://github.com/lukaf Need to update documentation. When running inside a container it has to run inside the pid namespace of the host (--pid=host). So try:

docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --pid=host --name=cadvisor test/cadvisor

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/822#issuecomment-135783362.

Clayton Coleman | Lead Engineer, OpenShift

jimmidyson commented 9 years ago

In Docker 1.7, libnetwork was introduced & changed how networking was done. This meant that all network stats from cadvisor for >= 1.7 were empty. The Docker recommended way to gather high resolution metrics is to setns into the container net ns to gather stats. This requires access to the host's pid ns.

I'm not sure it's too much of a bad thing - discussed with @liggitt & @pmorie, albeit briefly. The only other option that I know of is to mount /proc into the container & that is likely to screw up all kinds of stuff in the container (if it's even possible - doubt it is tbh).

If there are any other options, happy to hear them & implement differently if better.

lukaf commented 9 years ago

@jimmidyson That worked, thanks!

jimmidyson commented 9 years ago

@lukaf Cool! We might revisit this for a 0.16.1 release if an alternative approach comes along, but it will keep working no matter what now :)

rjnagal commented 9 years ago

That's the one reason I slightly liked reading network stats from /proc//root/sys/class/net/... We don't have to drop netns for cadvisor.

But if we do decide to drop netns, we can also start getting per-container cpuload from netlink. I'll let vishh decide.

On Fri, Aug 28, 2015 at 7:55 AM, Jimmi Dyson notifications@github.com wrote:

@lukaf https://github.com/lukaf Cool! We might revisit this for a 0.16.1 release if an alternative approach comes along, but it will keep working no matter what now :)

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/822#issuecomment-135796946.

jimmidyson commented 9 years ago

/proc/<pid>/root/sys/class/net/ won't be available unless we're running host pid namespace anyway so it's no better/safer when running in a container. Mounting proc will screw the cadvisor container up, if it's even possible. Could mount host /proc somewhere else in the container I guess but that doesn't feel nice to do.

What do you mean "via netlink"? Perhaps that's an option but don't know enough about it.

rjnagal commented 9 years ago

We already have root filesystem mapped in as a volume on /rootfs (so we use /rootfs/proc//root/sys/class/net). its used in many places within cadvisor.

The netlink comment isn't relevant for networking stats. There's netlink command to get taskstats which gives us cpuload per container. It only works when cadvisor is running under root netns, so we have it disabled for now.

On Fri, Aug 28, 2015 at 8:19 AM, Jimmi Dyson notifications@github.com wrote:

/proc//root/sys/class/net/ won't be available unless we're running host pid namespace anyway so it's no better/safer when running in a container. Mounting proc will screw the cadvisor container up, if it's even possible. Could mount host /proc somewhere else in the container I guess but that doesn't feel nice to do.

What do you mean "via netlink"? Perhaps that's an option but don't know enough about it.

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/822#issuecomment-135803277.

smarterclayton commented 9 years ago

If we start kube-proxy in a container it'll have to be in host netns - in practice we run containerized kubelet in host netns for simplicity of connection.

On Fri, Aug 28, 2015 at 12:27 PM, Rohit Jnagal notifications@github.com wrote:

We already have root filesystem mapped in as a volume on /rootfs (so we use /rootfs/proc//root/sys/class/net). its used in many places within cadvisor.

The netlink comment isn't relevant for networking stats. There's netlink command to get taskstats which gives us cpuload per container. It only works when cadvisor is running under root netns, so we have it disabled for now.

On Fri, Aug 28, 2015 at 8:19 AM, Jimmi Dyson notifications@github.com wrote:

/proc//root/sys/class/net/ won't be available unless we're running host pid namespace anyway so it's no better/safer when running in a container. Mounting proc will screw the cadvisor container up, if it's even possible. Could mount host /proc somewhere else in the container I guess but that doesn't feel nice to do.

What do you mean "via netlink"? Perhaps that's an option but don't know enough about it.

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/822#issuecomment-135803277.

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/822#issuecomment-135806485.

Clayton Coleman | Lead Engineer, OpenShift

jimmidyson commented 9 years ago

@rjnagal If we're mounting the whole rootfs anyway what does it matter if we're running in pid ns? Privileged & mounting rootfs - pid ns isn't exactly reducing security & makes things more consistent between running natively & running in container by using setns to net ns.

@smarterclayton Right but that still doesn't allow us to switch to other docker containers' net ns. Only way to do that that I know of is to run in host pid ns.

smarterclayton commented 9 years ago

Pid NS guarantees we cleaned child processes - running in host NS will not guarantee that.

On Fri, Aug 28, 2015 at 12:31 PM, Jimmi Dyson notifications@github.com wrote:

@rjnagal https://github.com/rjnagal If we're mounting the whole rootfs anyway what does it matter if we're running in pid ns? Privileged & mounting rootfs - pid ns isn't exactly reducing security & makes things more consistent between running natively & running in container by using setns to net ns.

@smarterclayton https://github.com/smarterclayton Right but that still doesn't allow us to switch to other docker containers' net ns. Only way to do that that I know of is to run in host pid ns.

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/822#issuecomment-135807467.

Clayton Coleman | Lead Engineer, OpenShift

jimmidyson commented 9 years ago

@smarterclayton Beyond the limits of my knowledge there :) Could you explain or point me somewhere to read up?

smarterclayton commented 9 years ago

When you're in a PID namespace you become PID 1 (whatever is docker cmd/entrypoint). If that process dies, the kernel nukes all the children spawned by PID 1. So if the kubelet launches child processes (like exec calls to a network plugin) and the kubelet suddenly dies, when you're in a PID namespace the children die, but outside a PID namespace they just become orphans. It depends on what the kubelet needs to do w.r.t. children, but that's the gist.

On Fri, Aug 28, 2015 at 12:46 PM, Jimmi Dyson notifications@github.com wrote:

@smarterclayton https://github.com/smarterclayton Beyond the limits of my knowledge there :) Could you explain or point me somewhere to read up?

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/822#issuecomment-135811137.

Clayton Coleman | Lead Engineer, OpenShift

jimmidyson commented 9 years ago

That's a good reason to change this then :)

@rjnagal Let me update to use your approach of /proc/... files - will get it done now.

jimmidyson commented 9 years ago

@rjnagal @smarterclayton Switched to /proc approach, works great. Nice idea @rjnagal! PR in - #870.