search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.12k stars 2.32k forks source link

wrong with container_fs_io #1718

Open ZhiqinYang opened 7 years ago

ZhiqinYang commented 7 years ago

Excuse me, I found that I run cadvisor0.26, container_fs_io_current always is 0, what's wrong with it?

I run with this commad: sudo docker run --volume=/:/rootfs:rw --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --userns=host -it --rm --privileged=true --net=host google/cadvisor:latest -enable_load_reader=true
[root@localhost ~]# docker info

Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 4 Server Version: 17.07.0-ce-rc1 Storage Driver: overlay Backing Filesystem: xfs Supports d_type: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 3addd840653146c90a254301d6c3a663c7fd6429 runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4 init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 3.10.0-514.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.656GiB Name: localhost.localdomain ID: CWT5:M2G7:QSNL:INAW:NS3O:5UPX:CNC4:6X75:V5X3:3HBL:FSOP:LN65 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

cadvisor logs: 1 oomparser.go:185] oomparser using systemd I0813 03:08:13.856726 1 factory.go:116] Factory "docker" was unable to handle container "/" I0813 03:08:13.856867 1 factory.go:105] Error trying to work out if we can handle /: / not handled by systemd handler I0813 03:08:13.856897 1 factory.go:116] Factory "systemd" was unable to handle container "/" I0813 03:08:13.856944 1 factory.go:112] Using factory "raw" for container "/" I0813 03:08:14.698353 1 reader.go:41] Family id for taskstats: 26 I0813 03:08:14.698465 1 cpuload.go:44] Using a netlink-based load reader I0813 03:08:14.715126 1 manager.go:913] Added container: "/" (aliases: [], namespace: "") I0813 03:08:14.854468 1 handler.go:325] Added event &{/ 2017-08-12 01:36:54.501999999 +0000 UTC containerCreation {}} I0813 03:08:14.976714 1 manager.go:288] Starting recovery of all containers I0813 03:08:15.207218 1 container.go:407] Start housekeeping for container "/" I0813 03:08:21.711022 1 reader.go:78] Task stats for "/var/lib/docker/overlay/de8852bea1acc108031ca461745411ee4f8d8cd5f6f94d17b69d8d93acf3182d/merged/sys/fs/cgroup/cpu,cpuacct": {NrSleeping:97 NrRunning:0 NrStopped:0 NrUninterruptible:0 NrIoWait:0} I0813 03:08:21.744609 1 container.go:422] [/] Housekeeping took 6.537315174s I0813 03:08:21.746943 1 reader.go:78] Task stats for "/var/lib/docker/overlay/de8852bea1acc108031ca461745411ee4f8d8cd5f6f94d17b69d8d93acf3182d/merged/sys/fs/cgroup/cpu,cpuacct": {NrSleeping:95 NrRunning:1 NrStopped:0 NrUninterruptible:0 NrIoWait:0} I0813 03:08:23.699066 1 reader.go:78] Task stats for "/var/lib/docker/overlay/de8852bea1acc108031ca461745411ee4f8d8cd5f6f94d17b69d8d93acf3182d/merged/sys/fs/cgroup/cpu,cpuacct": {NrSleeping:97 NrRunning:0 NrStopped:0 NrUninterruptible:0 NrIoWait:0} I0813 03:08:25.352488 1 reader.go:78] Task stats for "/var/lib/docker/overlay/de8852bea1acc108031ca461745411ee4f8d8cd5f6f94d17b69d8d93acf3182d/merged/sys/fs/cgroup/cpu,cpuacct": {NrSleeping:96 NrRunning:1 NrStopp I0813 03:09:13.623090 1 factory.go:109] Factory "systemd" can handle container "/system.slice/proc-sys-fs-binfmt_misc.mount", but ignoring. I0813 03:09:13.623112 1 manager.go:882] ignoring container "/system.slice/proc-sys-fs-binfmt_misc.mount" I0813 03:09:13.623128 1 factory.go:116] Factory "docker" was unable to handle container "/system.slice/run-docker-netns-default.mount" I0813 03:09:13.623145 1 factory.go:109] Factory "systemd" can handle container "/system.slice/run-docker-netns-default.mount", but ignoring. I0813 03:09:13.623166 1 manager.go:882] ignoring container "/system.slice/run-docker-netns-default.mount" I0813 03:09:13.623182 1 factory.go:116] Factory "docker" was unable to handle container "/system.slice/var-lib-docker-overlay.mount" I0813 03:09:13.623199 1 factory.go:109] Factory "systemd" can handle container "/system.slice/var-lib-docker-overlay.mount", but ignoring. I0813 03:09:13.623220 1 manager.go:882] ignoring container "/system.slice/var-lib-docker-overlay.mount" I0813 03:09:13.623236 1 factory.go:116] Factory "docker" was unable to handle container "/system.slice/dev-mqueue.mount" I0813 03:09:13.623253 1 factory.go:109] Factory "systemd" can handle container "/system.slice/dev-mqueue.mount", but ignoring. I0813 03:09:13.623272 1 manager.go:882] ignoring container "/system.slice/dev-mqueue.mount" I0813 03:09:13.623288 1 factory.go:116] Factory "docker" was unable to handle container "/system.slice/home.mount" I0813 03:09:13.623304 1 factory.go:109] Factory "systemd" can handle container "/system.slice/home.mount", but ignoring. I0813 03:09:13.623324 1 manager.go:882] ignoring container "/system.slice/home.mount"

HELP container_fs_io_current Number of I/Os currently in progress

TYPE container_fs_io_current gauge

container_fs_io_current{device="",id="/"} 0 container_fs_io_current{device="/dev/mapper/cl-home",id="/"} 0 container_fs_io_current{device="/dev/mapper/cl-root",id="/"} 0 container_fs_io_current{device="/dev/sda1",id="/"} 0 container_fs_io_current{device="overlay",id="/"} 0 container_fs_io_current{device="shm",id="/"} 0 container_fs_io_current{device="tmpfs",id="/"} 0

HELP container_fs_io_time_seconds_total Cumulative count of seconds spent doing I/Os

TYPE container_fs_io_time_seconds_total counter

container_fs_io_time_seconds_total{device="",id="/"} 0 container_fs_io_time_seconds_total{device="/dev/mapper/cl-home",id="/"} 0 container_fs_io_time_seconds_total{device="/dev/mapper/cl-root",id="/"} 0 container_fs_io_time_seconds_total{device="/dev/sda1",id="/"} 1.637e-06 container_fs_io_time_seconds_total{device="overlay",id="/"} 0 container_fs_io_time_seconds_total{device="shm",id="/"} 0 container_fs_io_time_seconds_total{device="tmpfs",id="/"} 0

HELP container_fs_io_time_weighted_seconds_total Cumulative weighted I/O time in seconds

TYPE container_fs_io_time_weighted_seconds_total counter

container_fs_io_time_weighted_seconds_total{device="/dev/mapper/cl-home",id="/"} 0 container_fs_io_time_weighted_seconds_total{device="/dev/mapper/cl-root",id="/"} 0 container_fs_io_time_weighted_seconds_total{device="/dev/sda1",id="/"} 2.633e-06 container_fs_io_time_weighted_seconds_total{device="overlay",id="/"} 0 container_fs_io_time_weighted_seconds_total{device="shm",id="/"} 0 container_fs_io_time_weighted_seconds_total{device="tmpfs",id="/"} 0

2017/08/13 03:10:55 {Timestamp:2017-08-13 03:10:44.348810093 +0000 UTC Cpu:{Usage:{Total:0 PerCpu:[0 0] User:0 System:0} CFS:{Periods:0 ThrottledPeriods:0 ThrottledTime:0} LoadAverage:0} DiskIo:{IoServiceBytes:[] IoServiced:[] IoQueued:[] Sectors:[] IoServiceTime:[] IoWaitTime:[] IoMerged:[] IoTime:[]} Memory:{Usage:0 Cache:0 RSS:0 Swap:0 WorkingSet:0 Failcnt:0 ContainerData:{Pgfault:0 Pgmajfault:0} HierarchicalData:{Pgfault:0 Pgmajfault:0}} Network:{InterfaceStats:{Name: RxBytes:0 RxPackets:0 RxErrors:0 RxDropped:0 TxBytes:0 TxPackets:0 TxErrors:0 TxDropped:0} Interfaces:[] Tcp:{Established:0 SynSent:0 SynRecv:0 FinWait1:0 FinWait2:0 TimeWait:0 Close:0 CloseWait:0 LastAck:0 Listen:0 Closing:0} Tcp6:{Established:0 SynSent:0 SynRecv:0 FinWait1:0 FinWait2:0 TimeWait:0 Close:0 CloseWait:0 LastAck:0 Listen:0 Closing:0} Udp:{Listen:0 Dropped:0 RxQueued:0 TxQueued:0} Udp6:{Listen:0 Dropped:0 RxQueued:0 TxQueued:0}} Filesystem:[] TaskStats:{NrSleeping:0 NrRunning:0 NrStopped:0 NrUninterruptible:0 NrIoWait:0} CustomMetrics:map[]} 2017/08/13 03:10:55 {Timestamp:2017-08-13 03:10:04.11215059 +0000 UTC Cpu:{Usage:{Total:52079563795 PerCpu:[4756043034 47323520761] User:150000000 System:3610000000} CFS:{Periods:0 ThrottledPeriods:0 ThrottledTime:0} LoadAverage:0} DiskIo:{IoServiceBytes:[{Device:/dev/sda Major:8 Minor:0 Stats:map[Total:0 Read:0 Write:0 Sync:0 Async:0]} {Device:/dev/dm-0 Major:253 Minor:0 Stats:map[Read:0 Write:0 Sync:0 Async:0 Total:0]}] IoServiced:[{Device:/dev/sda Major:8 Minor:0 Stats:map[Read:0 Write:0 Sync:0 Async:0 Total:0]} {Device:/dev/dm-0 Major:253 Minor:0 Stats:map[Read:0 Write:0 Sync:0 Async:0 Total:0]}] IoQueued:[] Sectors:[] IoServiceTime:[] IoWaitTime:[] IoMerged:[] IoTime:[]} Memory:{Usage:4096 Cache:4096 RSS:0 Swap:0 WorkingSet:0 Failcnt:0 ContainerData:{Pgfault:24 Pgmajfault:0} HierarchicalData:{Pgfault:24 Pgmajfault:0}} Network:{InterfaceStats:{Name: RxBytes:0 RxPackets:0 RxErrors:0 RxDropped:0 TxBytes:0 TxPackets:0 TxErrors:0 TxDropped:0} Interfaces:[] Tcp:{Established:0 SynSent:0 SynRecv:0 FinWait1:0 FinWait2:0 TimeWait:0 Close:0 CloseWait:0 LastAck:0 Listen:0 Closing:0} Tcp6:{Established:0 SynSent:0 SynRecv:0 FinWait1:0 FinWait2:0 TimeWait:0 Close:0 CloseWait:0 LastAck:0 Listen:0 Closing:0} Udp:{Listen:0 Dropped:0 RxQueued:0 TxQueued:0} Udp6:{Listen:0 Dropped:0 RxQueued:0 TxQueued:0}} Filesystem:[] TaskStats:{NrSleeping:1 NrRunning:0 NrStopped:0 NrUninterruptible:0 NrIoWait:0} CustomMetrics:map[]} I realy don't know what to do , any help ?

dashpole commented 7 years ago

Ill try and take a look at this soon

ZhiqinYang commented 7 years ago

thank you! hope hear from you soon! this is troubled for a long time!

Brandonage commented 7 years ago

I have exactly the same problem. Same configuration for the docker run command. Is there any solution for this?

lcfang commented 6 years ago

Hi @dashpole , Do you have some idea about this problem? Maybe I have the similar question.

dashpole commented 6 years ago

I think the original issue was solved here: https://github.com/google/cadvisor/pull/1642#issuecomment-321238965 Can you see if your cgroup mounts are correct? @ZhiqinYang can you tell us what you had to change with your docker run command?

Brandonage commented 6 years ago

@dashpole What do you mean exactly with cgroup mounts are correct?. I do mount cgroup in the container with --volume=/cgroup:/cgroup:ro

dashpole commented 6 years ago

@Brandonage I am not sure. I was hoping @ZhiqinYang would chime in

Maatary commented 4 years ago

Hi all, what is the status of this issue ?

olahouze commented 2 years ago

Hello

Is the situation resolved? Can we close this problem?

Sincerely