cAdvisor can't collect the stats info from the containers of system.slice

[QQYES]

Recently,we have a problem through cAdvisor 0.24,using Mesos and Marathon to control docker containers,but we find some containers start in the /system.slice directory and some containers start in the /system.slice /docker ,but containers start in the /system.slice directory the stats always be 0 including CPU and memory.And some in the /docker directory is correct in stats. We want to know why the cAdvisor can not collect the stats info from the containers of system.slice

the code from the restful api http://127.0.0.1:8080/api/v2.1/stats?type=docker&recursive=true&count=2:

/system.slice/var-lib-docker-containers-c5c4b26a111f9502cf30cc563439ff0e9d9f3a50d571671b4527897ab5c5b837-shm.mount":{"spec":{"creation_time":"2016-08-24T04:38:24.623180605Z","aliases":["mesos-725cdbc1-f1e5-47ca-a3a2-814725315516-S10.81e58c5d-46a2-4a15-a644-feadd54c60f3","c5c4b26a111f9502cf30cc563439ff0e9d9f3a50d571671b4527897ab5c5b837"],"namespace":"docker","has_cpu":true,"cpu":{"limit":1024,"max_limit":0},"has_memory":true,"memory":{"limit":9223372036854775807,"reservation":9223372036854775807,"swap_limit":9223372036854775807},"has_custom_metrics":false,"has_network":true,"has_filesystem":true,"has_diskio":true,"image":"10.161.24.239/cadvisor/cadvisor:v0.24.0-alpha1"},"stats":[{"timestamp":"2016-08-24T08:01:48.449914053Z","cpu":{"usage":{"total":0,"per_cpu_usage":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"user":0,"system":0},"cfs":{"periods":0,"throttled_periods":0,"throttled_time":0},"load_average":0},"diskio":{},"memory":{"usage":0,"cache":0,"rss":0,"swap":0,"working_set":0,"failcnt":0,"container_data":{"pgfault":0,"pgmajfault":0},"hierarchical_data":{"pgfault":0,"pgmajfault":0}},"network":{"interfaces":[{"name":"eth0","rx_bytes":4267012,"rx_packets":54416,"rx_errors":0,"rx_dropped":0,"tx_bytes":135328535,"tx_packets":52886,"tx_errors":0,"tx_dropped":0}],"tcp":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0},"tcp6":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0}},"filesystem":{"totalUsageBytes":45056,"baseUsageBytes":0}},{"timestamp":"2016-08-24T08:01:49.727745631Z","cpu":{"usage":{"total":0,"per_cpu_usage":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"user":0,"system":0},"cfs":{"periods":0,"throttled_periods":0,"throttled_time":0},"load_average":0},"cpu_inst":{"usage":{"total":0,"per_cpu_usage":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"user":0,"system":0}},"diskio":{},"memory":{"usage":0,"cache":0,"rss":0,"swap":0,"working_set":0,"failcnt":0,"container_data":{"pgfault":0,"pgmajfault":0},"hierarchical_data":{"pgfault":0,"pgmajfault":0}},"network":{"interfaces":[{"name":"eth0","rx_bytes":4271611,"rx_packets":54475,"rx_errors":0,"rx_dropped":0,"tx_bytes":135517667,"tx_packets":52957,"tx_errors":0,"tx_dropped":0}],"tcp":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0},"tcp6":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0}},"filesystem":{"totalUsageBytes":45056,"baseUsageBytes":0}}]}}

other thing

system: Linux resourcepool-0226-slave01 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

docker version: Docker version 1.12.0, build 8eab29e

[cloverstd]

I got the same problem.

I also have another problem, that I could not get the correct spec.memory.limit value, the value from cadivsor is 9223372036854776000, but my host machine total memory is 8GB

system

centos 7.2.1511 Linux node52 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker version docker version 1.12.1, build 23cf638

cadvisor version 0.23.8

[derekwaynecarr]

Is cadvisor running in a container in your setup?

[QQYES]

@derekwaynecarr yes

[derekwaynecarr]

This should be fixed in

https://github.com/google/cadvisor/releases/tag/v0.24.1

Can you verify?

On Saturday, October 29, 2016, QQYES notifications@github.com wrote:

@derekwaynecarr https://github.com/derekwaynecarr yes

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/cadvisor/issues/1438#issuecomment-257086146, or mute the thread https://github.com/notifications/unsubscribe-auth/AF8dbDjeZ_6-k7qAFXs9wEncM-lBsGeOks5q4yzNgaJpZM4Jry9F .

[QQYES]

@derekwaynecarr hello,the result of the v0.24.1 is the same to v0.24.0, I downloaded the image from the https://github.com/google/cadvisor/releases/tag/v0.24.1,and it does't work too.

the API returns: /system.slice/cbss-app-apps-docker-containers-5a2735a98e0bce4f62cdfe676d891813d25e530b2ccca14722e7f4e3e186924e-shm.mount":{"spec":{"creation_time":"2016-10-11T10:28:21.501636778Z","aliases":["mesos-75185ff8-8829-4af7-b4b8-0684e99ff143-S24.ff5873d3-58aa-4049-87db-b7fb0612fbde","5a2735a98e0bce4f62cdfe676d891813d25e530b2ccca14722e7f4e3e186924e"],"namespace":"docker","has_cpu":true,"cpu":{"limit":1024,"max_limit":0},"has_memory":true,"memory":{"limit":9223372036854775807,"reservation":9223372036854775807,"swap_limit":9223372036854775807},"has_custom_metrics":false,"has_network":true,"has_filesystem":true,"has_diskio":true,"image":"10.161.24.239/iap_docker/marathon-lb-source:1.3.3"},"stats":[{"timestamp":"2016-11-01T08:14:20.781558968Z","cpu":{"usage":{"total":0,"per_cpu_usage":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"user":0,"system":0},"cfs":{"periods":0,"throttled_periods":0,"throttled_time":0},"load_average":0},"diskio":{},"memory":{"usage":0,"cache":0,"rss":0,"swap":0,"working_set":0,"failcnt":0,"container_data":{"pgfault":0,"pgmajfault":0},"hierarchical_data":{"pgfault":0,"pgmajfault":0}},"network":{"interfaces":[{"name":"enp2s0f0","rx_bytes":846680017736,"rx_packets":935771674,"rx_errors":0,"rx_dropped":935593445,"tx_bytes":222,"tx_packets":3,"tx_errors":0,"tx_dropped":0},{"name":"enp2s0f1","rx_bytes":61754184715629,"rx_packets":171557733135,"rx_errors":0,"rx_dropped":11631,"tx_bytes":32119400044549,"tx_packets":151598538902,"tx_errors":0,"tx_dropped":0},{"name":"enp129s0f0","rx_bytes":0,"rx_packets":0,"rx_errors":0,"rx_dropped":0,"tx_bytes":0,"tx_packets":0,"tx_errors":0,"tx_dropped":0},{"name":"enp129s0f1","rx_bytes":0,"rx_packets":0,"rx_errors":0,"rx_dropped":0,"tx_bytes":0,"tx_packets":0,"tx_errors":0,"tx_dropped":0},{"name":"bond0","rx_bytes":62600018285193,"rx_packets":172491226093,"rx_errors":0,"rx_dropped":935980297,"tx_bytes":32118859739947,"tx_packets":151595794698,"tx_errors":0,"tx_dropped":0},{"name":"bond0.102","rx_bytes":58239182270100,"rx_packets":150156562431,"rx_errors":0,"rx_dropped":452,"tx_bytes":31595298620878,"tx_packets":143661676229,"tx_errors":0,"tx_dropped":0}],"tcp":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0},"tcp6":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0}},"filesystem":{"totalUsageBytes":40960,"baseUsageBytes":0}},{"timestamp":"2016-11-01T08:14:22.278044494Z","cpu":{"usage":{"total":0,"per_cpu_usage":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"user":0,"system":0},"cfs":{"periods":0,"throttled_periods":0,"throttled_time":0},"load_average":0},"cpu_inst":{"usage":{"total":0,"per_cpu_usage":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"user":0,"system":0}},"diskio":{},"memory":{"usage":0,"cache":0,"rss":0,"swap":0,"working_set":0,"failcnt":0,"container_data":{"pgfault":0,"pgmajfault":0},"hierarchical_data":{"pgfault":0,"pgmajfault":0}},"network":{"interfaces":[{"name":"enp2s0f0","rx_bytes":846680019560,"rx_packets":935771702,"rx_errors":0,"rx_dropped":935593473,"tx_bytes":222,"tx_packets":3,"tx_errors":0,"tx_dropped":0},{"name":"enp2s0f1","rx_bytes":61754191264289,"rx_packets":171557754678,"rx_errors":0,"rx_dropped":11631,"tx_bytes":32119402850871,"tx_packets":151598555967,"tx_errors":0,"tx_dropped":0},{"name":"enp129s0f0","rx_bytes":0,"rx_packets":0,"rx_errors":0,"rx_dropped":0,"tx_bytes":0,"tx_packets":0,"tx_errors":0,"tx_dropped":0},{"name":"enp129s0f1","rx_bytes":0,"rx_packets":0,"rx_errors":0,"rx_dropped":0,"tx_bytes":0,"tx_packets":0,"tx_errors":0,"tx_dropped":0},{"name":"bond0","rx_bytes":62600024835677,"rx_packets":172491247664,"rx_errors":0,"rx_dropped":935980325,"tx_bytes":32118862546269,"tx_packets":151595811763,"tx_errors":0,"tx_dropped":0},{"name":"bond0.102","rx_bytes":58239188392980,"rx_packets":150156581587,"rx_errors":0,"rx_dropped":452,"tx_bytes":31595301387666,"tx_packets":143661692655,"tx_errors":0,"tx_dropped":0}],"tcp":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0},"tcp6":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0}},"filesystem":{"totalUsageBytes":40960,"baseUsageBytes":0}}]}}

[derekwaynecarr]

I can investigate, but I had previously verified the fix on RHEL family. I will not be able to look at this again for a few days.

On Tuesday, November 1, 2016, QQYES notifications@github.com wrote:

@derekwaynecarr https://github.com/derekwaynecarr hello,the result of the is the same to v0.24.0, I downloaded the image from the https://github.com/google/cadvisor/releases/tag/v0.24.1,and it does't work too.the API returns: /system.slice/cbss-app-apps-docker-containers- 5a2735a98e0bce4f62cdfe676d891813d25e530b2ccca14722e7f4e3e186 924e-shm.mount":{"spec":{"creation_time":"2016-10-11T10: 28:21.501636778Z","aliases":["mesos-75185ff8-8829-4af7-b4b8- 0684e99ff143-S24.ff5873d3-58aa-4049-87db-b7fb0612fbde"," 5a2735a98e0bce4f62cdfe676d891813d25e530b2ccca14722e7f4e3e186 924e"],"namespace":"docker","has_cpu":true,"cpu":{"limit": 1024,"max_limit":0},"hasmemory":true,"memory":{"limit" :9223372036854775807,"reservation":9223372036854775807,"swap limit":9223372036854775807},"has_custom_metrics":false," has_network":true,"has_filesystem":true,"has_diskio":true,"image":" 10.161.24.239/iap_docker/marathon-lb-source:1.3.3"},"stats":[{"timestamp": "2016-11-01T08:14:20.781558968Z","cpu":{"usage":{" total":0,"per_cpu_usage":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0],"user":0,"system":0},"cfs":{"periods":0, "throttled_periods":0,"throttled_time":0},"load_average":0},"diskio":{}," memory":{"usage":0,"cache":0,"rss":0,"swap":0,"working_set": 0,"failcnt":0,"container_data":{"pgfault":0,"pgmajfault":0}, "hierarchical_data":{"pgfault":0,"pgmajfault":0}},"network": {"interfaces":[{"name":"enp2s0f0","rx_bytes":846680017736,"rx_packets": 935771674,"rx_errors":0,"rx_dropped":935593445,"tx_bytes": 222,"tx_packets":3,"tx_errors":0,"tx_dropped":0},{"name":" enp2s0f1","rx_bytes":61754184715629,"rx_packets": 171557733135,"rx_errors":0,"rx_dropped":11631,"tx_bytes": 32119400044549,"tx_packets":151598538902,"tx_errors":0," tx_dropped":0},{"name":"enp129s0f0","rxbytes":0,"rx packets":0,"rx_errors":0,"rx_dropped":0,"txbytes":0,"tx packets":0,"tx_errors":0,"tx_dropped":0},{"name":" enp129s0f1","rx_bytes":0,"rx_packets":0,"rxerrors":0,"rx dropped":0,"tx_bytes":0,"tx_packets":0,"txerrors":0,"tx dropped":0},{"name":"bond0","rxbytes":62600018285193,"rx packets":172491226093,"rx_errors":0,"rx_dropped":935980297,"tx_bytes": 32118859739947,"tx_packets":151595794698,"tx_errors":0," tx_dropped":0},{"name":"bond0.102","rx_bytes":58239182270100,"rx_packets": 150156562431,"rx_errors":0,"rx_dropped":452,"tx_bytes": 31595298620878,"tx_packets":143661676229,"tx_errors":0," tx_dropped":0}],"tcp":{"Established":0,"SynSent":0," SynRecv":0,"FinWait1":0,"FinWait2":0,"TimeWait":0," Close":0,"CloseWait":0,"LastAck":0,"Listen":0,"Closing":0},"tcp6":{" Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0," FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0," Closing":0}},"filesystem":{"totalUsageBytes":40960,"baseUsageBytes":0}},{" timestamp":"2016-11-01T08:14:22.278044494Z","cpu":{"usage": {"total":0,"per_cpu_usage":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0],"user":0,"system":0},"cfs":{"periods":0, "throttled_periods":0,"throttled_time":0},"load_average":0},"cpu_inst":{" usage":{"total":0,"per_cpuusage":[0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0],"user":0,"system":0}},"diskio": {},"memory":{"usage":0,"cache":0,"rss":0,"swap":0,"working set":0,"failcnt":0,"container_data":{"pgfault":0," pgmajfault":0},"hierarchical_data":{"pgfault":0," pgmajfault":0}},"network":{"interfaces":[{"name":"enp2s0f0","rx_bytes": 846680019560,"rx_packets":935771702,"rxerrors":0,"rx dropped":935593473,"tx_bytes":222,"tx_packets":3,"tx_errors" :0,"tx_dropped":0},{"name":"enp2s0f1","rx_bytes": 61754191264289,"rx_packets":171557754678,"rx_errors":0," rx_dropped":11631,"tx_bytes":32119402850871,"tx_packets": 151598555967,"tx_errors":0,"tx_dropped":0},{"name":" enp129s0f0","rx_bytes":0,"rx_packets":0,"rxerrors":0,"rx dropped":0,"tx_bytes":0,"tx_packets":0,"txerrors":0,"tx dropped":0},{"name":"enp129s0f1","rxbytes":0,"rx packets":0,"rx_errors":0,"rx_dropped":0,"txbytes":0,"tx packets":0,"tx_errors":0,"tx_dropped":0},{"name":"bond0"," rx_bytes":62600024835677,"rxpackets":172491247664,"rx errors":0,"rx_dropped":935980325,"tx_bytes":32118862546269,"tx_packets": 151595811763,"tx_errors":0,"tx_dropped":0},{"name":"bond0.102","rx_bytes": 58239188392980,"rx_packets":150156581587,"rx_errors":0," rx_dropped":452,"tx_bytes":31595301387666,"tx_packets": 143661692655,"tx_errors":0,"tx_dropped":0}],"tcp":{" Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0," FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0," Closing":0},"tcp6":{"Established":0,"SynSent":0,"SynRecv":0,"FinWait1":0," FinWait2":0,"TimeWait":0,"Close":0,"CloseWait":0,"LastAck":0,"Listen":0," Closing":0}},"filesystem":{"totalUsageBytes":40960,"baseUsageBytes":0}}]}}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/cadvisor/issues/1438#issuecomment-257510945, or mute the thread https://github.com/notifications/unsubscribe-auth/AF8dbIghSzNkCxgrch0n56l_J75lhBOSks5q5vUjgaJpZM4Jry9F .

[dneray]

Experiencing the same issue on coreOS 1235.1.0 with docker 1.12.3.

When I run cadvisor outside of a docker container, I can only gather stats for all containers in system.slice, the rest report 0. When I run cadvisor inside a docker container, I can't gather stats fir system.slice, those ones all report 0.

Have tried with v0.24.1

[roasties]

Hi, I'm having exactly the same porblem. Docker 1.12.3 & cAdvisor v0.24.1. Is there any update?

[F30]

I can confirm this for cAdvisor 0.28.3 running natively (not in a Docker container, but as a systemd unit) on Debian Stretch. This happens both when running cAdvisor as root and unprivileged user.

[Arlenmbx]

I'm having the same problem now with docker Version 17.12.0-ce and cadvisor version is release0.27,release0.28 and master branch. I run it both linux ubuntu host and in a container. host info is 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux