the "container_fs_reads_bytes_total" value of some container can not be found

[boat320]

I run the commend like this: /etc/cadvisor/cadvisor -port 8077 -logtostderr -docker_only true -enable_load_reader -privileged=true -docker_root=/data/docker_data -logtostderr

my cadvisor version is cAdvisor version v0.35.0 (49033161)

I want to know how the cadvisor to get "container_fs_reads_bytes_total" value

[dashpole]

Is it only missing for some containers? Can you share the output from curling the prometheus endpoint?

[boat320]

One of the report: container_fs_reads_bytes_total{container_label_com_docker_compose_config_hash="9e17228f1e0ef3f997dc0e05d2df4e917598eac2e3e6bba00e32dcac780c2287",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="rediscluster4",container_label_com_docker_compose_service="redisclustre4",container_label_com_docker_compose_version="1.18.0",device="/dev/sda",id="/docker/faed03aef755274e9d1803f7febd45b66e5778f80bc09f3ef993faf889c7e99e",image="harbor.sdjpay.org/prd/redis:5.0.7",instance="10.30.20.10:8077",job="consul_sd",name="ns3012059"}

[dashpole]

oh, as in it literally doesn't have a value at the end of the line?

[boat320]

what I mean is some containers do not have that metric

[dashpole]

That often indicates a problem getting metadata from the container runtime. Usually, the container exists in the output, but is missing the container name. If you search for the container id, is it missing?

Also, if you can share your cAdvisor log, that would be helpful as well.

[boat320]

the containerID is exist. I can see it in the other metric such as container_fs_io_current.

there is no log. How can I trun on the Debug level log. I run in with the command like that: "/etc/cadvisor/cadvisor -port 8077 -logtostderr -docker_only true -enable_load_reader -privileged=true -docker_root=/data/docker_data -logtostderr -v=3"

[dashpole]

I mean, often the metric will only have the "id" label set, but not "name" or others. Can you attach the output of a full scrape of the /metrics endpoint?

[boat320]

The metrics endpoint output ,you can see the key "container_fs_reads_bytes_total" and "container_fs_writes_bytes_total" lack some container metric.txt

[dashpole]

I looked through the output, and agree this is a bug. That metric could be missing if your kernel doesn't have cfq enabled (https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/fs/blkio.go#L169), but some metrics are present that would be missing if that was the case. I would have to have the cAdvisor log to continue to debug.

If you are manually running it, then I just need stdout from the process. --v=4 might help.

[vpavlin]

HI @dashpole we are seeing similar behaviour - we have this metric collected for some container, but not for others. Would love to provide whatever you need to figure out if this is still issue