/var/lib/lxcfs/proc/diskstats not returning anything

[l00mi]

I am not sure if this is by design, or not yet implemented. But is it intentional that /diskstats is not working at all?

It is used by tools like iotop or psutil.

[stgraber]

I don't think that's intentional. It should definitely filter out things that aren't used by the container, but not to that point :)

@hallyn

[hallyn]

I assume this is because all of the info files in blkio seem to be returning "Total 0", i.e.

''' cat /sys/fs/cgroup/blkio/user.slice/user-1000.slice/session-c2.scope/blkio.io_serviced_recursive Total 0 cat /sys/fs/cgroup/blkio/blkio.io_serviced_recursive Total 0 '''

[l00mi]

Okay I see, just for reference I discovered this after I was updating a base system from Ubuntu 15.10 to 16.04 where a container could not start an instance of MemSQL anymore.

Do you know quick workaround for the time being? I can't obviously not symlink something in place of /proc/diskstat also mount "over" the location seems not to work.

[stgraber]

You should be able to just "umount /proc/diskstat" to get the original file back.

[l00mi]

Works like a charm! Thanks a lot.

[hustcat]

Cgroup blkio only work for direct IO, buffer io will get nothing

[hallyn]

Sorry, I'm not getting it. Can someone explain the problem?

[pavelbrm]

Currently /proc/diskstats inside containers seems to be empty at all. Have tested in following envs:

• host - Ubuntu 16.04.3 with latest updates installed;
• lxd - both stable 2.0.10 and feature 2.20 versions;
• lxc & lxcfs - 2.0.8;
• guests (containers) - Ubuntu 16.04, Ubuntu 14.04, Centos 7;
• container mode - privileged and unprivileged;
• storage pool - dir and zfs;

So at this moments tools like psutil or gopsutil or any other which works with /proc/diskstats gets no information about disks io.

@stgraber

[l00mi]

@exordium-frozen umount /proc/diskstat helps a lot.

[hallyn]

If someone could suggest the best way to change how we are currently filtering diskstats, that would be great.

Otherwise, should we stop providing it?

[l00mi]

I would be happy if currently there is no filtering at all. At least we would have no side effects in some programs. As it is anyway as easy to unmount the lxfuse to get this information this hardly has any security implications.

[pavelbrm]

@l00mi thanks a lot. But why it behaving so? I mean why not show underlying diskstats? In this case how the container should be configured or what should be attached to it to made something appeared in the container's diskstats? In other words, if not unmounted what and when should be present in the file?

Thank you.

[l00mi]

@exordium-frozen that was my question at the beginning .. (-:

[pavelbrm]

@l00mi Oh, yeah, I see. Sorry for this confusion.

[brauner]

Uhm, is this still an issue with the latest version of LXCFS?

[C0rn3j]

Uhm, is this still an issue with the latest version of LXCFS?

Seems to be. Using lxcfs 3.0.3, lxd 3.0.3, this is how the file looks on boot:

# cat /proc/diskstats 
8       0 sda 657 0 5256 116 0 0 0 0 0 66 0

This is after umounting /proc/diskstats:

# grep -v loop /proc/diskstats 
 259       0 nvme0n1 12603612 630129 424451916 2522636 1286590453 42666817 117711515664 64383784 0 1991496 10600532
 259       1 nvme0n1p1 12603532 630129 424447524 2522628 1286590453 42666817 117711515664 64383784 0 1992860 10481312
 259       2 nvme1n1 14662849 701591 497991036 3032136 1287101283 42819316 117663437592 65372720 0 2060604 10467240
 259       3 nvme1n1p1 14662769 701591 497986644 3032128 1287101283 42819316 117663437592 65372720 0 2061912 10466900
 259       4 nvme2n1 12552881 640266 419285580 2573668 1287007763 42797586 117657558560 63872716 0 1887856 10156440
 259       5 nvme2n1p1 12552801 640266 419281188 2573656 1287007763 42797586 117657558560 63872716 0 1888856 10014832
 259       6 nvme3n1 14708315 699006 503076860 3137060 1287457010 42804582 117695447464 65629696 0 2018608 10974236
 259       7 nvme3n1p1 14708235 699006 503072468 3137056 1287457010 42804582 117695447464 65629696 0 2019624 10970240
 259       8 nvme4n1 14601215 694956 502012548 3133284 1286057235 42766147 117681825760 65108220 0 1934116 10862348
 259       9 nvme4n1p1 14601135 694956 502008156 3133268 1286057235 42766147 117681825760 65108220 0 1934964 10857664
 259      10 nvme5n1 12665084 644851 420968172 2604388 1287572593 42647725 117702413936 63247176 0 1801084 9940332
 259      11 nvme5n1p1 12665004 644851 420963780 2604380 1287572593 42647725 117702413936 63247176 0 1801880 9806392
   8       0 sda 110770 10977 10625170 65508 1510338 1180629 35586050 484296 0 249456 549292
   8       1 sda1 591 1518 18488 140 2 0 2 0 0 88 140
   8       2 sda2 93973 9213 10463554 58804 1405157 1168535 34647864 304248 0 80744 362536
   8       3 sda3 16111 246 138768 6456 105179 12094 938184 180048 0 169196 186512
   8      16 sdb 789 0 21864 1120 2 0 4096 0 0 1096 1120
   8      17 sdb1 746 0 19536 1036 2 0 4096 0 0 1020 1036

Which is weird, I would assume the filter would at least leave the nvme0 in, since /var/lib/lxd is on the NVMe and therefore / uses it?

# df -h | grep -v -e tmpfs -e none -e udev
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  4.4T  1.7T  1.9T  48% /
/dev/sda2        30G  2.9G   25G  11% /backup_all

If the device that / is on is being filtered out, it would explain why OP and others don't see any entries in /proc/diskstats at all if they don't have any other drives mounted.

[stgraber]

Yeah, in my case here, I'm getting a completely empty file for /proc/diskstats, so that feels wrong :)

[brauner]

Yeah, in my case here, I'm getting a completely empty file for /proc/diskstats, so that feels wrong :)

What version are you on?

[brauner]

This should not be an issue anymore. I've explicitly tried to handle this case.