docker stats displays wrong MEM USAGE and MEM % only on sysbox containers

[erickweil]

The docker statscommand seems fundamentally flawed on sysbox containers, not showing changes in either cpu and memory usage, always stays on 1 PID, also not catching the limit when running with --memory constraint.

The issue https://github.com/nestybox/sysbox/issues/303 marked this as solved, but I found this problem still happening on my tests.

Steps

1. First, Run two containers, one with sysbox runtime and another on runc:

   erick@erick-VirtualBox:~/testedocker/docker-vm$ docker run --privileged --name runc-container -d -P nestybox/ubuntu-bionic-systemd-docker:latest
   62c382e8b2e194f6d8d2f65cdc578888abb0f12caffc73527bc94266b1d70439
   erick@erick-VirtualBox:~/testedocker/docker-vm$ docker run --runtime=sysbox-runc --name sysbox-container -d -P nestybox/ubuntu-bionic-systemd-docker:latest
   36f226437bedabde221a7596846774a866fb089277478086dee22543963675c7
   erick@erick-VirtualBox:~/testedocker/docker-vm$ docker ps
   CONTAINER ID   IMAGE                                          COMMAND                  CREATED          STATUS          PORTS                                     NAMES
   36f226437bed   nestybox/ubuntu-bionic-systemd-docker:latest   "/sbin/init --log-le…"   3 seconds ago    Up 1 second     0.0.0.0:32770->22/tcp, :::32770->22/tcp   sysbox-container
   62c382e8b2e1   nestybox/ubuntu-bionic-systemd-docker:latest   "/sbin/init --log-le…"   18 seconds ago   Up 17 seconds   0.0.0.0:32769->22/tcp, :::32769->22/tcp   runc-container

2. then run docker stats to see memory and cpu usage:

   erick@erick-VirtualBox:~/testedocker/docker-vm$ docker stats --no-stream
   CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O        PIDS
   36f226437bed   sysbox-container   0.00%     2.277MiB / 3.808GiB   0.06%     3.41kB / 0B     0B / 8.19kB      1
   62c382e8b2e1   runc-container     0.33%     77.99MiB / 3.808GiB   2.00%     7.12kB / 670B   14.4MB / 311kB   28

   Look how the sysbox container wrongly show only 2.2M of memory usage, and there is no change on that, the container could be using Gigabytes of RAM but still a few Mb is shown.

the problem doesn't affect the 'normal' runc containers, showing correct ammounts of memory usage.

The system info on sys/fs/cgroup does show the correct ammounts:

erick@erick-VirtualBox:~/testedocker/docker-vm$ cat /sys/fs/cgroup/system.slice/docker-36f226437bedabde221a7596846774a866fb089277478086dee22543963675c7.scope/memory.current
64208896

System info

The test was run in a ubuntu virtual machine, with the following specs:

erick@erick-VirtualBox:~/testedocker/docker-vm$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 23.04
Release:    23.04
Codename:   lunar
erick@erick-VirtualBox:~/testedocker/docker-vm$ uname -a
Linux erick-VirtualBox 6.2.0-20-generic #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker -v
Docker version 24.0.2, build cb74dfc
erick@erick-VirtualBox:~/testedocker/docker-vm$ sysbox-runc --version
sysbox-runc
    edition:    Community Edition (CE)
    version:    0.6.2
    commit:     60ca93c783b19c63581e34aa183421ce0b9b26b7
    built at:   Mon Jun 12 03:49:19 UTC 2023
    built by:   Cesar Talledo
    oci-specs:  1.0.2-dev

As a side question, what is the recommended OS version and kernel version for running sysbox runtime?

[rodnymolina]

Hi @erickweil, my apologies for the belated response, somehow this issue fall off our radar. We are currently trying to prioritize Sysbox's functional issues, so it may take us some extra time to fix this one. Btw, I can see you have some familiarity with cgroups, so let us know if you have cycles (or interest) to fix this one yourself. We would really appreciate it.

[erickweil]

For other people looking for temporary solutions until this is solved, the below bash script does the job of showing the memory usage, in bytes.

CONTAINERS=$(docker ps -q --no-trunc)
for c in $CONTAINERS; do
       cat /sys/fs/cgroup/system.slice/docker-$c.scope/memory.current
done

But if you want a more readable output, below is a python program that displays in a table the above info, parsed into readable memory amounts. https://gist.github.com/erickweil/575eb6d027db647c1a8a2efb0195fe87

[sibidharan]

Setting systemd.unified_cgroup_hierarchy=0 in grub fixes the issue. Is there any disadvantages of this?

@ctalledo I am planning to use this setting since my app relies on docket stats for a lot of tasks. Please guild me if adding this in grub and making systemd use cgroupv2 has any immediate consequences?

[ctalledo]

Hi @sibidharan, thanks for following up and apologies for the belated response.

Setting systemd.unified_cgroup_hierarchy=0 in grub fixes the issue.

That's interesting and means the problem only occurs with cgroups v2 (not cgroups v1).

Is there any disadvantages of this?

No major disadvantages for Sysbox containers, but going forward the direction is to move to cgroups v2, so we need to fix it.