cgroups (documentation?) outdated / does not work when systemd mounts cgroups

[matthijskooijman]

I tried to use uwsgi to set some cgroup values, but failed. I don't think think much of this is uwsgi's fault, but I wanted to share my experience, since some things can be improved I think.

The instructions at https://uwsgi-docs.readthedocs.io/en/latest/Cgroups.html suggest mounting the cgroup manually, but on my system systemd already mounts cgroup filesystems in separate hierarchies at /sys/fs/cgroups, so I skipped that step. Then, the examples lists, e.g. cgroup=/cgroup/app1 end up with a mkdir error, since /cgroup does not exist. I tried changing to cgroup=/app1, but that just creates a directory with some files in the root e.g. really /app1, but the uwsgi log output still suggests that it actually configured the cgroup (but it created files instead of writing to existing files).

This is partly PEBCAK, but I had somewhat expected that I would specify cgroup names and uwsgi would figure out where to write them. I had not expected that you would be specifying absolute paths (especially since /cgroup/app1 did not look like a valid path, I would then have expected /sys/fs/cgroup).

Changing to cgroup=/sys/fs/cgroup/memory/app1 seems to work, except that it is limited to just a single cgroup controller (memory in this case), since systemd mounts most cgroup controllers in separated directories under /sys/fs/cgroup by default (unlike the uwsgi example, which mounts most of them in a single directory /cgroup). This limitation will be gone once cgroupv2 is used, since that only supports a single hierarchy.

I had also somewhat expected that these cgroups would be created below the current uwsgi cgroup. However, with cgroupv1, there can be multiple hierarchies (e.g. one for memory, one for cpu, etc.), so I guess this is not a feasible expectation (though with cgroupv2 there is only a single "unified" hierarchy, so then it could maybe be done).

Also, uwsgi writes the process id to the tasks file. However, according to the manpage, this file is intended to assign threads, not processes.

In cgroups v1, an individual thread can be moved to another cgroup by writing its thread ID (i.e., the kernel thread ID returned by clone(2) and gettid(2)) to the tasks file in a cgroup directory. This file can be read to discover the set of threads that are members of the cgroup.

It does seem to work, so maybe a pid is also a valid thread id? But it would be better to use the cgroup.procs file, especially since the tasks file (and thread support) was removed in cgroups v2).

In the end, I could not make this work, but that's because I need cgroupsv2 for memory.low, and my system is running with cgroupsv1 (and they cannot both be used at the same time, Debian is looking to switch to v2 in the upcoming bullseye release).

Still, I think there's a few things uwsgi could improve right now:

• Use cgroup.procs instead of tasks. AFAICS this should work with both v1 and v2.
• Fail when writing to non-existent files (e.g. cgroup.procs/tasks, but also non-existing cgroup variables, AFAICS creating new variables is never useful, it just means silent failure). This would mean calling fopen with r+ mode (which seems to be the only writable mode that does not pass O_CREAT).
• Add a note to the docs that systems that systems that already have cgroups mounted (e.g. at /sys/fs/cgroup might need to adapt the paths to point to that directory, and maybe include the cgroup controller to use as well).

[matthijskooijman]

Fail when writing to non-existent files (e.g. cgroup.procs/tasks, but also non-existing cgroup variables, AFAICS creating new variables is never useful, it just means silent failure). This would mean calling fopen with r+ mode (which seems to be the only writable mode that does not pass O_CREAT).

Looking more closely, it turns it is actually impossible to create variables inside the cgroup filesystem, that returns permission denied. However, it seems that currently uwsgi ignores fopen errors for the cgroup_opt values. This might be to allow multiple cgroup options to join multiple cgroups in different controllers (and then ignoring any options not set in that particular cgroup controller), but it also has the side effect of not showing any error for any invalid options (though it also does not log that the option was set, which is good).

My original "do not create" suggestion does still seem valid, though, since that ensures that if you use a path that does not actually point to the cgroup fs but just a normal fs, it fails instead of creating regular files and logging that cgroups were configured.

[rcarmo]

I've been working along the same lines, and I'm sad that neither this nor #2504 seem to be addressed in the docs or current commits.