Running docker container with --cpuset-cpus flag and checking with nproc in container does not show the right cpu count

[ricarcon]

When we initially run the docker container, in our debug and logging, we have found that the initial call to nproc shows the correct number of CPUs, but as we continue to check throughout the lifecycle of the container, nproc changes and have found that the contents of the cpuset.cpus file has changed too, to reflect the total number of CPUs in the underlying VM host that the container runs on. We've tried adding monitoring, but things like auditctl and ausearch do not work on a containerized environment, to see what is changing the contents of the cpuset.cpus file. Here's some sample output:

13:05:27  CPU Debug...1
13:05:27  12-15
13:05:27  tail: /sys/fs/cgroup/cpuset/cpuset.cpus: file truncated
13:05:27  12-15
13:05:28  4

and later on, in the same run we see this:

CPU Debug... 3
13:07:58  0-55
13:07:58  tail: /sys/fs/cgroup/cpuset/cpuset.cpus: file truncated
13:07:58  0-55
13:08:00  56

we start the container with this command: docker run -t -d -u 1000:1000 -u root -v /mnt:/mnt --cap-add SYS_PTRACE --cpuset-cpus=12,13,14,15 --memory=8192m --memory-swap=1024m

It also happens randomly, so above we see CPU Debug...1 but sometimes it happens in CPU Debug...2 or 3 or 4 etc. Any help will be appreciated to debug this and determine the underlying cause.

[ricarcon]

Anyone?

[thaJeztah]

This file is managed by the kernel, and docker does not dynamically change things (it configures the container when it's created, after that, it's the kernel managing everything), so it could be a kernel issue, or perhaps something else (hot plug CPUs?)

[pnoch]

exact same issue here

[xtrusia]

I also faced this issue, though it could not be docker issue. I share my info.

I tested Ubuntu Focal, Jammy, upstream 6.0.0 linux kernel. but they all have the same issue.

I ran below scenario.

cpu # is 0-24

1. sudo docker create --name binding1 --cpuset-cpus 10-23 -it ubuntu:latest /bin/bash -c "sh -c 'while true; do sleep 600; done'"
2. sudo docker container start binding1
3. while true; do sudo docker exec binding1 sh -c 'taskset -cp $$'; sleep 5; done

in the beginning, it showed me 10-23, but after some time ( from few hours to few days or weeks )

3's result.

pid 120956's current affinity list: 0-7,10-15

if it is not caused by kernel, is it possible that systemd has issue?

Thanks.

[xtrusia]

@thaJeztah Could you please adavise anything because I just found out that

/run/systemd/transient/docker-89de798b4c2421a00a447953bdaf194e51af0626335d4cd009dbf67714052122.scope.d/50-AllowedCPUs.conf has incorrect number.

if i create with 10-23 , 50-AllowedCPUs has 0-7 10-15

so initially container starts with 10-23, but systemctl daemon-reload make container's cpuset change to 0-7 10-15

I think if there find some code related to this file. I can find any fix for it.

Thanks in advance.

[tai271828]

fwiw, this issue may be caused by runc and fixed https://github.com/opencontainers/runc/pull/3611