Open st0rmi opened 1 year ago
I've done a bit more digging using strace and apparently the 'dotnet' command gets this path from reading /proc/PID/maps
. I've looked at it in the container and it looks like this for the 'cat' command itself:
# cat /proc/self/maps
55a28617e000-55a286180000 r--p 00000000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286180000-55a286185000 r-xp 00002000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286185000-55a286188000 r--p 00007000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286188000-55a286189000 r--p 00009000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286189000-55a28618a000 rw-p 0000a000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286ca3000-55a286cc4000 rw-p 00000000 00:00 0 [heap]
7f551a0da000-7f551a0fc000 rw-p 00000000 00:00 0
7f551a0fc000-7f551a151000 r--p 00000000 00:56 1061864 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_CTYPE
7f551a151000-7f551a152000 r--p 00000000 00:56 1061871 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_NUMERIC
7f551a152000-7f551a153000 r--p 00000000 00:56 1061874 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_TIME
7f551a153000-7f551a2c6000 r--p 00000000 00:56 1061863 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_COLLATE
7f551a2c6000-7f551a2c7000 r--p 00000000 00:56 1061869 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MONETARY
7f551a2c7000-7f551a2c8000 r--p 00000000 00:56 1061868 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MESSAGES/SYS_LC_MESSAGES
7f551a2c8000-7f551a2cf000 r--s 00000000 00:56 1458798 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f551a2cf000-7f551a2f1000 r--p 00000000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a2f1000-7f551a44b000 r-xp 00022000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a44b000-7f551a49a000 r--p 0017c000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a49a000-7f551a49e000 r--p 001ca000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a49e000-7f551a4a0000 rw-p 001ce000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a4a0000-7f551a4a6000 rw-p 00000000 00:00 0
7f551a4a6000-7f551a4a7000 r--p 00000000 00:56 1061872 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_PAPER
7f551a4a7000-7f551a4a8000 r--p 00000000 00:56 1061870 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_NAME
7f551a4a8000-7f551a4a9000 r--p 00000000 00:56 1061862 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_ADDRESS
7f551a4a9000-7f551a4aa000 r--p 00000000 00:56 1061873 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_TELEPHONE
7f551a4aa000-7f551a4ab000 r--p 00000000 00:56 1061866 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MEASUREMENT
7f551a4ab000-7f551a4ac000 r--p 00000000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4ac000-7f551a4cc000 r-xp 00001000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4cc000-7f551a4d4000 r--p 00021000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d4000-7f551a4d5000 r--p 00000000 00:56 1061865 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_IDENTIFICATION
7f551a4d5000-7f551a4d6000 r--p 00029000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d6000-7f551a4d7000 rw-p 0002a000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d7000-7f551a4d8000 rw-p 00000000 00:00 0
7ffd54194000-7ffd541b5000 rw-p 00000000 00:00 0 [stack]
7ffd541d8000-7ffd541db000 r--p 00000000 00:00 0 [vvar]
7ffd541db000-7ffd541dc000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
I believe the OverlayFS paths in this file are paths from the Docker host that should probably not be exposed to processes running inside the sysbox container.
Hi @st0rmi, thanks for trying Sysbox and for filing the issue.
dotnet publish -warnaserror --configuration Release -o build/server Server
This command is running inside the sysbox container correct? Sounds like it is, just want to double check.
Enabling more verbose output for dotnet shows that it appears to believe its path is /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet instead of just /usr/share/dotnet/dotnet:
Got it.
I've done a bit more digging using strace and apparently the 'dotnet' command gets this path from reading /proc/PID/maps
I see.
I've looked at it in the container and it looks like this for the 'cat' command itself:
# cat /proc/self/maps 55a28617e000-55a286180000 r--p 00000000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
That's strange; the paths under /proc/self/maps
should be relative to the container's chroot jail, so it should have reported /bin/cat
instead.
Here's what I see on my Ubuntu dev machine for example, when launching a container with Sysbox:
$ docker run --runtime=sysbox-runc -it --rm ubuntu
root@21a8be6a8b6e:/# cat /proc/self/maps
55e5afaf9000-55e5afafb000 r--p 00000000 00:a49 28363251 /usr/bin/cat
I believe the OverlayFS paths in this file are paths from the Docker host that should probably not be exposed to processes running inside the sysbox container.
Correct, the paths should be relative to the container's chroot jail. How can I reproduce on my side?
Hi @st0rmi, I was able to reproduce this issue by configuring the host machine with the shiftfs
kernel module:
$ docker run --runtime=sysbox-runc -it --rm alpine
/ # cat /proc/self/maps
563ed2ed9000-563ed2ee5000 r--p 00000000 00:2c 1806417 /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
563ed2ee5000-563ed2f81000 r-xp 0000c000 00:2c 1806417 /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
563ed2f81000-563ed2fa2000 r--p 000a8000 00:2c 1806417 /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
...
The issue is also reported in the shiftfs-dkms repo: https://github.com/toby63/shiftfs-dkms/issues/21#issuecomment-1441499170
I doubt this will be fixed as shiftfs is on the way to deprecation (replaced by the Linux kernel's ID-mapped mounts feature since kernel 5.12+).
If you have a host with kernel 5.12+, my suggestion is that you work-around this by removing shiftfs from your host (e.g., sudo rmmod shiftfs
) and then restarting Sysbox (sudo systemctl restart sysbox
).
If your host is kernel is < 5.12, then you usually need shiftfs do use Sysbox. I say usually because without shiftfs, mounting host files/dirs/volumes into a container won't work properly (e.g., docker run --runtime=sysbox-runc -v /some/host/vol:/mnt ...
) because they will show up inside the container as owned by nobody:nogroup
. If you are not mounting host file/dirs/volumes into a container, then you can try removing shiftfs and things should work (including running Docker inside the Sysbox container).
FYI, you can also ask Sysbox to not use shiftfs (even if present in the kernel) by configuring the systemd service unit for the sysbox-mgr such that you pass the --disable-shiftfs
flag to sysbox-mgr. See the Sysbox docs for more on this.
Hope that helps.
Hi,
I'm using sysbox for build containers to be able to generate Docker images inside a Docker container. Creating container images works, but for some reason the 'dotnet' command used to build C# .NET stuff doesn't.
It fails with the following error message indicating that it is using the wrong path to create a subprocess:
Enabling more verbose output for dotnet shows that it appears to believe its path is
/var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet
instead of just/usr/share/dotnet/dotnet
:I've also tried manually setting the environment variable DOTNET_HOST_PATH to the correct value, but it still got overriden to the overlay2 one. When I run the same command using the same build container image without sysbox, the 'dotnet' command works as expected.
The host is a fully-patched Ubuntu 20.04 server using the following kernel:
Linux be0b24a50b5e 5.4.0-139-generic #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 x86_64 GNU/Linux
Sysbox 0.5.2-0.linux is used.This is the output of the docker info command:
Does anyone have any idea what exactly is going on here? It feels like somehow an OverlayFS path is getting exposed to a process in a way it probably shouldn't be.