search

nestybox / sysbox

An open-source, next-generation "runc" that empowers rootless containers to run workloads such as Systemd, Docker, Kubernetes, just like VMs.
Apache License 2.0
2.78k stars 153 forks source link

/proc/<PID>/maps paths are incorrect inside a Sysbox container with shiftfs (breaks apps like dotnet inside the container) #636

Open st0rmi opened 1 year ago

st0rmi commented 1 year ago

Hi,

I'm using sysbox for build containers to be able to generate Docker images inside a Docker container. Creating container images works, but for some reason the 'dotnet' command used to build C# .NET stuff doesn't.

It fails with the following error message indicating that it is using the wrong path to create a subprocess:

# dotnet publish -warnaserror --configuration Release -o build/server Server

Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.400

Telemetry
---------
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
MSBuild version 17.3.0+92e077650 for .NET
  Determining projects to restore...
  Restored /home/jenkins/workspace/<Tool Name>/Server/Server.csproj (in 1.35 sec).
  Restored /home/jenkins/workspace/<Tool Name>/Shared/Shared.csproj (in 2.74 sec).
/usr/share/dotnet/sdk/6.0.400/Roslyn/Microsoft.CSharp.Core.targets(75,5): error MSB6004: The specified task executable location "/var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet" is invalid. [/home/jenkins/workspace/<Tool Name>/Shared/Shared.csproj]

Enabling more verbose output for dotnet shows that it appears to believe its path is /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet instead of just /usr/share/dotnet/dotnet:

MSBuild version 17.3.0+92e077650 for .NET
/usr/share/dotnet/sdk/6.0.400/MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll -maxcpucount -property:PublishDir=/home/jenkins/workspace/<Tool Name>/build/server -property:Configuration=Release -restore -target:Publish -verbosity:m -verbosity:diag -warnaserror Server/Server.csproj
Build started 02/15/2023 08:35:15.
Environment at start of build:
HOSTNAME = be0b24a50b5e
JAVA_HOME = /opt/java/openjdk
LANG = C.UTF-8
DOTNET_HOST_PATH = /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet
MSBuildExtensionsPath = /usr/share/dotnet/sdk/6.0.400/
SHLVL = 1
MSBuildSDKsPath = /usr/share/dotnet/sdk/6.0.400/Sdks
HOME = /root
PATH = /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD = /home/jenkins/workspace/<Tool Name>
MSBuildLoadMicrosoftTargetsReadOnly = true
NODE_VERSION = 16.15.1
OLDPWD = /home/jenkins/workspace
TERM = xterm
_ = /usr/bin/dotnet
JENKINS_AGENT_HOME = /home/jenkins

08:35:15.033     0>Process = "/var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet"
                   MSBuild executable path = "/usr/share/dotnet/sdk/6.0.400/MSBuild.dll"
                   Command line arguments = "/usr/share/dotnet/sdk/6.0.400/MSBuild.dll -maxcpucount -verbosity:m -restore -target:Publish -property:PublishDir=/home/jenkins/workspace/<Tool Name>/build/server -property:Configuration=Release -verbosity:diag -warnaserror Server -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll"

I've also tried manually setting the environment variable DOTNET_HOST_PATH to the correct value, but it still got overriden to the overlay2 one. When I run the same command using the same build container image without sysbox, the 'dotnet' command works as expected.

The host is a fully-patched Ubuntu 20.04 server using the following kernel: Linux be0b24a50b5e 5.4.0-139-generic #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 x86_64 GNU/Linux Sysbox 0.5.2-0.linux is used.

This is the output of the docker info command:

$ sudo docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 12
  Running: 1
  Paused: 0
  Stopped: 11
 Images: 27
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc sysbox-runc io.containerd.runc.v2
 Default Runtime: sysbox-runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc version:
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-139-generic
 Operating System: Ubuntu 20.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.782GiB
 Name: rtb-linuxbuild
 ID: MWHW:2JPI:B4UT:6GQQ:L6Z4:B3IQ:5K66:2ZZ4:VQV5:OO5G:UMYI:QKTZ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.25.0.0/16, Size: 24

WARNING: No swap limit support

Does anyone have any idea what exactly is going on here? It feels like somehow an OverlayFS path is getting exposed to a process in a way it probably shouldn't be.

st0rmi commented 1 year ago

I've done a bit more digging using strace and apparently the 'dotnet' command gets this path from reading /proc/PID/maps. I've looked at it in the container and it looks like this for the 'cat' command itself:

# cat /proc/self/maps
55a28617e000-55a286180000 r--p 00000000 00:56 1060266                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286180000-55a286185000 r-xp 00002000 00:56 1060266                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286185000-55a286188000 r--p 00007000 00:56 1060266                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286188000-55a286189000 r--p 00009000 00:56 1060266                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286189000-55a28618a000 rw-p 0000a000 00:56 1060266                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286ca3000-55a286cc4000 rw-p 00000000 00:00 0                          [heap]
7f551a0da000-7f551a0fc000 rw-p 00000000 00:00 0
7f551a0fc000-7f551a151000 r--p 00000000 00:56 1061864                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_CTYPE
7f551a151000-7f551a152000 r--p 00000000 00:56 1061871                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_NUMERIC
7f551a152000-7f551a153000 r--p 00000000 00:56 1061874                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_TIME
7f551a153000-7f551a2c6000 r--p 00000000 00:56 1061863                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_COLLATE
7f551a2c6000-7f551a2c7000 r--p 00000000 00:56 1061869                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MONETARY
7f551a2c7000-7f551a2c8000 r--p 00000000 00:56 1061868                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MESSAGES/SYS_LC_MESSAGES
7f551a2c8000-7f551a2cf000 r--s 00000000 00:56 1458798                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f551a2cf000-7f551a2f1000 r--p 00000000 00:56 1449723                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a2f1000-7f551a44b000 r-xp 00022000 00:56 1449723                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a44b000-7f551a49a000 r--p 0017c000 00:56 1449723                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a49a000-7f551a49e000 r--p 001ca000 00:56 1449723                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a49e000-7f551a4a0000 rw-p 001ce000 00:56 1449723                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a4a0000-7f551a4a6000 rw-p 00000000 00:00 0
7f551a4a6000-7f551a4a7000 r--p 00000000 00:56 1061872                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_PAPER
7f551a4a7000-7f551a4a8000 r--p 00000000 00:56 1061870                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_NAME
7f551a4a8000-7f551a4a9000 r--p 00000000 00:56 1061862                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_ADDRESS
7f551a4a9000-7f551a4aa000 r--p 00000000 00:56 1061873                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_TELEPHONE
7f551a4aa000-7f551a4ab000 r--p 00000000 00:56 1061866                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MEASUREMENT
7f551a4ab000-7f551a4ac000 r--p 00000000 00:56 1449510                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4ac000-7f551a4cc000 r-xp 00001000 00:56 1449510                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4cc000-7f551a4d4000 r--p 00021000 00:56 1449510                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d4000-7f551a4d5000 r--p 00000000 00:56 1061865                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_IDENTIFICATION
7f551a4d5000-7f551a4d6000 r--p 00029000 00:56 1449510                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d6000-7f551a4d7000 rw-p 0002a000 00:56 1449510                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d7000-7f551a4d8000 rw-p 00000000 00:00 0
7ffd54194000-7ffd541b5000 rw-p 00000000 00:00 0                          [stack]
7ffd541d8000-7ffd541db000 r--p 00000000 00:00 0                          [vvar]
7ffd541db000-7ffd541dc000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

I believe the OverlayFS paths in this file are paths from the Docker host that should probably not be exposed to processes running inside the sysbox container.

ctalledo commented 1 year ago

Hi @st0rmi, thanks for trying Sysbox and for filing the issue.

dotnet publish -warnaserror --configuration Release -o build/server Server

This command is running inside the sysbox container correct? Sounds like it is, just want to double check.

Enabling more verbose output for dotnet shows that it appears to believe its path is /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet instead of just /usr/share/dotnet/dotnet:

Got it.

I've done a bit more digging using strace and apparently the 'dotnet' command gets this path from reading /proc/PID/maps

I see.

I've looked at it in the container and it looks like this for the 'cat' command itself:

# cat /proc/self/maps
55a28617e000-55a286180000 r--p 00000000 00:56 1060266                    /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat

That's strange; the paths under /proc/self/maps should be relative to the container's chroot jail, so it should have reported /bin/cat instead.

Here's what I see on my Ubuntu dev machine for example, when launching a container with Sysbox:

$ docker run --runtime=sysbox-runc -it --rm ubuntu
root@21a8be6a8b6e:/# cat /proc/self/maps
55e5afaf9000-55e5afafb000 r--p 00000000 00:a49 28363251                  /usr/bin/cat

I believe the OverlayFS paths in this file are paths from the Docker host that should probably not be exposed to processes running inside the sysbox container.

Correct, the paths should be relative to the container's chroot jail. How can I reproduce on my side?

ctalledo commented 1 year ago

Hi @st0rmi, I was able to reproduce this issue by configuring the host machine with the shiftfs kernel module:

$ docker run --runtime=sysbox-runc -it --rm alpine
/ # cat /proc/self/maps
563ed2ed9000-563ed2ee5000 r--p 00000000 00:2c 1806417                    /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
563ed2ee5000-563ed2f81000 r-xp 0000c000 00:2c 1806417                    /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
563ed2f81000-563ed2fa2000 r--p 000a8000 00:2c 1806417                    /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
...

The issue is also reported in the shiftfs-dkms repo: https://github.com/toby63/shiftfs-dkms/issues/21#issuecomment-1441499170

I doubt this will be fixed as shiftfs is on the way to deprecation (replaced by the Linux kernel's ID-mapped mounts feature since kernel 5.12+).

If you have a host with kernel 5.12+, my suggestion is that you work-around this by removing shiftfs from your host (e.g., sudo rmmod shiftfs) and then restarting Sysbox (sudo systemctl restart sysbox).

If your host is kernel is < 5.12, then you usually need shiftfs do use Sysbox. I say usually because without shiftfs, mounting host files/dirs/volumes into a container won't work properly (e.g., docker run --runtime=sysbox-runc -v /some/host/vol:/mnt ...) because they will show up inside the container as owned by nobody:nogroup. If you are not mounting host file/dirs/volumes into a container, then you can try removing shiftfs and things should work (including running Docker inside the Sysbox container).

FYI, you can also ask Sysbox to not use shiftfs (even if present in the kernel) by configuring the systemd service unit for the sysbox-mgr such that you pass the --disable-shiftfs flag to sysbox-mgr. See the Sysbox docs for more on this.

Hope that helps.