Closed fredr closed 2 years ago
@vrothberg PTAL, I think this one's in libimage
Podman prefers the / be mounted -rshared. This could be triggering the issue.
Podman prefers the / be mounted -rshared. This could be triggering the issue.
Is this something I could test? I dont have sudo rights inside the sandbox, so I can't run sudo mount --make-rshared /
from within it.
I did figure out a bit more how to debug the bazel sandbox. By adding --sandbox_debug all the sandbox files will be saved after execution, and --verbose_failures, I get the failing command, including how the sandbox is setup. So to reproduce this panic, I can run
./linux-sandbox -w /run/user/1000 -w /home/fredr/.local/share/containers/storage -w /dev/shm -D -- podman run hello-world
-w makes the folder or file writable from within the sandbox, everything else should still be readable.
If I change /run/user/1000 to be mounted as an empty tempfs dir, I no longer get the panic, but instead I get this error (not sure if that is of any use):
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
cannot setresgid: Invalid argument
Error: potentially insufficient UIDs or GIDs available in user namespace (requested 0:0 for /home/fredr/.local/share/containers/storage/overlay/l): Check /etc/subuid and /etc/subgid: chown /home/fredr/.local/share/containers/storage/overlay/l: invalid argument
If I run podman ps
and such commands, I do not get the panic, but I still get
cannot setresgid: Invalid argument
@vrothberg PTAL, I think this one's in libimage
It's a Podman-side issue. It seems we're calling the libimage runtime without having initialized it; the nil deref is on the runtime object. I guess @rhatdan is on point.
@fredr can you paste the contents of /etc/subuid
and /etc/subgid
?
@fredr can you paste the contents of
/etc/subuid
and/etc/subgid
?
$ cat /etc/subgid /etc/subuid
fredr:100000:65536
fredr:100000:65536
I should also mention that podman works just fine outside of the sandbox
Can linux-sandbox be changed to expose / as mount-rshared?
BTW What is linux-sandbox?
In the meantime, I am going to have a look how we can prevent the segfault. Podman should error out or perform other counter measures.
@fredr do you have a simple reproducer?
Could you also rerun podman
with --log-level=debug
?
Can linux-sandbox be changed to expose / as mount-rshared?
Not in any way that I have been able to figure out unfortunately
BTW What is linux-sandbox?
it is part of the build tool bazel, https://docs.bazel.build/versions/main/sandboxing.html I'm quite new to it myself, but this is it: https://github.com/bazelbuild/bazel/blob/master/src/main/tools/linux-sandbox.cc
@fredr do you have a simple reproducer?
Since it relies on bazel, it wont be super simple, but in this gist: https://gist.github.com/fredr/dd0e5c3639fa109df82471292d6bc8c3
If you put BUILD and WORKSPACE in a directory and run:
bazel build //... --sandbox_writable_path=${XDG_RUNTIME_DIR} --sandbox_writable_path=${HOME}/.local/share/containers/storage --sandbox_debug --verbose_failure
Could you also rerun
podman
with--log-level=debug
?
Here I executed podman --log-level=debug run hello-world
inside the sandbox
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level=debug run hello-world)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/etc/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/fredr/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/fredr/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/fredr/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/fredr/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend file
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /home/fredr/.config/cni/net.d/87-podman.conflist
INFO[0000] Found CNI network kind (type=bridge) at /home/fredr/.config/cni/net.d/kind.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 49
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
cannot setresgid: Invalid argument
DEBU[0000] Pulling image hello-world (policy: missing)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x55910c5f5bd4]
goroutine 1 [running]:
github.com/containers/common/libimage.(*Runtime).Pull(0x0, {0x55910d378b68, 0xc000714a80}, {0x7ffd0d280783, 0xb}, 0x1, 0xc00056e280)
github.com/containers/common@v0.44.4/libimage/pull.go:117 +0x714
github.com/containers/podman/v3/pkg/domain/infra/abi.(*ImageEngine).Pull(0x55910cd0216a, {0x55910d378b68, 0xc000714a80}, {0x7ffd0d280783, 0xc00056e580}, {0x0, {0x0, 0x0}, {0x0, 0x0}, ...})
github.com/containers/podman/v3/pkg/domain/infra/abi/images.go:231 +0x1d6
github.com/containers/podman/v3/cmd/podman/containers.PullImage({_, _}, {{0x55910dfc3b88, 0x0, 0x0}, {0x55910dfc3b88, 0x0, 0x0}, {0x0, 0x0}, ...})
github.com/containers/podman/v3/cmd/podman/containers/create.go:300 +0x2fd
github.com/containers/podman/v3/cmd/podman/containers.run(0x55910dedb120, {0xc00030bc80, 0x1, 0x1})
github.com/containers/podman/v3/cmd/podman/containers/run.go:143 +0x43b
github.com/spf13/cobra.(*Command).execute(0x55910dedb120, {0xc00003c0b0, 0x1, 0x1})
github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x55910def1e20)
github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/cobra@v1.2.1/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
github.com/spf13/cobra@v1.2.1/command.go:895
main.Execute()
github.com/containers/podman/v3/cmd/podman/root.go:91 +0xbe
main.main()
github.com/containers/podman/v3/cmd/podman/main.go:39 +0x74
Thanks!
DEBU[0000] Not configuring container store
That seems to be it: Podman is not configuring the store if it's lacking the cap_sys_admin capability. In that case, Podman just continues but without the image runtime which explains how we run into the segfault.
@mheon @giuseppe I am totally undecided on what to do in this case though. Does a Podman without a configured store make sense?
The store-deactivation code I'm aware of was intended for performance reasons, to not require commands that would never require a store to initialize one; I think we're talking about different code here, though, because run will always need a store. Is this the code that @rhatdan added so that Podman as root could revert to pseudo-rootless functionality if CAP_SYS_ADMIN was not available? I'm not terribly familiar with it, but the concept seemed to make sense.
I'm referring to the following code: https://github.com/containers/podman/blob/main/libpod/runtime.go#L376-L391
I managed to build linux-sandbox
locally and reproduce the issue with many commands: we don't create a store but continue.
One very strange thing is that capsh --print
claims to have cap_sys_admin
...
Quick update: what works for me is to use linux-sandbox -w / -- podman ...
. This will just mount everything writable in the "sandbox".
Quick update: what works for me is to use
linux-sandbox -w / -- podman ...
. This will just mount everything writable in the "sandbox".
Interesting! When I try that I get:
ERRO[0000] set sticky bit on: chmod /run/user/1000/libpod: read-only file system
Yeah, this is the rootless-when-no-sysadmin code I was talking about - https://github.com/containers/podman/commit/722ea2f1f82ff16271b50b508d709e5da275e32a
Apparently was by @giuseppe instead of @rhatdan?
we switched from "detect rootless" to "detect if we have CAP_SYS_ADMIN" because running with EUID=0
is not enough to perform all the operations needed to mount the storage, pull images and run containers. It is useful for example when running Podman in a container as root but without capabilities, which is somehow equivalent to run as rootless, so we need to create a user namespace to gain there the needed capabilities.
Is there a way to avoid the segfault? Should we check CAP_SYS_ADMIN || CAP_SETUID && CAP_SETGID, because without one of those situations, Podman is not going to work.
What would happen if the condition is not met? Currently, we just don't configure the store and continue but I think we should error if there is no store.
when we do not have enough privileges, we re-exec and gain these privileges.
We should not get that far in the parent Podman process and re-exec from SetupRootless
(pkg/domain/infra/abi/system.go
).
Does the re-exec fail and Podman somehow keeps going without enough privileges?
Can it be that the bazel sandbox reports having those capabilities, but it actually doesn't? and that causes the panic?
Can it be that the bazel sandbox reports having those capabilities, but it actually doesn't? and that causes the panic?
I don't think so, these are coming from the kernel and we read them from /proc
.
I believe this is a discussion and not an issue with Podman transferring.
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Steps to reproduce the issue:
A gist for setting up and reproducing the error https://gist.github.com/fredr/dd0e5c3639fa109df82471292d6bc8c3
Download BUILD and WORKSPACE to a folder
In that folder, run:
Describe the results you received:
Describe the results you expected: I'm guessing something in the setup is wrong, and this should trigger an error message telling me what.
Additional information you deem important (e.g. issue happens only occasionally): Bazel executes within a sandbox, and it is when executing podman from inside that sandbox that this seems to happen. If I run the generated script that fails from my terminal, it works just fine.
Output of
podman version
:Output of
podman info --debug
:Package info (e.g. output of
rpm -q podman
orapt list podman
):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
/usr/bin/docker is symlinked to /usr/bin/podman