avagin
commented
1 year ago probably it has been broken by d1f3b45b38dce7601fd89dca263d8d1f9b98ae79.
Closed avagin closed 1 year ago
avagin
commented
1 year ago probably it has been broken by d1f3b45b38dce7601fd89dca263d8d1f9b98ae79.
avagin
commented
1 year ago @blechschmidt, pls take a look at this.
blechschmidt
commented
1 year ago I am sorry in case my commit introduced this.
@fishy, does this issue still occur and could you please provide some more instructions on how to reproduce this? It looks like the log is missing the debug messages (because the --debug flag has been removed from the wrapper script?). If the information is not too sensitive, could you post a log with debug messages including the container spec etc.? Could you maybe also post the other log files (i.e. not only *.create)?
I tried to reproduce this, but I was not able to so far. The following works fine for me with multiple versions:
# Create the wrapper script
sudo sh -c "echo -e \#\!/bin/bash\\\n\\\n/usr/local/bin/runsc --network host --ignore-cgroups --debug --debug-log \\'/tmp/runsc/runsc.log.%TEST%.%TIMESTAMP%.%COMMAND%\\' \\\"\\\$@\\\" > /usr/local/bin/runsc-podman"
# Make the wrapper executable
sudo chmod +x /usr/local/bin/runsc-podman
# Run podman without root
podman --runtime /usr/local/bin/runsc-podman run --security-opt=label=disable docker.io/library/busybox echo Hello, WorldThis outputs Hello, World, as expected. This is my log output for multiple gVisor releases:
Note that changing /dev/stdin ownership fails in all of my test runs, but it is only a warning message, not a fatal error. See: https://github.com/google/gvisor/blob/153f0217e6b2fa84c06a4ecfd98cbe87560ff15c/runsc/sandbox/sandbox.go#L1406-L1426
Thus, failure to change ownership should not result in abnormal termination, which is why I think that the error is not necessarily related to that warning. Notably, the warning is also printed in my run with version 20230214.0.
The log hints at the sandbox process being terminated unexpectedly for some other reason.
fishy
commented
1 year ago from:
$ podman --runtime=/home/fishy/bin/runsc-podman.sh run --userns=keep-id --rm -v "${PWD}":/data/ --user "$(id -u):$(id -g)" --platform= ghcr.io/reddit/thrift-compiler:0.18.1 --version
Error: OCI runtime error: /home/fishy/bin/runsc-podman.sh: creating container: cannot create sandbox: cannot read client sync file: waiting for sandbox to start: EOFwith:
fishy
commented
1 year ago The dockerfile is: https://github.com/reddit/thrift-compiler/blob/main/Dockerfile
blechschmidt
commented
1 year ago I can reproduce this symptom now, but I am not entirely sure whether this is the issue that you are experiencing as well, since I experience it for older versions, too. Depending on what version I am running, the error messages in the boot log differ.
Two more questions:
*.boot log please? It should reside in the same folder as the *.create log.20230306.0 in your case. Is that correct?*.boot and *.create logs of a working version, this would be great, so one could see the diff.fishy
commented
1 year ago So I'm sure it was broken on 20230320.0 and the version before it. I cannot remember what's the last version number it worked.
And also I no longer have a version that works so I don't have the debug logs. When it works I remove the debug logs from the wrapper because I don't really need all those logs laying around. When it's broken, I just switch to use crun as the runtime.
blechschmidt
commented
1 year ago Thank you. So this is what causes the boot process to fail:
W0505 10:13:07.234805 451062 util.go:64] FATAL ERROR: Error reading mounts file: error unmarshaling mounts: unexpected end of JSON inputSo it does not look to me like this is related to the error while chowning /dev/stdin, @avagin.
Sorry for having to ask you for logs again, @fishy, but could you please also post the .gofer log? It might help to see whether this is the same issue as #8205, which occurs in rootless podman. Do you happen to run podman inside another containerized environment? Does the "solution" to #8205, i.e. adding the -TESTONLY-unsafe-nonroot argument to the wrapper script, help in your case?
I tried to reproduce the same issue on a Debian machine (Vagrant and DigitalOcean VPS) to be as close as possible to your execution environment, but I am still failing. I observe a different error: W0506 11:15:00.611892 6249 server.go:116] Control auth failure: other UID = 1000, current UID = 0
fishy
commented
1 year ago Do you happen to run podman inside another containerized environment?
no
adding the -TESTONLY-unsafe-nonroot argument to the wrapper script, help in your case?
no
blechschmidt
commented
1 year ago Interestingly, remounting /proc/root as readonly succeeds in my environment but not in yours. What's your Linux kernel version?
fishy
commented
1 year ago $ uname -a
Linux perch 6.1.0-7-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-2 (2023-04-08) x86_64 GNU/LinuxI have a suspicion what the reason might be. --userns=keep-id might cause the mount operation to fail, but I have not looked at the podman implementation to confirm this and I can't explain how this might have worked with previous versions. Does it work if you run podman without the --userns=keep-id and/or without the --user argument?
Otherwise, if that's not it, I am completely clueless and would first need to figure out why the execution differs on your machine vs. on my machine. Do you have any local podman configuration that might cause a different behavior compared to my fresh machine (as I am on the same kernel)?
fishy
commented
1 year ago ok yes removing --userns=keep-id worked:
$ podman --runtime=/home/fishy/bin/runsc-podman.sh run --rm -v "${PWD}":/data/ --user "$(id -u):$(id -g)" --platform= ghcr.io/reddit/thrift-compiler:0.18.1 --version
Thrift version 0.18.1But for my use case that arg is essential, and runsc worked before with --userns=keep-id and podman.
bobobo1618
commented
1 year ago I've run into this error message as well. To reproduce:
dnf install -y podmansudo podman --log-level debug run --rm -it --runtime=runsc --runtime-flag 'debug' --runtime-flag 'debug-log=/tmp/runsc-debug.log' --security-opt label=disable docker.io/ubuntu:latest /bin/bashThis then comes up in the log:
I0717 22:38:19.451369 1 gofer.go:428] Remounting root as readonly: "/proc/root"
W0717 22:38:19.451539 1 util.go:64] FATAL ERROR: Error setting up root FS: remounting root as read-only with source: "/proc/root", target: "/proc/root", flags: 0x5021, err: operation not permitted
Error setting up root FS: remounting root as read-only with source: "/proc/root", target: "/proc/root", flags: 0x5021, err: operation not permittedUnlike @fishy, adding --runtime-flag 'TESTONLY-unsafe-nonroot' did help for me.
felschr
commented
1 year ago Any new infos here?
I'm running into this issue, perhaps because I'm using --uidmap / --gidmap.
--runtime-flag 'TESTONLY-unsafe-nonroot' does solve the issue for me, but it sounds like a horrible workaround for normal use.
blechschmidt
commented
1 year ago I've run into this error message as well. To reproduce:
* Run a clean install of Rocky Linux 9.2 (e.g. spin up a VM) * `dnf install -y podman` * https://gvisor.dev/docs/user_guide/install/ * `sudo podman --log-level debug run --rm -it --runtime=runsc --runtime-flag 'debug' --runtime-flag 'debug-log=/tmp/runsc-debug.log' --security-opt label=disable docker.io/ubuntu:latest /bin/bash`This then comes up in the log:
I0717 22:38:19.451369 1 gofer.go:428] Remounting root as readonly: "/proc/root" W0717 22:38:19.451539 1 util.go:64] FATAL ERROR: Error setting up root FS: remounting root as read-only with source: "/proc/root", target: "/proc/root", flags: 0x5021, err: operation not permitted Error setting up root FS: remounting root as read-only with source: "/proc/root", target: "/proc/root", flags: 0x5021, err: operation not permitted
For me, a bisection indicates that 6b317756e88456d58a4fd4c176b3ea4e6654096d introduced this error. Adding --runtime-flag 'directfs=false' to the podman command, as suggested by the commit message, helps in my case. @ayushr2, could you please have a look?
I am not sure if this is the same issue that @fishy is experiencing though (which I still cannot reproduce), since that commit has been published after the initial issue.
ayushr2
commented
1 year ago Regarding breakage due to directfs:
If the container spec does not specify a userns, directfs automatically adds a new userns with identity mappings (0:0:4294967295). I think this forced usage of userns is causing the podman issue with directfs. This is inline with findings above (i.e. --userns=keep-id causing failures even before directfs, specifying --uidmap / --gidmap causing failures). It seems like these issues are only reproducible on older Linux versions. Running uname -r on my Rocky Linux 9.2 VM shows 5.14.0-284.25.1.el9_2.x86_64. Podman works fine with directfs on my 6.4 Linux kernel VM.
Maybe there was a Linux bug related to userns that was fixed?
avagin
commented
1 year ago @blechschmidt Could you try out the next patch?
diff --git a/google3/third_party/gvisor/runsc/cmd/gofer.go b/google3/third_party/gvisor/runsc/cmd/gofer.go
--- a/google3/third_party/gvisor/runsc/cmd/gofer.go
+++ b/google3/third_party/gvisor/runsc/cmd/gofer.go
@@ -422,7 +422,7 @@ func (g *Gofer) setupRootFS(spec *specs.
// If root is a mount point but not read-only, we can change mount options
// to make it read-only for extra safety.
log.Infof("Remounting root as readonly: %q", root)
- flags := uintptr(unix.MS_BIND | unix.MS_REMOUNT | unix.MS_RDONLY | unix.MS_REC)
+ flags := uintptr(unix.MS_BIND | unix.MS_REMOUNT | unix.MS_RDONLY | unix.MS_REC | unix.MS_NOSUID | unix.MS_NODEV)
if err := specutils.SafeMount(root, root, "bind", flags, "", procPath); err != nil {
return fmt.Errorf("remounting root as read-only with source: %q, target: %q, flags: %#x, err: %v", root, root, flags, err)
}With this patch it works for me on 6.1.53-1-MANJARO as well as Rocky Linux on 5.14.0-284.11.1.el9_2.x86_64.
avagin
commented
1 year ago The problem here is that root contains mounts with locked flags:
https://elixir.bootlin.com/linux/latest/source/include/linux/mount.h#L58
such flags can't be cleared and it doesn't depend on the kernel version. This behavior was introduced a long time ago.
# cat /proc/72771/mountinfo
536 533 0:45 / / ro,nosuid,nodev,relatime - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",lowerdir=/var/lib/containers/storage/overlay/l/VEHVXUVKDAANLD3ASGVAXG53JN,upperdir=/var/lib/containers/storage/overlay/b8b3622d3a2ae348f084d4bc458db9d4a780a1975d14365b93263186a44433a7/diff,workdir=/var/lib/containers/storage/overlay/b8b3622d3a2ae348f084d4bc458db9d4a780a1975d14365b93263186a44433a7/work,metacopy=on,volatile
537 536 0:25 /containers/storage/overlay-containers/d4578d2d9a668321667bc6b704742089bf019135c66def3182aedbaf4a3454ee/userdata/resolv.conf /etc/resolv.conf rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=6520784k,nr_inodes=819200,mode=755,inode64
538 536 0:25 /containers/storage/overlay-containers/d4578d2d9a668321667bc6b704742089bf019135c66def3182aedbaf4a3454ee/userdata/hosts /etc/hosts rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=6520784k,nr_inodes=819200,mode=755,inode64
539 536 0:44 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",size=64000k,inode64
540 536 0:25 /containers/storage/overlay-containers/d4578d2d9a668321667bc6b704742089bf019135c66def3182aedbaf4a3454ee/userdata/.containerenv /run/.containerenv rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=6520784k,nr_inodes=819200,mode=755,inode64
541 536 0:25 /containers/storage/overlay-containers/d4578d2d9a668321667bc6b704742089bf019135c66def3182aedbaf4a3454ee/userdata/run/secrets /run/secrets rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=6520784k,nr_inodes=819200,mode=755,inode64
542 536 0:25 /containers/storage/overlay-containers/d4578d2d9a668321667bc6b704742089bf019135c66def3182aedbaf4a3454ee/userdata/hostname /etc/hostname rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,size=6520784k,nr_inodes=819200,mode=755,inode64
so I think there's a regression in a recent change?
Originally posted by @fishy in https://github.com/google/gvisor/issues/311#issuecomment-1481590310