juhp
commented
1 year ago Hm I can't reproduce in either of my Silverblue 37 and 38 VMs fwiw. I wonder how old your toolboxes are?
Closed tkockler closed 10 months ago
juhp
commented
1 year ago Hm I can't reproduce in either of my Silverblue 37 and 38 VMs fwiw. I wonder how old your toolboxes are?
tkockler
commented
1 year ago Interesting. Perhaps a problem with my machine.
I am new to silverblue, so it's a fresh install and the only things I did was playing around with toolboxes: creating, installing stuff and deleting them again.
The toolboxes are new (one and two days old) and I can switch between the working and the not working deployment and reproduce the error
tkockler
commented
1 year ago I tried to start the container with podman directly and got the following error message: "Error: unable to start container 8a48d59ee3223c8240d7a6dfd14e3af0cd4506ef394968cbf8307f40fe6c936a: crun: setrlimit RLIMIT_NPROC: Operation not permitted: OCI permission denied"
With this information I found: https://github.com/containers/podman/issues/6389 in the podman github. I only understand a fraction but:
tom@fedora38-sb:~$ podman inspect --format '{{ printf "%+v" .HostConfig.Ulimits }}' 8a48d59ee322 [{Name:RLIMIT_NOFILE Soft:524288 Hard:524288} {Name:RLIMIT_NPROC Soft:30726 Hard:30726}]
ulimit -u yields in the working deployment 30726 and 30725 in the not working one.
The described solution to set nproc for my user to 30726 worked: I can now start the container in both deployments.
Has anyone an idea how and why the limit changed for my user from 38.20230421.0 to 38.20230422.1?
travier
commented
1 year ago tkockler
commented
1 year ago Yes, creating a backup and restore it works also. The new container gets the new limits of my user account (30725).
cajus
commented
1 year ago Same issue here. But with distrobox. Backup/restore did not work as in the comments above (looks like I need to do something different for distrobox). Shredded the container and re-created it.
sandorex
commented
1 year ago I've had this happen twice, both distrobox and toolbox have same problems, hope this helps to find the issue
I've gone from silverblue 38.20230520.0 to kinoite 38.20230525.0 with following changes, and i had to recreate all containers cause of this error
Now it broke again after i've upgraded to 38.20230527.0 with following changes
The command from before, the limit is not as low as i thought it would be
% podman inspect --format '{{ printf "%+v" .HostConfig.Ulimits }}' cec460321adf
[{Name:RLIMIT_NOFILE Soft:524288 Hard:524288} {Name:RLIMIT_NPROC Soft:111318 Hard:111318}]Full rpm-ostree status
sandorex
commented
1 year ago I may have found a solution without having to recreate the containers, if someone else could confirm if this works for them too
I have booted back into kinoite 38.20230525.0 and even though the containers were created using that specific deployment they did not work, but running ulimit -a -S and ulimit -a -H showed different values than kinoite 38.20230527.0
For some reason hard limit for -u: processes was same as the soft limit of 111292 and when i increased it by adding following to /etc/security/limits.d/90-nproc.conf the containers worked fine
# this is an arbitrary number (3 times the original 111292)
* hard nproc 445168
rgolangh
commented
1 year ago Got the same on fedora silverblue 38 and a toolbox from a fedora 38 image. To work around it I exported the container and recreated the toolbox from that image.
podman container export fedora-toolbox-38 -o toolbox-f38.tar
podman import toolbox-38.tar toolbox-f38
toolbox create --image toolbox-f38I didn't have any extra time to debug but would be happy to supply more info
18:13 $ uname -a
Linux 6.2.15-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 11 17:37:39 UTC 2023 x86_64 GNU/Linux
18:14 $ rpm -q podman toolbox
podman-4.5.0-1.fc38.x86_64
toolbox-0.0.99.4-1.fc38.x86_64
@rgolangh could you check if ulimits changed you between updates?
ulimit -a -S and ulimit -a -H on both deployments
Cydox
commented
1 year ago Upstream issue: https://github.com/containers/podman/issues/18714 WIP fix: https://github.com/containers/podman/pull/18721
Cydox
commented
1 year ago Upstream fix got merged https://github.com/containers/podman/issues/18714
travier
commented
1 year ago Thanks a lot @Cydox !
juhp
commented
11 months ago I overrode podman to 4.6.0-1.fc38 - all my (old) fedora toolbox containers seemed to have survived... but my recent f39 distrobox still hit this issue, sadly...
Cydox
commented
11 months ago @juhp What matters is the podman version that the container was created with. If your f39 distrobox was created with a podman version without the fix, it will still break if the ulimit -u decreases compared to when the container got created.
juhp
commented
11 months ago Thanks @Cydox for clarifying, much appreciated - was afraid that might be the case.
ffseq
commented
10 months ago I also am having the same problem on ublue + distrobox. None of my containers would start, followed by the same errors. I used @sandorex's podman --inspect command and found my ulimits nprocs were off by 5:
Host: 127399
podman: 127404
Changing my /etc/security/limits.conf nproc to 127404 fixed it for me.
tpopela
commented
10 months ago Duplicate of https://github.com/containers/toolbox/issues/1312
ikke-t
commented
10 months ago This makes it work differently, it leads to this:
mount: /etc/machine-id: must be superuser to use mount.
dmesg(1) may have more information after failed mount system call.
Error: failed to bind /etc/machine-id to /run/host/etc/machine-idThis is another issue. Please file another issue here or upstream toolbox for investigation and tracking.
travier
commented
10 months ago This is fixed for new containers with podman 4.6 which landed in F37+. The workarounds for existing containers are in the first comment at the top. Closing.
Elrondo46
commented
4 months ago Problem returned with last version, same messsage. Have to recreate all pods to fix after kernel install
travier
commented
4 months ago Please file a new issue referencing this one
Workarounds
The issue is fixed in podman 4.6 but containers created before this release can not be fixed without being re-created or raising the
nproclimit.Save and restore containers
You can export and import your existing containers. See for example:
Set a higher ulimit for your user
Write the following file, replacing
usernameby your user and150000by a value larger thanulimut -u:Reboot to apply the configuration change.
Original issue text
Describe the bug After upgrading to 38.20230422.1 I can no longer enter existing containers
When I roll back to version 38.20230421.0 toolbox works as expected.
Newly created containers in 38.20230422.1 work fine, only existing containers are not working
To Reproduce Please describe the steps needed to reproduce the bug:
Expected behavior "toolbox enter test" brings me into the container
OS version: