mheon
commented
4 years ago Host upgrades are more likely to have caused this - any chance Podman was upgraded as part of this?
Closed mkoura closed 4 years ago
mheon
commented
4 years ago Host upgrades are more likely to have caused this - any chance Podman was upgraded as part of this?
mkoura
commented
4 years ago Yes, podman was upgraded from podman-2:2.0.4-1.fc32 to podman-2.0.5-1.fc32. The problem is still the same after downgrading to 1.8.2-2.fc32 (2.0.4 is no longer available in Fedora 32 repos).
mheon
commented
4 years ago Any chance you re-created the Toolbox container as part of this upgrade?
mkoura
commented
4 years ago I don't think so. The container was created about two months ago and it was running during the host's upgrade. I found out this morning after host's reboot that I can no longer start the container.
mheon
commented
4 years ago Can you include the output of podman inspect on the container?
mkoura
commented
4 years ago $ podman inspect fedora-toolbox-32
[
{
"Id": "2dbf302ca7ed9144f455c62ae3c53272494049ba64798d5461cc2ade56d111c1",
"Created": "2020-06-30T10:43:34.139954351+02:00",
"Path": "toolbox",
"Args": [
"--verbose",
"init-container",
"--home",
"/home/martink",
"--monitor-host",
"--shell",
"/bin/bash",
"--uid",
"1000",
"--user",
"martink"
],
"State": {
"OciVersion": "1.0.1-dev",
"Status": "configured",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "2020-08-24T13:14:41.10709109+02:00",
"FinishedAt": "2020-07-30T22:22:17.645155367+02:00",
"Healthcheck": {
"Status": "",
"FailingStreak": 0,
"Log": null
}
},
"Image": "7e4d1b69a35811974831b90fa4f3ac855a348740c5061d277ae015cd68c5f720",
"ImageName": "registry.fedoraproject.org/f32/fedora-toolbox:32",
"Rootfs": "",
"Pod": "",
"ResolvConfPath": "",
"HostnamePath": "",
"HostsPath": "",
"StaticDir": "/home/martink/.local/share/containers/storage/overlay-containers/2dbf302ca7ed9144f455c62ae3c53272494049ba64798d5461cc2ade56d111c1/userdata",
"OCIConfigPath": "/home/martink/.local/share/containers/storage/overlay-containers/2dbf302ca7ed9144f455c62ae3c53272494049ba64798d5461cc2ade56d111c1/userdata/config.json",
"OCIRuntime": "runc",
"LogPath": "/home/martink/.local/share/containers/storage/overlay-containers/2dbf302ca7ed9144f455c62ae3c53272494049ba64798d5461cc2ade56d111c1/userdata/ctr.log",
"LogTag": "",
"ConmonPidFile": "/run/user/1000/containers/overlay-containers/2dbf302ca7ed9144f455c62ae3c53272494049ba64798d5461cc2ade56d111c1/userdata/conmon.pid",
"Name": "fedora-toolbox-32",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "system_u:object_r:container_file_t:s0:c321,c667",
"ProcessLabel": "",
"AppArmorProfile": "",
"EffectiveCaps": null,
"BoundingCaps": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"ExecIDs": [],
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/home/martink/.local/share/containers/storage/overlay/37e5b6450989c8f85751e70aa56f52bb8d148dd1465dd068b6dba8a5c581488c/diff:/home/martink/.local/share/containers/storage/overlay/de421654540d334c4dce7c1f432cb6cf6b4b0459bda054dad259da4424117c46/diff",
"UpperDir": "/home/martink/.local/share/containers/storage/overlay/e1dfdac63818a1cb12291b0db85e7d271b648cde526b9c360a9912f106d60e40/diff",
"WorkDir": "/home/martink/.local/share/containers/storage/overlay/e1dfdac63818a1cb12291b0db85e7d271b648cde526b9c360a9912f106d60e40/work"
}
},
"Mounts": [
{
"Type": "bind",
"Name": "",
"Source": "/home/martink",
"Destination": "/home/martink",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Name": "",
"Source": "/run/media",
"Destination": "/run/media",
"Driver": "",
"Mode": "",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Name": "",
"Source": "/run/user/1000",
"Destination": "/run/user/1000",
"Driver": "",
"Mode": "",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Name": "",
"Source": "/run/user/1000/.flatpak-helper/monitor",
"Destination": "/run/host/monitor",
"Driver": "",
"Mode": "",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Name": "",
"Source": "/home/martink/Source/repos/toolbox/toolbox",
"Destination": "/usr/bin/toolbox",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Name": "",
"Source": "/run/dbus/system_bus_socket",
"Destination": "/run/dbus/system_bus_socket",
"Driver": "",
"Mode": "",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Name": "",
"Source": "/media",
"Destination": "/media",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Name": "",
"Source": "/dev",
"Destination": "/dev",
"Driver": "",
"Mode": "",
"Options": [
"nosuid",
"rbind"
],
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Name": "",
"Source": "/run",
"Destination": "/run/host/run",
"Driver": "",
"Mode": "",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Name": "",
"Source": "/var",
"Destination": "/run/host/var",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Name": "",
"Source": "/run/.heim_org.h5l.kcm-socket",
"Destination": "/run/.heim_org.h5l.kcm-socket",
"Driver": "",
"Mode": "",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Name": "",
"Source": "/etc",
"Destination": "/run/host/etc",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Name": "",
"Source": "/usr",
"Destination": "/run/host/usr",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Name": "",
"Source": "/mnt",
"Destination": "/mnt",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": true,
"Propagation": "rslave"
},
{
"Type": "bind",
"Name": "",
"Source": "/etc/profile.d/toolbox.sh",
"Destination": "/etc/profile.d/toolbox.sh",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Name": "",
"Source": "/tmp",
"Destination": "/run/host/tmp",
"Driver": "",
"Mode": "",
"Options": [
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rslave"
}
],
"Dependencies": [],
"NetworkSettings": {
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": ""
},
"ExitCommand": [
"/usr/bin/podman",
"--root",
"/home/martink/.local/share/containers/storage",
"--runroot",
"/run/user/1000/containers",
"--log-level",
"error",
"--cgroup-manager",
"cgroupfs",
"--tmpdir",
"/run/user/1000/libpod/tmp",
"--runtime",
"runc",
"--storage-driver",
"overlay",
"--storage-opt",
"overlay.mount_program=/usr/bin/fuse-overlayfs",
"--events-backend",
"file",
"container",
"cleanup",
"2dbf302ca7ed9144f455c62ae3c53272494049ba64798d5461cc2ade56d111c1"
],
"Namespace": "",
"IsInfra": false,
"Config": {
"Hostname": "toolbox",
"Domainname": "",
"User": "root:root",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=toolbox",
"VERSION=32",
"DISTTAG=f32container",
"FGC=f32",
"NAME=fedora-toolbox",
"TOOLBOX_PATH=/home/martink/Source/repos/toolbox/toolbox",
"container=oci",
"HOME=/root"
],
"Cmd": [
"toolbox",
"--verbose",
"init-container",
"--home",
"/home/martink",
"--monitor-host",
"--shell",
"/bin/bash",
"--uid",
"1000",
"--user",
"martink"
],
"Image": "registry.fedoraproject.org/f32/fedora-toolbox:32",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": "",
"OnBuild": null,
"Labels": {
"architecture": "x86_64",
"authoritative-source-url": "registry.fedoraproject.org",
"build-date": "2020-06-25T19:04:33.606928",
"com.github.containers.toolbox": "true",
"com.github.debarshiray.toolbox": "true",
"com.redhat.build-host": "osbs-node02.iad2.fedoraproject.org",
"com.redhat.component": "fedora-toolbox",
"distribution-scope": "public",
"license": "MIT",
"maintainer": "Debarshi Ray \u003crishi@fedoraproject.org\u003e",
"name": "f32/fedora-toolbox",
"release": "6",
"summary": "Base image for creating Fedora toolbox containers",
"usage": "This image is meant to be used with the toolbox command",
"vcs-ref": "366895197f89cf425e596faa006257d2c796c313",
"vcs-type": "git",
"vendor": "Fedora Project",
"version": "32"
},
"Annotations": {
"io.container.manager": "libpod",
"io.kubernetes.cri-o.Created": "2020-06-30T10:43:34.139954351+02:00",
"io.kubernetes.cri-o.TTY": "false",
"io.podman.annotations.autoremove": "FALSE",
"io.podman.annotations.init": "FALSE",
"io.podman.annotations.label": "disable",
"io.podman.annotations.privileged": "TRUE",
"io.podman.annotations.publish-all": "FALSE",
"org.opencontainers.image.stopSignal": "15"
},
"StopSignal": 15,
"CreateCommand": [
"podman",
"create",
"--dns",
"none",
"--env",
"TOOLBOX_PATH=/home/martink/Source/repos/toolbox/toolbox",
"--group-add",
"wheel",
"--hostname",
"toolbox",
"--ipc",
"host",
"--label",
"com.github.containers.toolbox=true",
"--label",
"com.github.debarshiray.toolbox=true",
"--name",
"fedora-toolbox-32",
"--network",
"host",
"--no-hosts",
"--pid",
"host",
"--privileged",
"--security-opt",
"label=disable",
"--ulimit",
"host",
"--userns=keep-id",
"--user",
"root:root",
"--volume",
"/run/.heim_org.h5l.kcm-socket:/run/.heim_org.h5l.kcm-socket",
"--volume",
"/media:/media:rslave",
"--volume",
"/mnt:/mnt:rslave",
"--volume",
"/run/media:/run/media:rslave",
"--volume",
"/etc/profile.d/toolbox.sh:/etc/profile.d/toolbox.sh:ro",
"--volume",
"/home/martink/Source/repos/toolbox/toolbox:/usr/bin/toolbox:ro",
"--volume",
"/run/user/1000:/run/user/1000",
"--volume",
"/run/user/1000/.flatpak-helper/monitor:/run/host/monitor",
"--volume",
"/run/dbus/system_bus_socket:/run/dbus/system_bus_socket",
"--volume",
"/home/martink:/home/martink:rslave",
"--volume",
"/etc:/run/host/etc",
"--volume",
"/dev:/dev:rslave",
"--volume",
"/run:/run/host/run:rslave",
"--volume",
"/tmp:/run/host/tmp:rslave",
"--volume",
"/usr:/run/host/usr:rw,rslave",
"--volume",
"/var:/run/host/var:rslave",
"registry.fedoraproject.org/f32/fedora-toolbox:32",
"toolbox",
"--verbose",
"init-container",
"--home",
"/home/martink",
"--monitor-host",
"--shell",
"/bin/bash",
"--uid",
"1000",
"--user",
"martink"
]
},
"HostConfig": {
"Binds": [
"/home/martink:/home/martink:rslave,rw,rbind",
"/run/media:/run/media:rslave,rw,nosuid,nodev,rbind",
"/run/user/1000:/run/user/1000:rw,rprivate,nosuid,nodev,rbind",
"/run/user/1000/.flatpak-helper/monitor:/run/host/monitor:rw,rprivate,nosuid,nodev,rbind",
"/home/martink/Source/repos/toolbox/toolbox:/usr/bin/toolbox:ro,rprivate,rbind",
"/run/dbus/system_bus_socket:/run/dbus/system_bus_socket:rw,rprivate,nosuid,nodev,rbind",
"/media:/media:rslave,rw,rbind",
"/dev:/dev:rslave,rw,nosuid,rbind",
"/run:/run/host/run:rslave,rw,nosuid,nodev,rbind",
"/var:/run/host/var:rslave,rw,rbind",
"/run/.heim_org.h5l.kcm-socket:/run/.heim_org.h5l.kcm-socket:rw,rprivate,nosuid,nodev,rbind",
"/etc:/run/host/etc:rw,rprivate,rbind",
"/usr:/run/host/usr:rw,rslave,rbind",
"/mnt:/mnt:rslave,rw,rbind",
"/etc/profile.d/toolbox.sh:/etc/profile.d/toolbox.sh:ro,rprivate,rbind",
"/tmp:/run/host/tmp:rslave,rw,nosuid,nodev,rbind"
],
"CgroupMode": "host",
"ContainerIDFile": "",
"LogConfig": {
"Type": "k8s-file",
"Config": null
},
"NetworkMode": "host",
"PortBindings": {},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [],
"CapDrop": [],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": [
"wheel"
],
"IpcMode": "host",
"Cgroup": "",
"Cgroups": "default",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "host",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"label=disable"
],
"Tmpfs": {},
"UTSMode": "private",
"UsernsMode": "private",
"ShmSize": 65536000,
"Runtime": "oci",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": 0,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": [
{
"Name": "RLIMIT_NOFILE",
"Soft": 524288,
"Hard": 524288
},
{
"Name": "RLIMIT_NPROC",
"Soft": 63397,
"Hard": 63397
}
],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
}
}
]Hmmm. Those rlimits don't seem like defaults - I wonder if the system rlimits changed, and that's causing us to be unable to start?
@rhatdan Any idea how we'd check that?
mkoura
commented
4 years ago Not sure where the ulimits values come from. Originally I just used toolbox create to create the container. On newly created container the ulimits are empty.
mheon
commented
4 years ago There was a brief period where Podman automatically created them, but if Podman added them, it would always have failed unless the system configuration changed, from my understanding...
mheon
commented
4 years ago Any chance you can downgrade to the previous Podman and check if it still works?
mkoura
commented
4 years ago I downgraded to podman-2.0.4-1.fc32 and it doesn't work. Probably kernel upgrade changed limit defaults, my current limits are:
$ ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63388
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 524288
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 63388
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimitedHm. NOFILE in Podman is set to 524288 and your system has 524288 set, so that's fine. NPROC, on the other hand, is set to 63397 in Podman and 63388 on your system - the Podman one is larger. That's probably the error there.
mkoura
commented
4 years ago Right. I was finally able to increase the NPROC limits and the container is able to start. Thanks for your help @mheon :+1:
rhatdan
commented
4 years ago Those values would only be set on a broken version of Podman from a while ago. I don't believe they are set any longer. We really can not fix this container at this point, other then destroying it and recreating it. Since you were able to reset your user account rlimits to make this work, I am going to close the issue.
/kind bug
Description
I have a container created using toolbox. After upgrading system inside the container using
dnf update, I can no longer start the container. I have also upgraded host's system and new kernel version was installed, not sure what caused the issue.Might be related to https://github.com/containers/podman/issues/7466 although I'm not running Fedora CoreOS.
Steps to reproduce the issue:
dnf upgradepodman --log-level debug start --attach fedora-toolbox-32Describe the results you received:
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
physical