`podman ps` command and RESTful api stucking forerver when there is a conmon process got stucking

[ttys3]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

when there is a conmon process got stucking, podman ps command and other command like podman info will also got stuck. the rest api also got stuck. the stuck will not resolve if the stucking conmon process not get killed.

Steps to reproduce the issue:

1. download scripts podman-create-container-via-api.sh and podman-start-container-via-api.sh

curl -LO https://gist.github.com/ttys3/6c3ac108e55e434842cd9797e13ffc15/raw/91d6dde6d3e7429b0361c818587df913f712e99f/podman-create-container-via-api.sh

curl -LO https://gist.github.com/ttys3/6c3ac108e55e434842cd9797e13ffc15/raw/91d6dde6d3e7429b0361c818587df913f712e99f/podman-start-container-via-api.sh

2. chmod a+rx podman-create-container-via-api.sh podman-start-container-via-api.sh

3. ensure you system have jq (https://github.com/stedolan/jq) installed, otherwise you have to mod the scripts to avoid jq

4. create the container, run

sudo ./podman-create-container-via-api.sh 

5. try to start the container

sudo ./podman-start-container-via-api.sh

Describe the results you received:

1. the api call exceeded 60 seconds timeout

   
   exists container found, id=0f44d6fb63699f5de959f0f5e8fa0588219ed88bbd6d004028ab88ad14024239

2022-02-03 16:58:15 |=================== begin POST /v1.0.0/libpod/containers/0f44d6fb63699f5de959f0f5e8fa0588219ed88bbd6d004028ab88ad14024239/start ===================>>>

curl: (28) Operation timed out after 60001 milliseconds with 0 bytes received start container failed, timeout is 60s now there is one conmon process got stuck forever... root 132608 0.0 0.0 8060 3584 ? S 16:58 0:00 /usr/bin/conmon --api-version 1 -c 0f44d6fb63699f5de959f0f5e8fa0588219ed88bbd6d004028ab88ad14024239 -u 0f44d6fb63699f5de959f0f5e8fa0588219ed88bbd6d004028ab88ad14024239 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/0f44d6fb63699f5de959f0f5e8fa0588219ed88bbd6d004028ab88ad14024239/userdata -p /run/containers/storage/overlay-containers/0f44d6fb63699f5de959f0f5e8fa0588219ed88bbd6d004028ab88ad14024239/userdata/pidfile -n redis-demo --exit-dir /run/libpod/exits --full-attach -s -l k8s-file:/var/lib/containers/redis-demo.fifo --log-level debug --syslog --conmon-pidfile /run/containers/storage/overlay-containers/0f44d6fb63699f5de959f0f5e8fa0588219ed88bbd6d004028ab88ad14024239/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 0f44d6fb63699f5de959f0f5e8fa0588219ed88bbd6d004028ab88ad14024239

2. if you try to run `sudo podman info` or `sudo podman ps`, you got stuck forever

3. kill the stucking conmon process: `sudo kill -9 132608`, then `sudo podman info` or `sudo podman ps` works again

**Describe the results you expected:**

 run `sudo podman info` or `sudo podman ps` got no stuck

**Additional information you deem important (e.g. issue happens only occasionally):**

**Output of `podman version`:**

Version: 3.4.4 API Version: 3.4.4 Go Version: go1.17.4 Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933 Built: Fri Dec 10 02:30:40 2021 OS/Arch: linux/amd64

**Output of `podman info --debug`:**

host: arch: amd64 buildahVersion: 1.23.1 cgroupControllers:

• cpuset
• cpu
• io
• memory
• hugetlb
• pids
• rdma
• misc cgroupManager: systemd cgroupVersion: v2 conmon: package: /usr/bin/conmon is owned by conmon 1:2.1.0-1 path: /usr/bin/conmon version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c' cpus: 16 distribution: distribution: arch version: unknown eventLogger: journald hostname: wudeng idMappings: gidmap: null uidmap: null kernel: 5.16.5-arch1-1 linkmode: dynamic logDriver: journald memFree: 22366375936 memTotal: 33594998784 ociRuntime: name: crun package: /usr/bin/crun is owned by crun 1.4.2-1 path: /usr/bin/crun version: |- crun version 1.4.2 commit: f6fbc8f840df1a414f31a60953ae514fa497c748 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: exists: true path: /run/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true seccompProfilePath: /etc/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1 version: |- slirp4netns version 1.1.12 commit: 7a104a101aa3278a2152351a082a6df71f57c9a3 libslirp: 4.6.1 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.3 swapFree: 17179865088 swapTotal: 17179865088 uptime: 50m 25.99s plugins: log:
• k8s-file
• none
• journald network:
• bridge
• macvlan volume:
• local registries: hub.k8s.lan: Blocked: false Insecure: true Location: hub.k8s.lan MirrorByDigestOnly: false Mirrors: null Prefix: hub.k8s.lan search:
• docker.io store: configFile: /etc/containers/storage.conf containerStore: number: 3 paused: 0 running: 2 stopped: 1 graphDriverName: overlay graphOptions: overlay.mountopt: nodev graphRoot: /var/lib/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageStore: number: 79 runRoot: /run/containers/storage volumePath: /var/lib/containers/storage/volumes version: APIVersion: 3.4.4 Built: 1639074640 BuiltTime: Fri Dec 10 02:30:40 2021 GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933 GoVersion: go1.17.4 OsArch: linux/amd64 Version: 3.4.4

**Package info (e.g. output of `rpm -q podman` or `apt list podman`):**

Name : podman Version : 3.4.4-1 Description : Tool and library for running OCI-based containers in pods Architecture : x86_64 URL : https://github.com/containers/podman Licenses : Apache Groups : None Provides : None Depends On : cni-plugins conmon containers-common crun fuse-overlayfs iptables libdevmapper.so=1.02-64 libgpgme.so=11-64 libseccomp.so=2-64 slirp4netns Optional Deps : apparmor: for AppArmor support btrfs-progs: support btrfs backend devices [installed] catatonit: --init flag support podman-docker: for Docker-compatible CLI Required By : cockpit-podman podman-compose Optional For : None Conflicts With : None Replaces : None Installed Size : 72.79 MiB Packager : David Runge dvzrv@archlinux.org Build Date : Fri 10 Dec 2021 02:30:40 AM CST Install Date : Thu 03 Feb 2022 02:45:48 AM CST Install Reason : Explicitly installed Install Script : No Validated By : Signature

**Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)**

Yes

**Additional environment details (AWS, VirtualBox, physical, etc.):**

system memory 32GiB
cpu 8 core 16 threads
cpu freq > 4.0Ghz

why you need a named pipe as log file ? this is because:  https://github.com/containers/podman/issues/13081

this new open issue here is to avoid any condition related to `nomad` or `nomad-driver-podman`.

the reason why `conmon` got stuck is because `conmon` opens log file in blocking mode for writing, so when it got a named fifo pipe,
it will wait forever until there is one reader.

about podman version: **both v3.4.4 latest stable and v4.x (main branch) has the same problem**

[mheon]

The core issue here, perhaps, is that we trust Conmon (and thus the OCI runtime) to run quickly, such that they can be run in a critical section without blocking the lock for long. This does not seem to be a sane assumption under all circumstances - see this issue, but also the previous way we did sdnotify (which blocked at this point until the container was actually fully up).

We can potentially address this by tightening the container start timeout, which right now is absurdly long (I want to say 10 minutes?), such that we just declare a container as "failed" if it takes longer than, say, 30 seconds; this is still an absurdly long time, but it should be within most timeouts, so things should theoretically proceed. We probably also want to kill Conmon in these cases, so the container doesn't continue to start in the background after we declare it has failed.

This also seems like something we could more easily address with the conmon-rs effort - if we're restructuring Conmon, handling these things on the Conmon side and writing an API that can't block us for multiple minutes when we try to start a container seems like a good addition.

[Luap99]

I think we had the same issue for container stop, right? We fixed that by adding the stopping state so we could unlock while we wait for the oci runtime. It should be possible to do something like this for start.

[mheon]

That gets a bit more complicated... What if the user wants to stop a starting container? We'd have to have some sort of way to kill the starting OCI runtime and all associated processes before it's actually fully spun-up.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[Arno500]

In the meantime, do you have any suggestion to workaround the issue, either with a systemd directive to orchestrate stuff a bit better, or anything else? It's kind of handicapping not being able to restart a node when using Nomad :/

Also this issue is hard to debug because there is not much logs

Thank you very much!

[ttys3]

@Arno500 here's the real reason why this happend: https://github.com/containers/podman/issues/13081#issuecomment-1053601112

a simple solution is, patch conmon to with O_NONBLOCK set when it is openning a log

[Arno500]

Would it be worth it to make a PR from your fork to nomad-plugin-podman in this case?