search

mgoltzsche / podman-static

static podman binaries and container image
Apache License 2.0
253 stars 60 forks source link

conmon error when using podman v5.2.2 (rootless) #103

Closed HareToAme closed 1 month ago

HareToAme commented 2 months ago

Issue Description

This issue seems to be specific to the machine; the same operation yields different results (success or failure) on different machines. I found some similar issues as follows, but they didn't resolve my error.

For detailed information, please see the text below.

Steps to reproduce the issue

As a non-root user, install Podman using the following commands:

VERSION=v5.2.2
curl -fsSL https://github.com/mgoltzsche/podman-static/releases/download/$VERSION/podman-linux-amd64.tar.gz | tar -xzf -
cp -r podman-linux-amd64/etc/* ~/.config
cp -r podman-linux-amd64/usr/local/* ~/.local/
echo -e "conmon_path = [\"$HOME/.local/lib/podman/conmon\"]\nhelper_binaries_dir = [\"$HOME/.local/lib/podman\"]\n\n[containers]\ninit_path = \"$HOME/.local/lib/podman/catatonit\"" >> ~/.config/containers/containers.conf
sed -e "s|runroot = \".*\"|runroot = \"/run/user/$UID\"|" \
 -e "s|graphroot = \".*\"|graphroot = \"$HOME/.local/lib/containers/storage\"|" \
 -e "s|mount_program = \".*\"|mount_program = \"$HOME/.local/bin/fuse-overlayfs\"|" \
-i ~/.config/containers/storage.conf
echo 'export PATH=$PATH:$HOME/.local/bin' >> ~/.bashrc && source ~/.bashrc

The user then changes graphroot = "/home/xlhuang/.local/lib/containers/storage" to graphroot = "/data/xlhuang/.local/lib/containers/storage". (This change has no effect on the issue)

Attempt to start a container:

podman --log-level debug run --name ubuntu_me -it ubuntu:22.04

Describe the results you received

When executing the above command, the following error occurs:

INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path conmon: open /sys/fs/cgroup/cgroup.subtree_control: permission denied

and finally

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Here is the complete log information.

(base) xlhuang@a100:~$ podman --log-level debug run --name ubuntu_me -it ubuntu:22.04
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --name ubuntu_me -it ubuntu:22.04) 
DEBU[0000] Using conmon: "/home/xlhuang/.local/lib/podman/conmon" 
INFO[0000] Using sqlite as database backend             
DEBU[0000] systemd-logind: Unknown object '/'.          
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /data/xlhuang/.local/lib/containers/storage 
DEBU[0000] Using run root /run/user/1016                
DEBU[0000] Using static dir /data/xlhuang/.local/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1016/libpod/tmp      
DEBU[0000] Using volume path /data/xlhuang/.local/lib/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: ignore_chown_errors=true            
DEBU[0000] overlay: mount_program=/home/xlhuang/.local/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] using runtime "crun" from $PATH: "/home/xlhuang/.local/bin/crun" 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Using OCI runtime "/home/xlhuang/.local/bin/crun" 
INFO[0000] Setting parallel job count to 193            
DEBU[0000] Pulling image ubuntu:22.04 (policy: missing) 
DEBU[0000] Looking up image "ubuntu:22.04" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/home/xlhuang/.config/containers/registries.conf" 
DEBU[0000] Trying "docker.io/library/ubuntu:22.04" ...  
DEBU[0000] parsed reference into "[overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188" 
DEBU[0000] Found image "ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage 
DEBU[0000] Found image "ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage ([overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188) 
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188" 
DEBU[0000] Looking up image "docker.io/library/ubuntu:22.04" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/library/ubuntu:22.04" ...  
DEBU[0000] parsed reference into "[overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188" 
DEBU[0000] Found image "docker.io/library/ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage 
DEBU[0000] Found image "docker.io/library/ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage ([overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188) 
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188" 
DEBU[0000] Looking up image "ubuntu:22.04" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/library/ubuntu:22.04" ...  
DEBU[0000] parsed reference into "[overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188" 
DEBU[0000] Found image "ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage 
DEBU[0000] Found image "ubuntu:22.04" as "docker.io/library/ubuntu:22.04" in local containers storage ([overlay@/data/xlhuang/.local/lib/containers/storage+/run/user/1016:overlay.ignore_chown_errors=true,overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188) 
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188" 
DEBU[0000] Inspecting image 53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188 
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188" 
DEBU[0000] Inspecting image 53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188 
DEBU[0000] Inspecting image 53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188 
DEBU[0000] Inspecting image 53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188 
DEBU[0000] using systemd mode: false                    
DEBU[0000] setting container name ubuntu_me             
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading default seccomp profile              
DEBU[0000] Allocated lock 2 for container c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c 
DEBU[0000] exporting opaque data as blob "sha256:53a843653cbcd9e10be207e951d907dc2481d9c222de57d24cfcac32e5165188" 
DEBU[0000] Created container "c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c" 
DEBU[0000] Container "c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c" has work directory "/data/xlhuang/.local/lib/containers/storage/overlay-containers/c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c/userdata" 
DEBU[0000] Container "c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c" has run directory "/run/user/1016/overlay-containers/c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c/userdata" 
DEBU[0000] Handling terminal attach                     
INFO[0000] Received shutdown.Stop(), terminating!        PID=27146
DEBU[0000] Enabling signal proxying                     
DEBU[0000] overlay: mount_data=lowerdir=/data/xlhuang/.local/lib/containers/storage/overlay/l/CRDDWRJAW5MIO6OMMFIWGFD7UO,upperdir=/data/xlhuang/.local/lib/containers/storage/overlay/9b15d2571579adc76eb9c48292397a6c7eef7c8dfa253a0c174fbd06b359c5c9/diff,workdir=/data/xlhuang/.local/lib/containers/storage/overlay/9b15d2571579adc76eb9c48292397a6c7eef7c8dfa253a0c174fbd06b359c5c9/work,nodev,fsync=0 
DEBU[0000] Mounted container "c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c" at "/data/xlhuang/.local/lib/containers/storage/overlay/9b15d2571579adc76eb9c48292397a6c7eef7c8dfa253a0c174fbd06b359c5c9/merged" 
DEBU[0000] Created root filesystem for container c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c at /data/xlhuang/.local/lib/containers/storage/overlay/9b15d2571579adc76eb9c48292397a6c7eef7c8dfa253a0c174fbd06b359c5c9/merged 
DEBU[0000] Made network namespace at /run/user/1016/netns/netns-5fdfd6cf-5603-21ec-fec3-2c5810fc5927 for container c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c 
DEBU[0000] pasta arguments: --config-net --dns-forward 169.254.0.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1016/netns/netns-5fdfd6cf-5603-21ec-fec3-2c5810fc5927 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Workdir "/" resolved to host path "/data/xlhuang/.local/lib/containers/storage/overlay/9b15d2571579adc76eb9c48292397a6c7eef7c8dfa253a0c174fbd06b359c5c9/merged" 
DEBU[0000] Created OCI spec for container c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c at /data/xlhuang/.local/lib/containers/storage/overlay-containers/c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c/userdata/config.json 
DEBU[0000] /home/xlhuang/.local/lib/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /home/xlhuang/.local/lib/podman/conmon  args="[--api-version 1 -c c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c -u c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c -r /home/xlhuang/.local/bin/crun -b /data/xlhuang/.local/lib/containers/storage/overlay-containers/c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c/userdata -p /run/user/1016/overlay-containers/c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c/userdata/pidfile -n ubuntu_me --exit-dir /run/user/1016/libpod/tmp/exits --persist-dir /run/user/1016/libpod/tmp/persist/c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c --full-attach -l k8s-file:/data/xlhuang/.local/lib/containers/storage/overlay-containers/c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /run/user/1016/overlay-containers/c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c/userdata/conmon.pid --exit-command /zhdd/home/xlhuang/.local/bin/podman --exit-command-arg --root --exit-command-arg /data/xlhuang/.local/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1016 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1016/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /data/xlhuang/.local/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.ignore_chown_errors=true --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/home/xlhuang/.local/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,fsync=0 --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c]"
INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path conmon: open /sys/fs/cgroup/cgroup.subtree_control: permission denied 
DEBU[0000] Cleaning up container c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c 
DEBU[0000] Tearing down network namespace at /run/user/1016/netns/netns-5fdfd6cf-5603-21ec-fec3-2c5810fc5927 for container c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c 
DEBU[0000] Unmounted container "c8de1934106099b98a8e139b0360eb12f5aaf22c3af2c7141dad783f86138f9c" 
DEBU[0000] ExitCode msg: "container create failed (no logs from conmon): conmon bytes \"\": readobjectstart: expect { or n, but found \x00, error found in #0 byte of ...||..., bigger context ...||..." 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
DEBU[0000] Shutting down engines                        
(base) xlhuang@a100:~$

Added --cgroups=disabled parameter to the command:

podman --log-level debug run --cgroups=disabled --name ubuntu_me -it ubuntu:22.04

This prevented the Failed to add conmon to cgroupfs sandbox cgroup... error, but the final error Error: container create failed (no logs from conmon): conmon bytes ""... remained the same.

Describe the results you expected

The same steps executed on a VMware Linux virtual machine (Ubuntu 20.04) as a non-root user worked successfully.

Here is the complete log information.

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --name ubuntu_me -it ubuntu:20.04) 
DEBU[0000] Using conmon: "/home/todokanaikoi/.local/lib/podman/conmon" 
INFO[0000] Using sqlite as database backend             
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/todokanaikoi/.local/lib/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /home/todokanaikoi/.local/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/todokanaikoi/.local/lib/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: ignore_chown_errors=true            
DEBU[0000] overlay: mount_program=/home/todokanaikoi/.local/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] using runtime "crun" from $PATH: "/home/todokanaikoi/.local/bin/crun" 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] using runtime "runc" from $PATH: "/home/todokanaikoi/.local/bin/runc" 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Using OCI runtime "/home/todokanaikoi/.local/bin/crun" 
INFO[0000] Setting parallel job count to 13             
DEBU[0000] Pulling image ubuntu:20.04 (policy: missing) 
DEBU[0000] Looking up image "ubuntu:20.04" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/home/todokanaikoi/.config/containers/registries.conf" 
DEBU[0000] Trying "docker.io/library/ubuntu:20.04" ...  
DEBU[0000] parsed reference into "[overlay@/home/todokanaikoi/.local/lib/containers/storage+/run/user/1000:overlay.ignore_chown_errors=true,overlay.mount_program=/home/todokanaikoi/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba" 
DEBU[0000] Found image "ubuntu:20.04" as "docker.io/library/ubuntu:20.04" in local containers storage 
DEBU[0000] Found image "ubuntu:20.04" as "docker.io/library/ubuntu:20.04" in local containers storage ([overlay@/home/todokanaikoi/.local/lib/containers/storage+/run/user/1000:overlay.ignore_chown_errors=true,overlay.mount_program=/home/todokanaikoi/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba) 
DEBU[0000] exporting opaque data as blob "sha256:9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba" 
DEBU[0000] Looking up image "docker.io/library/ubuntu:20.04" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/library/ubuntu:20.04" ...  
DEBU[0000] parsed reference into "[overlay@/home/todokanaikoi/.local/lib/containers/storage+/run/user/1000:overlay.ignore_chown_errors=true,overlay.mount_program=/home/todokanaikoi/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba" 
DEBU[0000] Found image "docker.io/library/ubuntu:20.04" as "docker.io/library/ubuntu:20.04" in local containers storage 
DEBU[0000] Found image "docker.io/library/ubuntu:20.04" as "docker.io/library/ubuntu:20.04" in local containers storage ([overlay@/home/todokanaikoi/.local/lib/containers/storage+/run/user/1000:overlay.ignore_chown_errors=true,overlay.mount_program=/home/todokanaikoi/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba) 
DEBU[0000] exporting opaque data as blob "sha256:9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba" 
DEBU[0000] Looking up image "ubuntu:20.04" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/library/ubuntu:20.04" ...  
DEBU[0000] parsed reference into "[overlay@/home/todokanaikoi/.local/lib/containers/storage+/run/user/1000:overlay.ignore_chown_errors=true,overlay.mount_program=/home/todokanaikoi/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba" 
DEBU[0000] Found image "ubuntu:20.04" as "docker.io/library/ubuntu:20.04" in local containers storage 
DEBU[0000] Found image "ubuntu:20.04" as "docker.io/library/ubuntu:20.04" in local containers storage ([overlay@/home/todokanaikoi/.local/lib/containers/storage+/run/user/1000:overlay.ignore_chown_errors=true,overlay.mount_program=/home/todokanaikoi/.local/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba) 
DEBU[0000] exporting opaque data as blob "sha256:9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba" 
DEBU[0000] Inspecting image 9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba 
DEBU[0000] exporting opaque data as blob "sha256:9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba" 
DEBU[0000] Inspecting image 9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba 
DEBU[0000] Inspecting image 9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba 
DEBU[0000] Inspecting image 9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba 
DEBU[0000] using systemd mode: false                    
DEBU[0000] setting container name ubuntu_me             
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading default seccomp profile              
DEBU[0000] Allocated lock 0 for container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 
DEBU[0000] exporting opaque data as blob "sha256:9df6d6105df2788299e5cbbf3dfd09aa6b3b5a10b784b214c35a035f76deb4ba" 
DEBU[0000] Created container "a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20" 
DEBU[0000] Container "a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20" has work directory "/home/todokanaikoi/.local/lib/containers/storage/overlay-containers/a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20/userdata" 
DEBU[0000] Container "a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20" has run directory "/run/user/1000/overlay-containers/a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20/userdata" 
DEBU[0000] Handling terminal attach                     
INFO[0000] Received shutdown.Stop(), terminating!        PID=2451
DEBU[0000] Enabling signal proxying                     
DEBU[0000] overlay: mount_data=lowerdir=/home/todokanaikoi/.local/lib/containers/storage/overlay/l/RCITLHZT5YP4FSH36H6SW2BFUG,upperdir=/home/todokanaikoi/.local/lib/containers/storage/overlay/8084091effd0b6b44d1fa319e44858e44a7293f554a143b585f37c4bfd6634da/diff,workdir=/home/todokanaikoi/.local/lib/containers/storage/overlay/8084091effd0b6b44d1fa319e44858e44a7293f554a143b585f37c4bfd6634da/work,nodev,fsync=0 
DEBU[0000] Made network namespace at /run/user/1000/netns/netns-c91eefb3-a9e6-506b-6949-578b19c5253a for container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 
DEBU[0000] pasta arguments: --config-net --dns-forward 169.254.0.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1000/netns/netns-c91eefb3-a9e6-506b-6949-578b19c5253a 
DEBU[0000] Mounted container "a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20" at "/home/todokanaikoi/.local/lib/containers/storage/overlay/8084091effd0b6b44d1fa319e44858e44a7293f554a143b585f37c4bfd6634da/merged" 
DEBU[0000] Created root filesystem for container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 at /home/todokanaikoi/.local/lib/containers/storage/overlay/8084091effd0b6b44d1fa319e44858e44a7293f554a143b585f37c4bfd6634da/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Workdir "/" resolved to host path "/home/todokanaikoi/.local/lib/containers/storage/overlay/8084091effd0b6b44d1fa319e44858e44a7293f554a143b585f37c4bfd6634da/merged" 
DEBU[0000] Created OCI spec for container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 at /home/todokanaikoi/.local/lib/containers/storage/overlay-containers/a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20/userdata/config.json 
DEBU[0000] /home/todokanaikoi/.local/lib/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /home/todokanaikoi/.local/lib/podman/conmon  args="[--api-version 1 -c a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 -u a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 -r /home/todokanaikoi/.local/bin/crun -b /home/todokanaikoi/.local/lib/containers/storage/overlay-containers/a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20/userdata -p /run/user/1000/overlay-containers/a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20/userdata/pidfile -n ubuntu_me --exit-dir /run/user/1000/libpod/tmp/exits --persist-dir /run/user/1000/libpod/tmp/persist/a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 --full-attach -l k8s-file:/home/todokanaikoi/.local/lib/containers/storage/overlay-containers/a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /run/user/1000/overlay-containers/a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20/userdata/conmon.pid --exit-command /home/todokanaikoi/.local/bin/podman --exit-command-arg --root --exit-command-arg /home/todokanaikoi/.local/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/todokanaikoi/.local/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.ignore_chown_errors=true --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/home/todokanaikoi/.local/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,fsync=0 --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20]"
DEBU[0000] Received: 2476                               
INFO[0000] Got Conmon PID as 2474                       
DEBU[0000] Created container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 in OCI runtime 
DEBU[0000] found local resolver, using "/run/systemd/resolve/resolv.conf" to get the nameservers 
DEBU[0000] Attaching to container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 
DEBU[0000] Received a resize event: {Width:130 Height:36} 
DEBU[0000] Starting container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 with command [/bin/bash] 
DEBU[0000] Started container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 
DEBU[0000] Notify sent successfully                     
root@a80b4d7f1ca5:/# DEBU[0010] Sending signal 28 to container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 
DEBU[0010] Received a resize event: {Width:236 Height:60} 
root@a80b4d7f1ca5:/# DEBU[0010] Received a resize event: {Width:231 Height:58} 
DEBU[0010] Sending signal 28 to container a80b4d7f1ca51c4614da99744c06cdbe081a826310e917df471e90ebd384ef20 
root@a80b4d7f1ca5:/# exit

podman info output

(base) xlhuang@a100:~$ podman info
host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /home/xlhuang/.local/lib/podman/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 86.23
    systemPercent: 3.17
    userPercent: 10.6
  cpus: 64
  databaseBackend: sqlite
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: file
  freeLocks: 2048
  hostname: a100
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1016
      size: 1
    - container_id: 1
      host_id: 1345184
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1016
      size: 1
    - container_id: 1
      host_id: 1345184
      size: 65536
  kernel: 5.15.0-100-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 83611553792
  memTotal: 540667334656
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /home/xlhuang/.local/lib/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: Unknown
    path: /home/xlhuang/.local/lib/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: Unknown
    path: /home/xlhuang/.local/bin/crun
    version: |-
      crun version 1.16.1
      commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
      rundir: /run/user/1016/crun
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /home/xlhuang/.local/bin/pasta
    package: Unknown
    version: |
      pasta 2024_06_24.1ee2eca
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1016/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.0.1-2_amd64
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 251960922112
  swapTotal: 270333374464
  uptime: 4441h 53m 54.00s (Approximately 185.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /zhdd/home/xlhuang/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
    overlay.mount_program:
      Executable: /home/xlhuang/.local/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fuse-overlayfs: version 1.13-dev
        fusermount3 version: 3.10.5
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /data/xlhuang/.local/lib/containers/storage
  graphRootAllocated: 7181714976768
  graphRootUsed: 6774170271744
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1016
  transientStore: false
  volumePath: /data/xlhuang/.local/lib/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 0
  BuiltTime: Thu Jan  1 08:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

(base) xlhuang@a100:~$

-----------------------------------------------------------
podman info output of VMware Linux virtual machine (Ubuntu 20.04):

todokanaikoi@todokanaikoi-virtual-machine:~$ podman info
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /home/todokanaikoi/.local/lib/podman/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 93.62
    systemPercent: 4.86
    userPercent: 1.52
  cpus: 4
  databaseBackend: sqlite
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  freeLocks: 2047
  hostname: todokanaikoi-virtual-machine
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-119-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 6469529600
  memTotal: 8285827072
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /home/todokanaikoi/.local/lib/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: Unknown
    path: /home/todokanaikoi/.local/lib/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: Unknown
    path: /home/todokanaikoi/.local/bin/crun
    version: |-
      crun version 1.16.1
      commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /home/todokanaikoi/.local/bin/pasta
    package: Unknown
    version: |
      pasta 2024_06_24.1ee2eca
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 0h 3m 22.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /home/todokanaikoi/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
    overlay.mount_program:
      Executable: /home/todokanaikoi/.local/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fuse-overlayfs: version 1.13-dev
        fusermount3 version: 3.16.2
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /home/todokanaikoi/.local/lib/containers/storage
  graphRootAllocated: 83424108544
  graphRootUsed: 53924687872
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000
  transientStore: false
  volumePath: /home/todokanaikoi/.local/lib/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 0
  BuiltTime: Thu Jan  1 08:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

todokanaikoi@todokanaikoi-virtual-machine:~$

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

The initial error that appeared was:

Error: crun: mount proc to /proc: Operation not permitted: OCI permission denied

However, after trying again, the error consistently became:

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

I have reason to believe that this issue is related to some changes on the server. Initially, I was using podman-4.2.1-r1-x86_64.AppImage on the server without any problems, but one day it suddenly stopped working and the aforementioned error occurred. Therefore, I tried using the latest version 5.2.2, but the same error still appeared on that server. I am unable to determine the cause of the problem.

Since I belong to the docker group on the server, my temporary solution is to replace podman with docker to execute the startup command. This will not cause any errors.

HareToAme commented 1 month ago

This issue has been successfully resolved at https://github.com/containers/podman/issues/23952.

I will close this issue, and I hope that example will be helpful to anyone with a similar situation.