Container Crashes, No Anomalies

Issue Description

$ podman run -d --name mariadb --pod zm -v /home/bill/nginx/etc/mariadb:/etc/my.cnf.d:Z -v /home/bill/mariadb:/var/lib/mysql:z -v /home/bill/zm/sql_build:/usr/share/zoneminder/db:z --rm --tz=local --log-level debug --tls-verify=false localhost:5000/mariadb INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run -d --name mariadb --pod zm -v /home/bill/nginx/etc/mariadb:/etc/my.cnf.d:Z -v /home/bill/mariadb:/var/lib/mysql:z -v /home/bill/zm/sql_build:/usr/share/zoneminder/db:z --rm --tz=local --log-level debug --tls-verify=false localhost:5000/mariadb) DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/bill/.local/share/containers/storage DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/bill/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/bill/.local/share/containers/storage/volumes DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is not being used DEBU[0000] Cached value indicated that native-diff is usable DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 25
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image localhost:5000/mariadb (policy: missing) DEBU[0000] Looking up image "localhost:5000/mariadb" in local containers storage DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Trying "localhost:5000/mariadb:latest" ...
DEBU[0000] reference "[overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]localhost:5000/mariadb:latest" does not resolve to an image ID DEBU[0000] Trying "localhost:5000/mariadb:latest" ...
DEBU[0000] reference "[overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]localhost:5000/mariadb:latest" does not resolve to an image ID DEBU[0000] Trying "localhost:5000/mariadb" ...
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000] Attempting to pull candidate localhost:5000/mariadb:latest for localhost:5000/mariadb DEBU[0000] parsed reference into "[overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]localhost:5000/mariadb:latest" Trying to pull localhost:5000/mariadb:latest... DEBU[0000] Copying source image //localhost:5000/mariadb:latest to destination image [overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]localhost:5000/mariadb:latest DEBU[0000] Using registries.d directory /etc/containers/registries.d DEBU[0000] Trying to access "localhost:5000/mariadb:latest" DEBU[0000] No credentials matching localhost:5000/mariadb found in /run/user/1000/containers/auth.json DEBU[0000] No credentials matching localhost:5000/mariadb found in /home/bill/.config/containers/auth.json DEBU[0000] No credentials matching localhost:5000/mariadb found in /home/bill/.docker/config.json DEBU[0000] No credentials matching localhost:5000/mariadb found in /home/bill/.dockercfg DEBU[0000] No credentials for localhost:5000/mariadb found DEBU[0000] Lookaside configuration: using "default-docker" configuration DEBU[0000] No signature storage configuration found for localhost:5000/mariadb:latest, using built-in default file:///home/bill/.local/share/containers/sigstore DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/localhost:5000 DEBU[0000] Sigstore attachments: using "default-docker" configuration DEBU[0000] GET https://localhost:5000/v2/
DEBU[0000] Ping https://localhost:5000/v2/ err Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://localhost:5000/v2/", Err:(*errors.errorString)(0x561db8044600)}) DEBU[0000] GET http://localhost:5000/v2/
DEBU[0000] Ping http://localhost:5000/v2/ status 200
DEBU[0000] GET http://localhost:5000/v2/mariadb/manifests/latest DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" DEBU[0000] Using SQLite blob info cache at /home/bill/.local/share/containers/cache/blob-info-cache-v1.sqlite DEBU[0000] IsRunningImageAllowed for image docker:localhost:5000/mariadb:latest DEBU[0000] Using default policy section
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
DEBU[0000] Downloading /v2/mariadb/blobs/sha256:cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c DEBU[0000] GET http://localhost:5000/v2/mariadb/blobs/sha256:cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c Getting image source signatures DEBU[0000] Reading /home/bill/.local/share/containers/sigstore/mariadb@sha256=2acab763afd7fa215dd69020965c9fde71c8eab93fb5347c9843dace49eb8828/signature-1 DEBU[0000] Not looking for sigstore attachments: disabled by configuration DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json] DEBU[0000] ... will first try using the original manifest unmodified DEBU[0000] Checking if we can reuse blob sha256:d910bff0e279726b3b3fe05c3e2680dd786efd22456f92b62c0dc75b73c32719: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true DEBU[0000] Checking if we can reuse blob sha256:94d7953f76d275a6e79eef8adaaa8dc590eabae3be5266b8f2cab7cd7ff407f0: general substitution = true, compression for MIME type "application/vnd.oci.image.layer.v1.tar+gzip" = true DEBU[0000] Failed to retrieve partial blob: convert_images not configured DEBU[0000] Downloading /v2/mariadb/blobs/sha256:d910bff0e279726b3b3fe05c3e2680dd786efd22456f92b62c0dc75b73c32719 DEBU[0000] GET http://localhost:5000/v2/mariadb/blobs/sha256:d910bff0e279726b3b3fe05c3e2680dd786efd22456f92b62c0dc75b73c32719 DEBU[0000] Skipping blob sha256:94d7953f76d275a6e79eef8adaaa8dc590eabae3be5266b8f2cab7cd7ff407f0 (already present): DEBU[0000] Detected compression format gzip
DEBU[0000] Using original blob without modification
Copying blob d910bff0e279 done | Copying blob 94d7953f76d2 skipped: already exists
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported DEBU[0000] Check for idmapped mounts support
Copying blob d910bff0e279 done | Copying blob 94d7953f76d2 skipped: already exists
DEBU[0001] No compression detected
DEBU[0001] Compression change for blob sha256:cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c ("application/vnd.oci.image.config.v1+json") not supported DEBU[0001] Using original blob without modification
Copying config cca5b82945 done | Writing manifest to image destination DEBU[0001] setting image creation date to 2024-02-15 14:49:29.477555032 +0000 UTC DEBU[0001] created new image ID "cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" with metadata "{}" DEBU[0001] added name "localhost:5000/mariadb:latest" to image "cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" DEBU[0001] Pulled candidate localhost:5000/mariadb:latest successfully DEBU[0001] Looking up image "cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" in local containers storage DEBU[0001] Trying "cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" ... DEBU[0001] parsed reference into "[overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]@cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" DEBU[0001] Found image "cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" as "cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" in local containers storage DEBU[0001] Found image "cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" as "cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" in local containers storage ([overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]@cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c) DEBU[0001] exporting opaque data as blob "sha256:cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" DEBU[0001] User mount /home/bill/nginx/etc/mariadb:/etc/my.cnf.d options [Z] DEBU[0001] User mount /home/bill/mariadb:/var/lib/mysql options [z] DEBU[0001] User mount /home/bill/zm/sql_build:/usr/share/zoneminder/db options [z] DEBU[0001] Looking up image "localhost:5000/mariadb" in local containers storage DEBU[0001] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0001] Trying "localhost:5000/mariadb:latest" ...
DEBU[0001] parsed reference into "[overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]@cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" DEBU[0001] Found image "localhost:5000/mariadb" as "localhost:5000/mariadb:latest" in local containers storage DEBU[0001] Found image "localhost:5000/mariadb" as "localhost:5000/mariadb:latest" in local containers storage ([overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]@cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c) DEBU[0001] exporting opaque data as blob "sha256:cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" DEBU[0001] Inspecting image cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c DEBU[0001] exporting opaque data as blob "sha256:cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" DEBU[0001] Inspecting image cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c DEBU[0001] Inspecting image cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c DEBU[0001] Inspecting image cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c DEBU[0001] using systemd mode: false
DEBU[0001] adding container to pod zm
DEBU[0001] setting container name mariadb
DEBU[0001] Loading seccomp profile from "/etc/containers/seccomp.json" INFO[0001] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host DEBU[0001] Adding mount /proc
DEBU[0001] Adding mount /dev
DEBU[0001] Adding mount /dev/pts
DEBU[0001] Adding mount /dev/mqueue
DEBU[0001] Adding mount /sys
DEBU[0001] Adding mount /sys/fs/cgroup
DEBU[0001] Allocated lock 3 for container f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d DEBU[0001] parsed reference into "[overlay@/home/bill/.local/share/containers/storage+/run/user/1000/containers]@cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" DEBU[0001] exporting opaque data as blob "sha256:cca5b829452166d3ef96d659d38ab2f4d934313096ca374c59ba3956ed0cd17c" DEBU[0001] Created container "f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d" DEBU[0001] Container "f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d" has work directory "/home/bill/.local/share/containers/storage/overlay-containers/f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d/userdata" DEBU[0001] Container "f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d" has run directory "/run/user/1000/containers/overlay-containers/f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d/userdata" DEBU[0001] Cached value indicated that volatile is being used DEBU[0001] overlay: mount_data=lowerdir=/home/bill/.local/share/containers/storage/overlay/l/MQI5UNBRX6SCDB524OHRSXAGYX:/home/bill/.local/share/containers/storage/overlay/l/FRW4TN2D52YS6U4V6TXNG7QBKD,upperdir=/home/bill/.local/share/containers/storage/overlay/157501ffd1dafc96c94f629ffe971379b8736fe832e9c2702d72919a0bedf9b8/diff,workdir=/home/bill/.local/share/containers/storage/overlay/157501ffd1dafc96c94f629ffe971379b8736fe832e9c2702d72919a0bedf9b8/work,userxattr,volatile,context="system_u:object_r:container_file_t:s0:c327,c383" DEBU[0001] Mounted container "f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d" at "/home/bill/.local/share/containers/storage/overlay/157501ffd1dafc96c94f629ffe971379b8736fe832e9c2702d72919a0bedf9b8/merged" DEBU[0001] Timezone /usr/share/zoneinfo/America/Los_Angeles does not exist in the container, create our own copy from the host DEBU[0001] Created root filesystem for container f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d at /home/bill/.local/share/containers/storage/overlay/157501ffd1dafc96c94f629ffe971379b8736fe832e9c2702d72919a0bedf9b8/merged DEBU[0001] /etc/system-fips does not exist on host, not mounting FIPS mode subscription DEBU[0001] Setting Cgroups for container f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d to user-libpod_pod_0d7c01f58f0b06ff6d00f38d0190d72b9a261a561581f53e946d984e16bf5848.slice:libpod:f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d DEBU[0001] reading hooks from /usr/share/containers/oci/hooks.d DEBU[0001] Workdir "/" resolved to host path "/home/bill/.local/share/containers/storage/overlay/157501ffd1dafc96c94f629ffe971379b8736fe832e9c2702d72919a0bedf9b8/merged" DEBU[0001] Created OCI spec for container f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d at /home/bill/.local/share/containers/storage/overlay-containers/f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d/userdata/config.json DEBU[0001] Created cgroup path user.slice/user-libpod_pod_0d7c01f58f0b06ff6d00f38d0190d72b9a261a561581f53e946d984e16bf5848.slice for parent user.slice and name libpod_pod_0d7c01f58f0b06ff6d00f38d0190d72b9a261a561581f53e946d984e16bf5848 DEBU[0001] Created cgroup user.slice/user-libpod_pod_0d7c01f58f0b06ff6d00f38d0190d72b9a261a561581f53e946d984e16bf5848.slice DEBU[0001] Got pod cgroup as user.slice/user-1000.slice/user@1000.service/user.slice/user-libpod_pod_0d7c01f58f0b06ff6d00f38d0190d72b9a261a561581f53e946d984e16bf5848.slice DEBU[0001] /usr/bin/conmon messages will be logged to syslog DEBU[0001] running conmon: /usr/bin/conmon args="[--api-version 1 -c f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d -u f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d -r /usr/bin/crun -b /home/bill/.local/share/containers/storage/overlay-containers/f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d/userdata -p /run/user/1000/containers/overlay-containers/f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d/userdata/pidfile -n mariadb --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l journald --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/bill/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/bill/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d]" INFO[0001] Running conmon under slice user-libpod_pod_0d7c01f58f0b06ff6d00f38d0190d72b9a261a561581f53e946d984e16bf5848.slice and unitName libpod-conmon-f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d.scope [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0001] Received: 7283
INFO[0001] Got Conmon PID as 7281
DEBU[0001] Created container f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d in OCI runtime DEBU[0001] Starting container f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d with command [mariadbd --user=mysql] DEBU[0001] Started container f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d DEBU[0001] Notify sent successfully
f7ff162c0aed108432f22c466940cc6c0f4de74019cadf4129364a9039aec08d DEBU[0001] Called run.PersistentPostRunE(podman run -d --name mariadb --pod zm -v /home/bill/nginx/etc/mariadb:/etc/my.cnf.d:Z -v /home/bill/mariadb:/var/lib/mysql:z -v /home/bill/zm/sql_build:/usr/share/zoneminder/db:z --rm --tz=local --log-level debug --tls-verify=false localhost:5000/mariadb) DEBU[0001] Shutting down engines

Steps to reproduce the issue

FROM alpine-bills ENV TZ='America/Los_Angeles' RUN echo && echo " Install mariadb " \ && apk add --update --no-cache \ mariadb \ mariadb-client \ && mkdir /run/mysqld \ && chown -R mysql /run/mysqld \ && echo && echo " Cleanup " \ && rm -rf \ $HOME/.cache \ /tmp/ \ /var/cache/apk/ \ /var/tmp/*

CMD [ "mariadbd", "--user=mysql" ]

CMD [ "/bin/ash" ]
$ podman image build -f mariadb.imgbuild -t mariadb .
$ podman push --tls-verify=false mariadb localhost:5000/mariadb
$ podman run -d --name mariadb --pod zm -v /home/carl/nginx/etc/mariadb:/etc/my.cnf.d:Z -v /home/carl/mariadb:/var/lib/mysql:z -v /home/carl/zm/sql_build:/usr/share/zoneminder/db:z --rm --tz=local --tls-verify=false localhost:5000/mariadb

Describe the results you received

Describe the results you received "Shutting down engines."

No evident problems.

Describe the results you expected

To have a running container inside a pod.

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.
On CentOS 9 to create image:
$ rpm -qi podman
Name        : podman
Epoch       : 2
Version     : 4.6.1
Release     : 5.el9
Architecture: x86_64
Install Date: Mon 04 Sep 2023 10:19:15 AM PDT

On OpenSuse MicroOS to run the image:
$ rpm -qi podman
Name        : podman
Version     : 4.9.2
Release     : 1.1
Architecture: x86_64
Install Date: Sun 11 Feb 2024 01:19:36 PM PST

Podman in a container

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

selinux is permissive. sealert finds no violations.

Directory owners are all set to 100099, the image's user.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

Fails consistently. Mariadb makes binary logs but can not make non-binary ones in the data dir.

containers / podman