Quadlet failing to pull

chsasank commented 2 weeks ago

Issue Description

Podman service file somehow timing out before pulling.

Steps to reproduce the issue

create the following .container file

$ cat /home/sasank/.config/containers/systemd/matrix-postgres.container
[Container]
Image=docker.io/postgres:12-alpine
Pod=matrix.pod
EnvironmentFile=/home/sasank/.johnny/matrix/postgres.env
PodmanArgs=
Exec=
Volume=/home/sasank/.johnny/matrix/postgres-data:/var/lib/postgresql/data

[Service]
Restart=always

[Install]
WantedBy=default.target

Reload daemon

$ systemctl --user daemon-reload

Start it

$ systemctl --user start matrix-postgres.service
Job for matrix-postgres.service canceled.

Describe the results you received

Look for status

$ systemctl --user status matrix-postgres.service
● matrix-postgres.service
     Loaded: loaded (/home/sasank/.config/containers/systemd/matrix-postgres.container; generated)
     Active: activating (start) since Wed 2024-10-09 18:02:02 IST; 324ms ago
   Main PID: 3618748 (podman)
      Tasks: 12 (limit: 38034)
     Memory: 13.2M
        CPU: 24ms
     CGroup: /user.slice/user-1002.slice/user@1002.service/app.slice/matrix-postgres.service
             └─3618748 /usr/bin/podman run --name=systemd-matrix-postgres --cidfile=/run/user/1002/matrix-postgres.cid --replace --rm --cgroups=split --sdnotify=conmon -d -v /home/sasank/.johnny/matrix/postg>

Oct 09 18:02:02 JOHNAIC systemd[904]: Starting matrix-postgres.service...
Oct 09 18:02:02 JOHNAIC matrix-postgres[3618748]: Trying to pull docker.io/library/postgres:12-alpine...
Oct 09 18:02:02 JOHNAIC matrix-postgres[3618748]: Pulling image //postgres:12-alpine inside systemd: setting pull timeout to 5m0s
Oct 09 18:02:03 JOHNAIC systemd[904]: Stopped matrix-postgres.service.

Describe the results you expected

Expected the image to pull and run

podman info output

host: arch: amd64 buildahVersion: 1.37.4 cgroupControllers:

memory
pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon_100:2.1.12-1_amd64 path: /usr/bin/conmon version: 'conmon version 2.1.12, commit: e21e7c85b7637e622f21c57675bf1154fc8b1866' cpuUtilization: idlePercent: 94.15 systemPercent: 1.57 userPercent: 4.27 cpus: 12 databaseBackend: boltdb distribution: codename: jammy distribution: neon version: "22.04" eventLogger: journald freeLocks: 2024 hostname: JOHNAIC idMappings: gidmap:
- container_id: 0 host_id: 1002 size: 1
- container_id: 1 host_id: 231072 size: 65536 uidmap:
- container_id: 0 host_id: 1002 size: 1
- container_id: 1 host_id: 231072 size: 65536 kernel: 6.5.0-45-generic linkmode: dynamic logDriver: journald memFree: 3304464384 memTotal: 33372499968 networkBackend: netavark networkBackendInfo: backend: netavark dns: package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64 path: /usr/libexec/podman/aardvark-dns version: aardvark-dns 1.6.0 package: netavark_1.3.0-0ubuntu22.04+obs22.9_amd64 path: /usr/libexec/podman/netavark version: netavark 1.3.0 ociRuntime: name: crun package: crun_101:1.14.4-0ubuntu22.04+obs70.24_amd64 path: /usr/bin/crun version: |- crun version 1.14.4 commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1 rundir: /run/user/1002/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL os: linux pasta: executable: /usr/bin/pasta package: passt_100:0.0+20240906.6b38f072-1_amd64 version: | pasta 0.0+20240906.6b38f072 Copyright Red Hat GNU General Public License, version 2 or later https://www.gnu.org/licenses/old-licenses/gpl-2.0.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. remoteSocket: exists: false path: /run/user/1002/podman/podman.sock rootlessNetworkCmd: pasta security: apparmorEnabled: false capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns_100:1.3.1-1_amd64 version: |- slirp4netns version 1.3.1 commit: unknown libslirp: 4.6.1 SLIRP_CONFIG_VERSION_MAX: 5 libseccomp: 2.5.3 swapFree: 30199177216 swapTotal: 36709785600 uptime: 174h 6m 15.00s (Approximately 7.25 days) variant: "" plugins: authorization: null log:
k8s-file
none
passthrough
journald network:
bridge
macvlan
ipvlan volume:
local registries: search:
docker.io store: configFile: /home/sasank/.config/containers/storage.conf containerStore: number: 12 paused: 0 running: 11 stopped: 1 graphDriverName: overlay graphOptions: {} graphRoot: /home/sasank/.local/share/containers/storage graphRootAllocated: 463303442432 graphRootUsed: 149483036672 graphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Supports shifting: "false" Supports volatile: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 26 runRoot: /run/user/1002/containers transientStore: false volumePath: /home/sasank/.local/share/containers/storage/volumes version: APIVersion: 5.2.4 Built: 0 BuiltTime: Thu Jan 1 05:30:00 1970 GitCommit: "" GoVersion: go1.23.2 Os: linux OsArch: linux/amd64 Version: 5.2.4

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response

chsasank commented 2 weeks ago

If I pull the image manually, it works as expected

$ podman pull docker.io/library/postgres:12-alpine
$ systemctl --user start  matrix-postgres.service 
$ systemctl --user status  matrix-postgres.service 
● matrix-postgres.service
     Loaded: loaded (/home/sasank/.config/containers/systemd/matrix-postgres.container; generated)
     Active: active (running) since Wed 2024-10-09 18:07:30 IST; 3s ago
   Main PID: 3623990 (conmon)
      Tasks: 8 (limit: 38034)
     Memory: 19.1M
        CPU: 98ms
     CGroup: /user.slice/user-1002.slice/user@1002.service/app.slice/matrix-postgres.service
             ├─libpod-payload-c5a823b759653d471854b424856674ad7fc3342a94ba05c72afc8f1f0a31b770
             │ ├─3623992 postgres
             │ ├─3624031 "postgres: checkpointer   "
             │ ├─3624032 "postgres: background writer   "
             │ ├─3624033 "postgres: walwriter   "
             │ ├─3624034 "postgres: autovacuum launcher   "
             │ ├─3624035 "postgres: stats collector   "
             │ └─3624036 "postgres: logical replication launcher   "
             └─runtime
               └─3623990 /usr/bin/conmon --api-version 1 -c c5a823b759653d471854b424856674ad7fc3342a94ba05c72afc8f1f0a31b770 -u c5a823b759653d471854b424856674ad7fc3342a94ba05c72afc8f1f0a31b770 -r /usr/bin/cr>

Oct 09 18:07:30 JOHNAIC matrix-postgres[3623951]: c5a823b759653d471854b424856674ad7fc3342a94ba05c72afc8f1f0a31b770
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: PostgreSQL Database directory appears to contain a database; Skipping initialization
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.318 UTC [1] LOG:  starting PostgreSQL 12.20 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309,>
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.318 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.318 UTC [1] LOG:  listening on IPv6 address "::", port 5432
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.322 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.340 UTC [22] LOG:  database system was shut down at 2024-10-09 12:18:54 UTC
Oct 09 18:07:30 JOHNAIC systemd-matrix-postgres[3623990]: 2024-10-09 12:37:30.355 UTC [1] LOG:  database system is ready to accept connections

vrothberg commented 2 weeks ago

How long does it take to pull the image? It looks like you're running into a timeout that you can extend if needed.

chsasank commented 2 weeks ago

When I pulled the image manually it hardly takes twenty seconds.

$ podman image rm docker.io/postgres:12-alpine
Untagged: docker.io/library/postgres:12-alpine
Deleted: 994c5880d6731b241afaae87c0b25ab2f005785f4175ef147ebcd3f0c652343c

$ podman image prune
WARNING! This command removes all dangling images.
Are you sure you want to continue? [y/N] y

$ time podman image pull docker.io/postgres:12-alpine
Trying to pull docker.io/library/postgres:12-alpine...
Getting image source signatures
Copying blob fb60b7546f99 done   | 
Copying blob e7368e03b632 done   | 
Copying blob 43c4264eed91 skipped: already exists  
Copying blob 738c839a1612 done   | 
Copying blob a4a6aafd4b69 done   | 
Copying blob 038cdbd7acd1 done   | 
Copying blob e8698ad92e74 done   | 
Copying blob 3a291eff5ef1 done   | 
Copying blob 4a28cf0127a7 done   | 
Copying blob e9be2a968901 done   | 
Copying blob eb78f2fdb93e done   | 
Copying config 994c5880d6 done   | 
Writing manifest to image destination
994c5880d6731b241afaae87c0b25ab2f005785f4175ef147ebcd3f0c652343c

real    0m11.857s
user    0m3.106s
sys     0m0.956s

chsasank commented 2 weeks ago

Can reproduce the issue by simply removing the image

$ podman image rm docker.io/postgres:12-alpine                                                                                                             
Untagged: docker.io/library/postgres:12-alpine                                                                                                                                                                  
Deleted: 994c5880d6731b241afaae87c0b25ab2f005785f4175ef147ebcd3f0c652343c                                                                                                                                       

$ systemctl --user stop  matrix-postgres.service                                                                                                           

$ systemctl --user start  matrix-postgres.service   
Job for matrix-postgres.service canceled.

$ systemctl --user status  matrix-postgres.service  
○ matrix-postgres.service
     Loaded: loaded (/home/sasank/.config/containers/systemd/matrix-postgres.container; generated)
     Active: inactive (dead) since Wed 2024-10-09 18:13:07 IST; 2s ago
    Process: 3630984 ExecStart=/usr/bin/podman run --name=systemd-matrix-postgres --cidfile=/run/user/1002/matrix-postgres.cid --replace --rm --cgroups=split --sdnotify=conmon -d -v /home/sasank/.johnny/matr>
    Process: 3631096 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1002/matrix-postgres.cid (code=exited, status=0/SUCCESS)
   Main PID: 3630984 (code=exited, status=0/SUCCESS) 
        CPU: 71ms

Oct 09 18:13:07 JOHNAIC systemd[904]: Starting matrix-postgres.service...
Oct 09 18:13:07 JOHNAIC matrix-postgres[3630984]: Trying to pull docker.io/library/postgres:12-alpine... 
Oct 09 18:13:07 JOHNAIC matrix-postgres[3630984]: Pulling image //postgres:12-alpine inside systemd: setting pull timeout to 5m0s
Oct 09 18:13:07 JOHNAIC systemd[904]: Stopped matrix-postgres.service.

chsasank commented 2 weeks ago

Surprisingly I can not reproduce this for a different container

$ cat /home/sasank/.config/containers/systemd/matrix-synapse.container                                                                                     
[Container]                                                                                                                                                                                                     
Image=docker.io/matrixdotorg/synapse:v1.116.0                                                                                                                                                                   
Pod=matrix.pod                                                                                                                                                                                                  
EnvironmentFile=/home/sasank/.johnny/matrix/synapse.env                                                                                                                                                         
PodmanArgs=                                                                                                                                                                                                     
Exec=                                                                                                                                                                                                           
Volume=/home/sasank/.johnny/matrix/matrix-data:/data                                                                                                                                                            

[Service]                                                                                                                                                                                                       
Restart=always                                                                                                                                                                                                  

[Install]                                                                                                                                                                                                       
WantedBy=default.target                              

$ podman image rm docker.io/matrixdotorg/synapse:v1.116.0

$ time systemctl --user start  matrix-synapse.service

real    0m12.101s
user    0m0.002s
sys     0m0.000s

ygalblum commented 2 weeks ago

In the Quadlet file you are setting: Image=docker.io/postgres:12-alpine. However, the image FQDN is docker.io/library/postgres:12-alpine (this is what you pull).

podman run knows to translate the incorrect name into the correct one when looking for it locally. But, when it is not available, it tried to download according to the provided value which fails at the server side.

chsasank commented 2 weeks ago

Tried with that too. Won't work

$ cat  /home/sasank/.config/containers/systemd/matrix-postgres.container
[Container]
Image=docker.io/library/postgres:12-alpine
Pod=matrix.pod
EnvironmentFile=/home/sasank/.johnny/matrix/postgres.env
PodmanArgs=
Exec=
Volume=/home/sasank/.johnny/matrix/postgres-data:/var/lib/postgresql/data

[Service]
Restart=always

[Install]
WantedBy=default.target

$ systemctl --user start  matrix-postgres.service
Job for matrix-postgres.service canceled.

$ systemctl --user status  matrix-postgres.service
○ matrix-postgres.service
     Loaded: loaded (/home/sasank/.config/containers/systemd/matrix-postgres.container; generated)
     Active: inactive (dead) since Wed 2024-10-09 18:24:16 IST; 4s ago
    Process: 3646387 ExecStart=/usr/bin/podman run --name=systemd-matrix-postgres --cidfile=/run/user/1002/matrix-postgres.cid --replace --rm --cgroups=split --sdnotify=conmon -d -v /home/sasank/.johnny/matr>
    Process: 3646458 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1002/matrix-postgres.cid (code=exited, status=0/SUCCESS)
   Main PID: 3646387 (code=exited, status=0/SUCCESS)
        CPU: 57ms

Oct 09 18:24:16 JOHNAIC systemd[904]: Starting matrix-postgres.service...
Oct 09 18:24:16 JOHNAIC matrix-postgres[3646387]: Trying to pull docker.io/library/postgres:12-alpine...
Oct 09 18:24:16 JOHNAIC matrix-postgres[3646387]: Pulling image //postgres:12-alpine inside systemd: setting pull timeout to 5m0s
Oct 09 18:24:16 JOHNAIC systemd[904]: Stopped matrix-postgres.service.

ygalblum commented 2 weeks ago

I tested the Quadlet file and it works on my setup (Fedora 40). So, I think there is something in your setup that's causing it. BTW from the timestamps, I don't think it's a timeout issue because it seems that the service is stopped immediately.

Can you please share your setup? I can see that it depends on other Quadlets.

chsasank commented 1 week ago

How do I share the setup? I have pasted podman info above. I installed podman as in here: https://outline.von-neumann.ai/s/e188eaf2-3269-46d6-a5db-cf95ee9700e2. I use kde neon which is ubuntu 22.04 based release

ygalblum commented 1 week ago

I can see that it depends on other Quadlets

Sorry, if I wasn't clear. I can see that this .container file is only one part of a bigger deployment (it's part of a pod). So, I wanted to get all Quadlet files and their dependencies (e.g. env files) to see if it reproduces on my setup

containers / podman