search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.83k stars 2.42k forks source link

Podman fails to run containers if image is not from docker.io after updating to 4.9.4 #22578

Closed matteo-gambarutti closed 6 months ago

matteo-gambarutti commented 6 months ago

Issue Description

After the last system updates on RHEL 9, podman got update to version 4.9.4. I have quadlet files to manage my containers and some of them do not start anymore: only the ones where the image is not from docker.io registry.

In particular I get the following error:

May 02 14:25:44 homeserver unifi_network_application[25827]: Error: short-name resolution enforced but cannot prompt without a TTY
May 02 14:25:44 homeserver systemd[1169]: unifi_network_application.service: Main process exited, code=exited, status=125/n/a
May 02 14:25:44 homeserver systemd[1169]: unifi_network_application.service: Failed with result 'exit-code'.
May 02 14:25:44 homeserver systemd[1169]: Failed to start Unifi Network Application.
May 02 14:25:44 homeserver systemd[1169]: unifi_network_application.service: Scheduled restart job, restart counter is at 1.
May 02 14:25:44 homeserver systemd[1169]: Stopped Unifi Network Application.

I find it strange to see Error: short-name resolution enforced but cannot prompt without a TTY as this is the image definition the quadlet file:

Image=lscr.io/linuxserver/unifi-network-application:latest

How can this being fixed? As I said, no issues with containers with and image coming from docker.io.

I did not change anything in the quadlet files definition. Everything was working fine before the updates.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Define quadlet file with image coming from a registry which is not docker.io
  2. systemctl --user daemon-reload
  3. Check error

Describe the results you received

Error: short-name resolution enforced but cannot prompt without a TTY

Describe the results you expected

A running container

podman info output

[podman@homeserver ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: fb8c4bf50dbc044a338137871b096eea8041a1fa'
  cpuUtilization:
    idlePercent: 99.26
    systemPercent: 0.27
    userPercent: 0.47
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: rhel
    version: "9.4"
  eventLogger: journald
  freeLocks: 2017
  hostname: homeserver
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 5.14.0-427.13.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1673453568
  memTotal: 3733553152
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-3.el9_4.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/1002/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/1002/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4139773952
  swapTotal: 4139773952
  uptime: 2h 26m 8.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 5
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 101289099264
  graphRootUsed: 6962618368
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /tmp/containers-user-1002/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1713184388
  BuiltTime: Mon Apr 15 14:33:08 2024
  GitCommit: ""
  GoVersion: go1.21.7 (Red Hat 1.21.7-1.el9)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel```

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

Luap99 commented 6 months ago

Please provide the full quadlet file and run systcemctl cat unifi_network_application.service to see the full podman command to check if quadlet may have generated something incorrectly.

Also if you use RHEL please contact RHEL support as upstream really only supports the lastest version or try if you can reproduce with podman 5.

matteo-gambarutti commented 6 months ago

Here the full quadlet file:

Description=Unifi Network Application
After=local-fs.target

[Container]
Image=lscr.io/linuxserver/unifi-network-application:latest
ContainerName=unifi_network_application
AutoUpdate=registry
Environment=PUID=${PODMAN_PUID}
Environment=PGID=${PODMAN_PGID}
Environment=TZ=Europe/Amsterdam
Environment=MONGO_USER=unifi
Environment=MONGO_HOST=192.168.1.116
Environment=MONGO_PORT=27017
Environment=MONGO_DBNAME=unifi
Secret=unifi_mongodb_unifi_password,type=env,target=MONGO_PASS
PublishPort=3478:3478/udp
PublishPort=8080:8080
PublishPort=8443:8443
PublishPort=8843:8843
PublishPort=8880:8880
PublishPort=10001:10001/udp
Volume=${CONFIG_DIR}/unifi_network_application:/config:Z
User=$(id -u):$(id -g)
UserNS=keep-id
HealthStartPeriod=2m
HealthCmd=CMD-SHELL curl -f --insecure https://localhost:8443 || exit 1
HealthInterval=30s
HealthRetries=2
HealthOnFailure=kill

[Service]
Restart=on-failure

[Install]
WantedBy=multi-user.target default.target

Here the full podman command from systemctl status:

ExecStart=/usr/bin/podman run --name=unifi_network_application --cidfile=/run/user/1002/unifi_network_application.cid --replace --rm --cgroups=split --sdnotify=conmon -d --user $(id -u):$(id -g) --userns keep-id -v ${CONFIG_DIR}/unifi_network_application:/config:Z --label io.containers.autoupdate=registry --publish 3478:3478/udp --publish 8080:8080 --publish 8443:8443 --publish 8843:8843 --publish 8880:8880 --publish 10001:10001/udp --env MONGO_DBNAME=unifi --env MONGO_HOST=192.168.1.116 --env MONGO_PORT=27017 --env MONGO_USER=unifi --env PGID=${PODMAN_PGID} --env PUID=${PODMAN_PUID} --env TZ=Europe/Amsterdam --secret unifi_mongodb_unifi_password,type=env,target=MONGO_PASS --health-cmd CMD-SHELL curl -f --insecure https://localhost:8443 || exit 1 --health-interval 30s --health-on-failure kill --health-retries 2 --health-start-period 2m lscr.io/linuxserver/unifi-network-application:latest (code=exited, status=125)

Luap99 commented 6 months ago

please show the systcemctl cat output, seeing the actual quoting is really important

matteo-gambarutti commented 6 months ago

This is the output:

[podman@homeserver ~]$ systemctl --user cat unifi_network_application.service
# /run/user/1002/systemd/generator/unifi_network_application.service
# Automatically generated by /usr/lib/systemd/user-generators/podman-user-generator
#
[Unit]
Description=Unifi Network Application
After=local-fs.target
SourcePath=/home/podman/.config/containers/systemd/unifi_network_application.container
RequiresMountsFor=%t/containers

[X-Container]
Image=lscr.io/linuxserver/unifi-network-application:latest
ContainerName=unifi_network_application
AutoUpdate=registry
Environment=PUID=${PODMAN_PUID}
Environment=PGID=${PODMAN_PGID}
Environment=TZ=Europe/Amsterdam
Environment=MONGO_USER=unifi
Environment=MONGO_HOST=192.168.1.116
Environment=MONGO_PORT=27017
Environment=MONGO_DBNAME=unifi
Secret=unifi_mongodb_unifi_password,type=env,target=MONGO_PASS
PublishPort=3478:3478/udp
PublishPort=8080:8080
PublishPort=8443:8443
PublishPort=8843:8843
PublishPort=8880:8880
PublishPort=10001:10001/udp
Volume=${CONFIG_DIR}/unifi_network_application:/config:Z
User=$(id -u):$(id -g)
UserNS=keep-id
HealthStartPeriod=2m
HealthCmd=CMD-SHELL curl -f --insecure https://localhost:8443 || exit 1
HealthInterval=30s
HealthRetries=2
HealthOnFailure=kill

[Service]
Restart=on-failure
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/bin/podman rm -v -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=unifi_network_application --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d --user "$(id -u):$(id -g)" --userns keep-id -v ${CONFIG_DIR}/unifi_network_application:/config:Z --label io.containers.autoupdate=registry --publish 3478:3478/udp --publish 8080:8080 --publish 8443:8443 --publish 8843:8843 --publish 8880:8880 --publish 10001:10001/udp --env MONGO_DBNAME=unifi --env MONGO_HOST=192.168.1.116 --env MONGO_PORT=27017 --env MONGO_USER=unifi --env PGID=${PODMAN_PGID} --env PUID=${PODMAN_PUID} --env TZ=Europe/Amsterdam --secret unifi_mongodb_unifi_password,type=env,target=MONGO_PASS --health-cmd "CMD-SHELL curl -f --insecure https://localhost:8443 || exit 1" --health-interval 30s --health-on-failure kill --health-retries 2 --health-start-period 2m lscr.io/linuxserver/unifi-network-application:latest

[Install]
WantedBy=multi-user.target default.target
Luap99 commented 6 months ago

setting something like "$(id -u):$(id -g)" is not possible systemd does not run the command through your shell so even if the pull works this will not get expanded and is invalid syntax for the --user option so I don't see how this ever would have worked even before the update.

To be clear my assumption is that the cli args are wrong somehow, because pulling lscr.io/linuxserver/unifi-network-application:latest will not result in such error rather something like a extra space somewhere, i.e. podman run --rm -p 8080 8080 lscr.io/linuxserver/unifi-network-application:latest This it would assume 8080 is the image name although I don't see such case in you command right now.

matteo-gambarutti commented 6 months ago

About that "$(id -u):$(id -g)": I got crazy for like 4 months and that small change immediately did the trick as my directories mounted inside the container no longer change user/group. It did work, but do not ask me why ;)

Indeed pulling the image works with podman pull .... I'm pretty sure it comes from the last podman update has the issue arose immediately after the system reboot.

matteo-gambarutti commented 6 months ago

I just downgraded to podman version 4.6.1 and all of my containers do work as expected. I didn't have to do anything, just a reboot of the system (after the downgrade) and they came back running on their own.

Luap99 commented 6 months ago

Can you do another systemctl cat to see if there are any differences between the generated units

matteo-gambarutti commented 6 months ago

For sure:

[podman@homeserver ~]$ systemctl --user cat unifi_network_application.service > output.txt
[podman@homeserver ~]$ cat output.txt
# /run/user/1002/systemd/generator/unifi_network_application.service
# Automatically generated by /usr/lib/systemd/user-generators/podman-user-generator
#
[Unit]
Description=Unifi Network Application
After=local-fs.target
SourcePath=/home/podman/.config/containers/systemd/unifi_network_application.container
RequiresMountsFor=%t/containers

[X-Container]
Image=lscr.io/linuxserver/unifi-network-application:latest
ContainerName=unifi_network_application
AutoUpdate=registry
Environment=PUID=${PODMAN_PUID}
Environment=PGID=${PODMAN_PGID}
Environment=TZ=Europe/Amsterdam
Environment=MONGO_USER=unifi
Environment=MONGO_HOST=192.168.1.116
Environment=MONGO_PORT=27017
Environment=MONGO_DBNAME=unifi
Secret=unifi_mongodb_unifi_password,type=env,target=MONGO_PASS
PublishPort=3478:3478/udp
PublishPort=8080:8080
PublishPort=8443:8443
PublishPort=8843:8843
PublishPort=8880:8880
PublishPort=10001:10001/udp
Volume=${CONFIG_DIR}/unifi_network_application:/config:Z
User=$(id -u):$(id -g)
UserNS=keep-id
HealthStartPeriod=2m
HealthCmd=CMD-SHELL curl -f --insecure https://localhost:8443 || exit 1
HealthInterval=30s
HealthRetries=2
HealthOnFailure=kill

[Service]
Restart=on-failure
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=unifi_network_application --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d --user 0 --userns keep-id -v ${CONFIG_DIR}/unifi_network_application:/config:Z --label io.containers.autoupdate=registry --publish 3478:3478/udp --publish 8080:8080 --publish 8443:8443 --publish 8843:8843 --publish 8880:8880 --publish 10001:10001/udp --env MONGO_DBNAME=unifi --env MONGO_HOST=192.168.1.116 --env MONGO_PORT=27017 --env MONGO_USER=unifi --env PGID=${PODMAN_PGID} --env PUID=${PODMAN_PUID} --env TZ=Europe/Amsterdam --secret unifi_mongodb_unifi_password,type=env,target=MONGO_PASS --health-cmd "CMD-SHELL curl -f --insecure https://localhost:8443 || exit 1" --health-interval 30s --health-on-failure kill --health-retries 2 --health-start-period 2m lscr.io/linuxserver/unifi-network-application:latest

[Install]
WantedBy=multi-user.target default.target
Luap99 commented 6 months ago

As I suspected the difference is --user "$(id -u):$(id -g)" vs --user 0 (working one)

matteo-gambarutti commented 6 months ago

Yeah indeed, just noticed the same! What would it be the correct syntax then to not have podman changing the ownership of mounted directories inside containers? Thanks a lot for your time btw!

Luap99 commented 6 months ago

Well if --user 0 worked for you then I suggest you set User=0 in your quadlet file.

matteo-gambarutti commented 6 months ago

Upgraded again to 4.9.1 and changed to User=0 in the quadlet file fixed the issue. Thanks a lot for you help @Luap99!