search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.83k stars 2.42k forks source link

Quadlet does not automatically generate systemd service from .container files running as root on boot #20660

Closed latenightdef closed 1 year ago

latenightdef commented 1 year ago

Issue Description

I have a container file in another partition symlinked to /etc/containers/systemd. The command /usr/libexec/podman/quadlet -dryrun shows the output of container file just fine, which shows:

quadlet-generator[1965]: Loading source unit file /etc/containers/systemd/homeassistant.container
---homeassistant.service---
[Unit]
Description=Home Assistant Container
SourcePath=/etc/containers/systemd/homeassistant.container
RequiresMountsFor=%t/containers
RequiresMountsFor=/media/sasha/Data1/Apps/homeassistant/config

[X-Container]
ContainerName=homeassistant
Image=lscr.io/linuxserver/homeassistant:latest
PublishPort=8123:8123
Volume=/media/sasha/Data1/Apps/homeassistant/config:/config
Environment=PUID=1000
Environment=PGID=1000
Environment=TZ=Asia/Bangkok
AddDevice=/dev/ttyUSB0

[Service]
Restart=always
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=homeassistant --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d --device=/dev/ttyUSB0 -v /media/sasha/Data1/Apps/homeassistant/config:/config --publish 8123:8123 --env PGID=1000 --env PUID=1000 --env TZ=Asia/Bangkok lscr.io/linuxserver/homeassistant:latest

[Install]
WantedBy=multi-user.target default.target

When the host is restarted, the generated systemd service does not show up and the container doesn't autostart. I have to manually run systemctl daemon-reload to see the generated service appear in systemctl list-units and start the service manually to use the container.

But when I run the container in rootless mode (symlinking the same container file to ~/.config/containers/systemd), it autostarts normally.

Steps to reproduce the issue

Steps to reproduce the issue

  1. symlink the container file to /etc/containers/systemd
  2. systemctl daemon-reload and start the generated service
  3. reboot system

Describe the results you received

The container does not autostart

Describe the results you expected

The container autostarts just like when running in rootless.

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 86.52
    systemPercent: 6.98
    userPercent: 6.5
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: server
    version: "39"
  eventLogger: journald
  freeLocks: 2044
  hostname: efosa
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.5.10-300.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 6768173056
  memTotal: 10356961280
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.1-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.1
      commit: 1084f9527c143699b593b44c23555fb3cc4ff2f3
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231004.gf851084-1.fc39.x86_64
    version: |
      pasta 0^20231004.gf851084-1.fc39.x86_64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 0h 10m 54.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/sasha/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/sasha/.local/share/containers/storage
  graphRootAllocated: 254461083648
  graphRootUsed: 12919050240
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/sasha/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762721
  BuiltTime: Tue Oct 31 21:32:01 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2



### Podman in a container

No

### Privileged Or Rootless

Privileged

### Upstream Latest Release

No

### Additional environment details

_No response_

### Additional information

_No response_
ygalblum commented 1 year ago

@latenightdef thanks for reporting this issue.

I think the issue here is a race condition between Quadlet and the mounting of the partition in which the actual file resides. Such a race will cause the file to not be available for Quadlet to access when it is executed at boot. It also explains why this issue does not happen once boot is completed and why the same works for rootless.

Can you please check the Journal logs to see if Quadlet prints an error of the sort of no such file or directory?

latenightdef commented 1 year ago

I tried running journalctl -b -0 | grep "No such file or directory" and it shows

Nov 12 17:27:49 efosa syncthing[1866]: crontab: can't open 'abc': No such file or directory

which belongs to another rootless container and not the one I'm trying to autostart

latenightdef commented 1 year ago

I think the issue here is a race condition between Quadlet and the mounting of the partition in which the actual file resides. Such a race will cause the file to not be available for Quadlet to access when it is executed at boot. It also explains why this issue does not happen once boot is completed and why the same works for rootless.

Probably, since after copying the actual homeassistant.container file to /etc/containers/systemd, the homeassistant container autostarts correctly.

ygalblum commented 1 year ago

I'm not sure this issue can be resolved.

Quadlet and systemd-fstab-generator are both systemd generators. These generators are executed in parallel and as a result cannot depend on one another. In your case, Quadlet needs to depend on systemd-fstab-generator to run for systemd to mount the volume before Quadlet can start. However, there is no way to guaranty it.

@alexlarsson @rhatdan WDYT?

rhatdan commented 1 year ago

Seems reasonable. I guess the quadlet file needs to be on the same file system as /etc/containers/systemd or /usr/containers/systemd, other file systems are not guaranteed to be mounted.