.container using EnvironmentFile no longer starts in 4.5.1

[acrispino]

Issue Description

After upgrading to 4.5.1-4 on Centos Stream 9, a .container file that was working with 4.4.1-9 no longer starts. It seems to have to do with how EnvironmentFile is parsed to create the service ExecStart

I've worked around it but is the change in behavior a bug? Is this related to #17906?

Steps to reproduce the issue

# /home/ec2-user/.config/containers/systemd/mysleep.container
[Unit]
Description=The sleep container
After=local-fs.target

[Container]
Image=registry.access.redhat.com/ubi9-minimal:latest
Exec=sleep 1000
EnvironmentFile=%n/env

[Install]
WantedBy=multi-user.target default.target

# /home/ec2-user/.config/containers/systemd/mysleep.service/env
EXAMPLE=1

$ systemctl --user daemon-reload && systemctl --user start mysleep
Job for mysleep.service failed because the control process exited with error code.
See "systemctl --user status mysleep.service" and "journalctl --user -xeu mysleep.service" for details.

Describe the results you received

Here's the output of /usr/libexec/podman/quadlet -dryrun -user. I've left out the lines that are unchanged between the two versions

(podman-2:4.4.1-9.el9.aarch64):

[Service]
# ...
ExecStopPost=-rm -f %t/%N.cid
# ...
ExecStart=/usr/bin/podman run --name=systemd-%N --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --sdnotify=conmon -d --env-file /home/ec2-user/.config/containers/systemd/%n/env registry.access.redhat.com/ubi9-minimal:latest sleep 1000

(podman-2:4.5.1-4.el9.aarch64):

[Service]
# ...
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
# ...
ExecStart=/usr/bin/podman run --name=systemd-%N --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d --env-file %n/env registry.access.redhat.com/ubi9-minimal:latest sleep 1000

The ExecStart line no longer has the .container file's directory prepended for --env-file. If I change the container's configuration to EnvironmentFile=./%n/env then the generated --env-file is the same as in 4.4.1 and the service can start again.

Describe the results you expected

For the container EnvironmentFile=%n/env to be parsed as a relative path

podman info output

host:
  arch: arm64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.el9.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: d1efc4610f5ea9873449fcd3211b5509fbde8aec'
  cpuUtilization:
    idlePercent: 99.02
    systemPercent: 0.19
    userPercent: 0.79
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: '"centos"'
    version: "9"
  eventLogger: journald
  hostname: ip-172-31-4-76.us-west-1.compute.internal
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.0-333.el9.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 144281600
  memTotal: 691834880
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.el9.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-3.el9.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 2h 45m 5.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/ec2-user/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ec2-user/.local/share/containers/storage
  graphRootAllocated: 31397490688
  graphRootUsed: 1998802944
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/ec2-user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1686840823
  BuiltTime: Thu Jun 15 14:53:43 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/arm64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

AWS AMI ami-02ddfebde24219626

Additional information

No response

[vrothberg]

Thanks for reaching out, @acrispino!

@ygalblum do you have time to look into this?

[Luap99]

I think that is intentional: https://github.com/containers/podman/pull/17930

[Luap99]

@ygalblum reping

I wonder if we have to keep a list of systemd specifiers that expand to and absolute path and the one that are just a name in which case we need to keep the relative path? Or maybe rethinking the whole thing, would it be smarter to let qualdet not expend the relative path and set WorkingDirectory= instead, then podman would resolve the path at runtime which may be better?

[ygalblum]

Sorry for the delay in my response.

@Luap99 I think you have a point there. In the current main branch, Quadlet supports setting WorkingDirectory= only for .kube files in order to allow the resolving of relative paths set inside the .yaml file. However, since the feature is focuses on these relative paths, it does not prevent the resolution for the command line parameters. This feature can be enhanced to support .container files and to not resolve the command line parameters. But, as always, we will need to keep backward compatibility in our mind.

@acrispino for you specific case, since whatever is decided will still take time to implement and publish, you can force relative path resolution by changing:

EnvironmentFile=%n/env

to

EnvironmentFile=./%n/env

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@ygalblum @acrispino Still worth pursuing?

[ygalblum]

@acrispino is my suggestion of forcing the relative path by prefixing it with ./ enough for this issue?

[acrispino]

Yes, that's how I've worked around it and my services are running fine.

[Luap99]

So @ygalblum is this something you like to address or should we just close wontfix given it is like this now for almost a year.

[ygalblum]

@Luap99 I think documenting it is enough.