[quadlet] nested symlinks in unit search path not resolved

Issue Description

This issue is related to https://github.com/containers/podman/issues/20504 or more precisely https://github.com/containers/podman/issues/20504#issuecomment-1808921064

I was using the Podman unit search path at /usr/share/containers/systemd/ and linked it to a directory called /var/quadlets which itself contained various symlinks to quadlets as shown in https://github.com/raballew/app-lifecycle/blob/66b182de56948a536e5d80707108a8170debe14b/demo.sh#L116

I did not run the script for some time but after updating to a newer version of Podman the generation of the unit files fails silently. QUADLET_UNIT_DIRS is not working for me as I am targeting production usage.

Steps to reproduce the issue

create a symlink from a search path with nested symlinks to a directory that contains quadlet unit files
run quadlet

Describe the results you received

Unit files are not generated from Quadlets.

Describe the results you expected

Quadlet finds the unit files.

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 92.78
    systemPercent: 1.39
    userPercent: 5.83
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "38"
  eventLogger: journald
  freeLocks: 2035
  hostname: pwallrab
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.4-100.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 15725125632
  memTotal: 33423994880
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.2-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.2
      commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231107.g56d9f6d-1.fc38.x86_64
    version: |
      pasta 0^20231107.g56d9f6d-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 11h 28m 31.00s (Approximately 0.46 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  mirror.gcr.io:
    Blocked: false
    Insecure: false
    Location: mirror.gcr.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: mirror.gcr.io
    PullFromMirror: ""
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  configFile: /home/pwallrab/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/pwallrab/.local/share/containers/storage
  graphRootAllocated: 139179020288
  graphRootUsed: 70563901440
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 22
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/pwallrab/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762611
  BuiltTime: Tue Oct 31 15:30:11 2023
  GitCommit: ""
  GoVersion: go1.20.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

Podman in a container

Privileged Or Rootless

None

Upstream Latest Release

Additional environment details

Additional information

No response

I did some further research. It seems to be working with Podman v4.4.2 but breaks with v4.7.2. I did not test the releases in-between.

Also the directory structure I am using looks like this:

/usr/share/containers/systemd/ -> /var/quadlets -> /var/quadlet_<digest> containing one or multiple symlinks to /var/apps/<digest>/*.container files

sudo ls -lisa /usr/share/containers/systemd/
total 12
1835009 4 drwxr-xr-x.  2 root root 4096 Dec 11 21:28 .
      2 4 drwxr-xr-x. 26 root root 4096 Dec 11 21:36 ..
1835010 4 lrwxrwxrwx.  1 root root  164 Dec 11 21:28 fedora-minimal.container -> /var/apps/71c21f8e19e238078518b08d85cf539bc36e7cbcadfd933bf639e8b2dfb3bfa6_24f7bf533a98ef846027224f451eb1c4e90ffcde9e4edc958c3eb270cc8d8f1b/fedora-minimal.container

ls -lisa /usr/share/containers/systemd
262511 0 lrwxrwxrwx. 1 root root 13 Dec 11 21:28 /usr/share/containers/systemd -> /var/quadlets

sudo ls -lisa /var/quadlet_31cf070f6cc223bdbda05dce2cfd9c8f21fa7d9958a17461ba74b2af634ab43c/
total 12
1835009 4 drwxr-xr-x.  2 root root 4096 Dec 11 21:28 .
      2 4 drwxr-xr-x. 26 root root 4096 Dec 11 21:36 ..
1835010 4 lrwxrwxrwx.  1 root root  164 Dec 11 21:28 fedora-minimal.container -> /var/apps/71c21f8e19e238078518b08d85cf539bc36e7cbcadfd933bf639e8b2dfb3bfa6_24f7bf533a98ef846027224f451eb1c4e90ffcde9e4edc958c3eb270cc8d8f1b/fedora-minimal.container

sudo ls -lisa /var/apps/71c21f8e19e238078518b08d85cf539bc36e7cbcadfd933bf639e8b2dfb3bfa6_24f7bf533a98ef846027224f451eb1c4e90ffcde9e4edc958c3eb270cc8d8f1b/fedora-minimal.container
1703944 4 -rw-r--r--. 1 root root 488 Dec 11 21:27 /var/apps/71c21f8e19e238078518b08d85cf539bc36e7cbcadfd933bf639e8b2dfb3bfa6_24f7bf533a98ef846027224f451eb1c4e90ffcde9e4edc958c3eb270cc8d8f1b/fedora-minimal.container

containers / podman