podman cannot use host automounted directories over bind mounts unless pre-mounted

[paulraines68]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Bind mounted automount points that are not already mounted fail inside the container

Output of podman version:

API Version:  3.4.1-dev
Go Version:   go1.16.7
Built:        Tue Oct 19 12:11:42 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: dc08a6edf03cc2dadfe803eac14b896b44cc4721'
  cpus: 24
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: file
  hostname: nymeria.nmr.mgh.harvard.edu
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 5829
      size: 1
    - container_id: 1
      host_id: 101000000
      size: 100000
    uidmap:
    - container_id: 0
      host_id: 5829
      size: 1
    - container_id: 1
      host_id: 101000000
      size: 100000
  kernel: 4.18.0-310.el8.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 53618540544
  memTotal: 67218280448
  ociRuntime:
    name: crun
    package: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.2
      commit: 4f6c8e0583c679bfee6a899c05ac6b916022561b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/5829/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 4614778880
  swapTotal: 4614778880
  uptime: 318h 15m 30s (Approximately 13.25 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /autofs/homes/011/raines/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.5
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /autofs/homes/011/raines/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /tmp/run-5829
  volumePath: /autofs/homes/011/raines/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.1-dev
  Built: 1634659902
  BuiltTime: Tue Oct 19 12:11:42 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.4.1-dev

Package info (e.g. output of rpm -q podman or apt list podman):

  buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64
  container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch
  containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch
  crun-1.2-1.module_el8.6.0+954+963caf36.x86_64
  podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64
  podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64
  runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Its the latest version that dnf gives me on CentOS8 Stream and Yes I looked in the Guide

Additional environment details (AWS, VirtualBox, physical, etc.):

I am trying out rootless podman as an alternative to singularity for our HPC users. One first issue had to do with secondary groups which has mostly been resolved as per https://github.com/containers/podman/issues/4185

Now I have another issue with the interaction of podman with the host's automounted directories. Our nodes have a /autofs directory with a few autofs mananged subdirs each with their own maps with names like 'cluster' and 'space'. When running singularity containers we can just pass "-B /autofs" to it and all those NFS volumes are available in the container whether they were mounted at the time the container was started or not.

But in podman we are finding after passing "-v /autofs:/autofs" that only those NFS volumes already mounted work. When accessing the path of an automount not yet mounted in the container one gets the error "Symbolic link loop". Doing this does cause the automount to happen on the host side. But even trying to access the path a second time still fails with the same error.

$ df -h | grep cluster
server1:/vault/cluster/matlab 1020G  273G  748G  27% /autofs/cluster/matlab
server2:/local_mount/space/pubsw   1.1T  943G   89G  92% /autofs/cluster/pubsw
$ podman --runtime /usr/bin/crun run --security-opt label=disable --annotation run.oci.keep_original_groups=1 -v /autofs:/autofs -it --rm --userns=keep-id  budtest:1.0 sh 
$ \ls /autofs/cluster/matlab | head -2
5.0
5.3
$ ls /autofs/cluster/itgroup | head -2
ls: can't open '/autofs/cluster/itgroup': Symbolic link loop
$ ls /autofs/cluster/itgroup | head -2
ls: can't open '/autofs/cluster/itgroup': Symbolic link loop
$ exit
$ df -h | grep cluster
server1:/vault/cluster/matlab     1020G  273G  748G  27% /autofs/cluster/matlab
server2:/local_mount/space/pubsw       1.1T  943G   89G  92% /autofs/cluster/pubsw
server3:/local_mount/space/itgroup  504G  495G  9.4G  99% /autofs/cluster/itgroup

On the syslog of the host one sees

Oct 27 12:09:33 client1 automount[1704]: attempting to mount entry /autofs/cluster/itgroup
Oct 27 12:09:34 client1 automount[1704]: mounted /autofs/cluster/itgroup
Oct 27 12:09:34 client1 automount[1704]: do_mount_indirect: indirect trigger not valid or already mounted /autofs/cluster/core
Oct 27 12:09:34 client1 automount[1704]: dev_ioctl_send_fail:499: AUTOFS_DEV_IOCTL_FAIL: error Invalid argument
Oct 27 12:09:34 client1 automount[1704]: do_mount_indirect: indirect trigger not valid or already mounted /autofs/cluster/core

The last two lines repeats many times

Is this understood? Any solution?

Yes, if one is certain of what remote volumes one will need running podman with explict bind options like "-v /autofs/cluster/itgroup:/autofs/cluster/itgroup" even when it is not currently mounted works as it forces the mount at the podman start time. But I would like to avoid having to do that to enable generic podman based scripts that users can use without thinking through all the locations that could be accessed -- like we do in Singularity.

[mheon]

I think we'd need to look at Singularity and figure out what exactly they're doing to make this work.

Can you provide more detailed info on how Singularity is being invoked vs how Podman is invoked? Are you mounting specific directories in /autofs with Podman but the whole /autofs directory with Singularity?

[rhatdan]

Does -v /autofs:/autofs:slave work?

[paulraines68]

Does -v /autofs:/autofs:slave work?

Yes, that does work! Thanks, did not know about that.

[paulraines68]

@mheon

In Singularity I am simply running:

singularity exec -B /autofs -B /run/user fedora.sif bash