Podman is unable to read from AWS EFS (NFS) when used as additional file storage

Issue Description

When building an image stored in additional local storage (AWS EFS), podman shows the following error - `[1/4] STEP 1/7: FROM node:18-alpine AS build

time="2024-05-01T00:10:12Z" level=error msg="node:18-alpine exists in local storage but may be corrupted (remove the image to resolve the issue): layer not known"`

Steps to reproduce the issue

Provision EFS share in AWS
Mount EFS share mount -t efs -o tls fs-123...:/ /var/lib/mycontainers
Pull image using podman time podman --root /var/lib/mycontainers pull docker.io/node:18-alpine
Build image using podman build .

Note: although podman pull was successful, a "corrupt image" error occurs when building the image using podman build.

Describe the results you received

`[1/4] STEP 1/7: FROM node:18-alpine AS build

time="2024-05-01T00:10:12Z" level=error msg="node:18-alpine exists in local storage but may be corrupted (remove the image to resolve the issue): layer not known"`

Describe the results you expected

podman successfully builds an image from the additional storage.

podman info output

+ podman info

host:

  arch: amd64

  buildahVersion: 1.33.3

  cgroupControllers:

  - cpuset

  - cpu

  - cpuacct

  - blkio

  - memory

  - devices

  - freezer

  - net_cls

  - perf_event

  - net_prio

  - hugetlb

  - pids

  cgroupManager: cgroupfs

  cgroupVersion: v1

  conmon:

    package: conmon-2.1.10-1.fc39.x86_64

    path: /usr/bin/conmon

    version: 'conmon version 2.1.10, commit: '

  cpuUtilization:

    idlePercent: 91.67

    systemPercent: 1.63

    userPercent: 6.7

  cpus: 8

  databaseBackend: sqlite

  distribution:

    distribution: fedora

    variant: container

    version: "39"

  eventLogger: file

  freeLocks: 2048

  hostname: 

  idMappings:

    gidmap: null

    uidmap: null

  kernel: 5.10.209-198.858.amzn2.x86_64

  linkmode: dynamic

  logDriver: k8s-file

  memFree: 18091565056

  memTotal: 65994149888

  networkBackend: netavark

  networkBackendInfo:

    backend: netavark

    dns:

      package: aardvark-dns-1.10.0-1.fc39.x86_64

      path: /usr/libexec/podman/aardvark-dns

      version: aardvark-dns 1.10.0

    package: netavark-1.10.3-1.fc39.x86_64

    path: /usr/libexec/podman/netavark

    version: netavark 1.10.3

  ociRuntime:

    name: crun

    package: crun-1.14.3-1.fc39.x86_64

    path: /usr/bin/crun

    version: |-

      crun version 1.14.3

      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98

      rundir: /run/crun

      spec: 1.0.0

      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL

  os: linux

  pasta:

    executable: /usr/bin/pasta

    package: passt-0^20231230.gf091893-1.fc39.x86_64

    version: |

      pasta 0^20231230.gf091893-1.fc39.x86_64

      Copyright Red Hat

      GNU General Public License, version 2 or later

        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>

      This is free software: you are free to change and redistribute it.

      There is NO WARRANTY, to the extent permitted by law.

  remoteSocket:

    exists: false

    path: /run/podman/podman.sock

  security:

    apparmorEnabled: false

    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT

    rootless: false

    seccompEnabled: true

    seccompProfilePath: /usr/share/containers/seccomp.json

    selinuxEnabled: false

  serviceIsRemote: false

  slirp4netns:

    executable: /usr/bin/slirp4netns

    package: slirp4netns-1.2.2-1.fc39.x86_64

    version: |-

      slirp4netns version 1.2.2

      commit: 0ee2d87523e906518d34a6b423271e4826f71faf

      libslirp: 4.7.0

      SLIRP_CONFIG_VERSION_MAX: 4

      libseccomp: 2.5.3

  swapFree: 0

  swapTotal: 0

  uptime: 27h 29m 35.00s (Approximately 1.12 days)

  variant: ""

plugins:

  authorization: null

  log:

  - k8s-file

  - none

  - passthrough

  - journald

  network:

  - bridge

  - macvlan

  - ipvlan

  volume:

  - local

registries:

  search:

  - registry.fedoraproject.org

  - registry.access.redhat.com

  - docker.io

  - quay.io

store:

  configFile: /etc/containers/storage.conf

  containerStore:

    number: 0

    paused: 0

    running: 0

    stopped: 0

  graphDriverName: overlay

  graphOptions:

    overlay.ignore_chown_errors: "true"

    overlay.imagestore: /var/lib/mycontainers

    overlay.mount_program:

      Executable: /usr/bin/fuse-overlayfs

      Package: fuse-overlayfs-1.12-2.fc39.x86_64

      Version: |-

        fusermount3 version: 3.16.1

        fuse-overlayfs: version 1.12

        FUSE library version 3.16.1

        using FUSE kernel interface version 7.38

    overlay.mountopt: nodev,fsync=0

  graphRoot: /var/lib/containers/storage

  graphRootAllocated: 549743210496

  graphRootUsed: 60121776128

  graphStatus:

    Backing Filesystem: xfs

    Native Overlay Diff: "false"

    Supports d_type: "true"

    Supports shifting: "true"

    Supports volatile: "true"

    Using metacopy: "false"

  imageCopyTmpDir: /var/tmp

  imageStore:

    number: 0

  runRoot: /var/run/containers/storage

  transientStore: false

  volumePath: /var/lib/containers/storage/volumes

version:

  APIVersion: 4.9.0

  Built: 1706090847

  BuiltTime: Wed Jan 24 10:07:27 2024

  GitCommit: ""

  GoVersion: go1.21.6

  Os: linux

  OsArch: linux/amd64

  Version: 4.9.0

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Fuse driver already installed - https://github.com/kuberenetes-learning-group/fuse-device-plugin/blob/master/README_EN.md

This is what storage.conf looks like:


[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options] additionalimagestores = ["/var/lib/mycontainers"]

[storage.options.overlay] ignore_chown_errors = "true" mount_program = "/usr/bin/fuse-overlayfs" mountopt = "nodev,fsync=0"



### Additional information

_No response_

containers / podman