RUN bind mount not cached when using extra build context

[djhoese]

Issue Description

I have a Containerfile that includes a RUN with a bind mount to a file from a second build context specified using --build-context mycon=my_dir to the podman build command. No matter what I do this step is never reused/cached and is always rerun when building. My understanding is that this shouldn't be the case.

If I use the default build context and don't specify a from= in my RUN mount then it works fine. However even if I specify an additional build context that is the base context it fails (see below).

Steps to reproduce the issue

Steps to reproduce the issue

1. Create directory test_mounting and change to it.

2. Create test_mounting.Containerfile with content below.

3. Run:

   echo "TEST" >> test.txt

4. Run

   podman build --log-level info -t test-mounting:latest --build-context mycon=. -f test_mounting.Containerfile .

Containerfile

FROM centos:7 AS compilers

WORKDIR /work

# prove that some level of caching is working
RUN echo "Current directory: ${PWD}" && ls -lR .

#COPY --from=ccap test.txt new_test.txt

RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt

RUN ls -la /work && echo "DONE" >> a_file.txt

The non-mounted RUN calls are just to show that things are run and cached or not cached.

Describe the results you received

INFO[0000] podman filtering at log level info
INFO[0000] Using boltdb as database backend
INFO[0000] Setting parallel job count to 121
STEP 1/5: FROM centos:7 AS compilers
STEP 2/5: WORKDIR /work
--> Using cache 288eadf77cb754531263c358b1ca080d8a717c8baf6ef1093bd2edc8c89a4bd1
--> 288eadf77cb7
STEP 3/5: RUN echo "Current directory: ${PWD}" && ls -lR .
--> Using cache 1eb8be4dfbc26c8e66a5ed0ce37b1ae661f845c9febb6a057e9ba507d86d60a0
--> 1eb8be4dfbc2
STEP 4/5: RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt
644 0 root 0 root 0 1717293066 1717293066 1717293066
--> 339f29b8e127
STEP 5/5: RUN ls -la /work && echo "DONE" >> a_file.txt
total 0
drwxr-xr-x 1 root root  6 Jun  2 01:51 .
dr-xr-xr-x 1 root root 17 Jun  2 01:51 ..
COMMIT test-mounting:latest
--> 8c5f5e0f331e
Successfully tagged localhost/test-mounting:latest
8c5f5e0f331e982352fb257acccae7c6e1e426f911201a483ffa5efca95815e5

Describe the results you expected

I expected the RUN with the mount to be cached. In my real world case my extra build context is not the base context and I'm using --cache-to and --cache-from. The step right after this mount RUN is a lot of compiling so it is very noticeable when it isn't cached.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 3ea3d7f99779af0fcd69ec16c211a7dc3b4efb60'
  cpuUtilization:
    idlePercent: 99.69
    systemPercent: 0.08
    userPercent: 0.23
  cpus: 40
  databaseBackend: boltdb
  distribution:
    distribution: rocky
    version: "9.4"
  eventLogger: file
  freeLocks: 2046
  hostname: bumi.ssec.wisc.edu
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 700
      size: 1
    - container_id: 1
      host_id: 1672864
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 53807
      size: 1
    - container_id: 1
      host_id: 1672864
      size: 65536
  kernel: 5.14.0-284.11.1.el9_2.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 12738236416
  memTotal: 135071936512
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-3.el9_4.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/53807/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/53807/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4287033344
  swapTotal: 8002727936
  uptime: 5394h 38m 55.00s (Approximately 224.75 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/davidh/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/davidh/.local/share/containers/storage
  graphRootAllocated: 214006284288
  graphRootUsed: 176895717376
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 36
  runRoot: /run/user/53807/containers
  transientStore: false
  volumePath: /home/davidh/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1714526144
  BuiltTime: Wed May  1 01:15:44 2024
  GitCommit: ""
  GoVersion: go1.21.9 (Red Hat 1.21.9-2.el9_4)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

$ uname -a
Linux bumi.ssec.wisc.edu 5.14.0-284.11.1.el9_2.x86_64 containers/podman#1 SMP PREEMPT_DYNAMIC Tue May 9 17:09:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -q podman
podman-4.9.4-3.el9_4.x86_64

Additional information

I originally noticed this in a local gitlab CI runner, but narrowed it down to the above example.

Side question: Does file modification time invalidate the layer cache for a COPY and/or bind mount?

[djhoese]

I'm unfamiliar with this code base, but if this is expected to work but no one has the time to debug it I'd be grateful if someone could point me to a good starting location in the code. If I find the time maybe I can at least shorten the overall time it takes for this ticket to get resolved.

[rhatdan]

If it works with a docker build then it should work with buildah. Most likely there is no one able to work on this now.

[djhoese]

Hm I thought I had tested this with docker, but it looks like you're right @rhatdan. This behaves the same with docker:

$ docker build -t test-mounting:latest --build-context mycon=. -f test_mounting.Containerfile .
[+] Building 0.9s (11/11) FINISHED                                                                                                                                                                                           docker:default
 => [internal] load .dockerignore                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                        0.0s
 => [internal] load build definition from test_mounting.Containerfile                                                                                                                                                                  0.0s
 => => transferring dockerfile: 395B                                                                                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/centos:7                                                                                                                                                                            0.0s
 => [context mycon] load .dockerignore                                                                                                                                                                                                 0.0s
 => => transferring mycon: 2B                                                                                                                                                                                                          0.0s
 => [compilers 1/5] FROM docker.io/library/centos:7                                                                                                                                                                                    0.0s
 => [context mycon] load from client                                                                                                                                                                                                   0.0s
 => => transferring mycon: 437B                                                                                                                                                                                                        0.0s
 => CACHED [compilers 2/5] WORKDIR /work                                                                                                                                                                                               0.0s
 => CACHED [compilers 3/5] RUN echo "Current directory: ${PWD}" && ls -lR .                                                                                                                                                            0.0s
 => [compilers 4/5] RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt                                                                                                     0.3s
 => [compilers 5/5] RUN ls -la /work && echo "DONE" >> a_file.txt                                                                                                                                                                      0.5s
 => exporting to image                                                                                                                                                                                                                 0.1s
 => => exporting layers                                                                                                                                                                                                                0.1s
 => => writing image sha256:6824f59b7c8a214595432afecfcde008e1ebf0bd715eb58814bf3a385f238bfd                                                                                                                                           0.0s
 => => naming to docker.io/library/test-mounting:latest 

Note no CACHE line for the RUN --mount. If I remove the Containerfile's from=mycon portion and use the base context then I see the CACHE line:

 => CACHED [compilers 2/5] WORKDIR /work                                                                                                                                                                                               0.0s
 => CACHED [compilers 3/5] RUN echo "Current directory: ${PWD}" && ls -lR .                                                                                                                                                            0.0s
 => CACHED [compilers 4/5] RUN --mount=type=bind,source=test.txt,target=test.txt stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt                                                                                                         0.0s
 => CACHED [compilers 5/5] RUN ls -la /work && echo "DONE" >> a_file.txt    

So in that case...I guess I file this with docker somewhere?

EDIT: I found an old terminal session output where in my real world use case the docker output did cache the RUN --mount with the extra context. Now I don't know what's going on.

EDIT 2: Looks like docker caches it if the extra context isn't the current directory (or maybe the same as the base context).

EDIT 3: Podman doesn't seem to cache in any of these situations.

[rhatdan]

@flouthoc PTAL

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.