`podman build` not invalidating layers after a change in a `--mount=type=bind` directory

AndrewLipscomb commented 2 months ago

Issue Description

When building an image using podman build and a build mount of type --mount=type=bind - a change in content within that mount directory does not cause the layer of the build to be invalidated and run again.

docker build on the other hand does do this - I would expect the same behaviour between both programs

Steps to reproduce the issue

I have a demo of the problem at https://github.com/AndrewLipscomb/podman-build-cache-issue-demo/tree/main

But in short

Make content in a folder in directory
Mount that folder as part of an image build
Build the image
Change the content in the directory and rebuild the image
The image will come out the same due to that layer not invalidating

For some reproduction instructions in full

Tested with podman 5.0.3 and podman 5.1.2 - both did the same thing Docker version for comparison was

Docker version 26.1.3, build b72abbb6f0

Describe the results you received

The layer is not invalidating on a mount directory content change

Describe the results you expected

The layer should invalidate on a mount directory content change

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.12-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 97.07
    systemPercent: 0.7
    userPercent: 2.24
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2046
  hostname: andrew.swarmfarm.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.9.2-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 1502134272
  memTotal: 32409886720
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.11.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.11.0
    package: /usr/lib/podman/netavark is owned by netavark 1.11.0-2
    path: /usr/lib/podman/netavark
    version: netavark 1.11.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.15-1
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2024_06_24.1ee2eca-1
    version: |
      pasta 2024_06_24.1ee2eca
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 16142110720
  swapTotal: 17064521728
  uptime: 151h 15m 9.00s (Approximately 6.29 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/andrew.lipscomb/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrew.lipscomb/.local/share/containers/storage
  graphRootAllocated: 485565878272
  graphRootUsed: 427259781120
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 138
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrew.lipscomb/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.2
  Built: 1720733172
  BuiltTime: Fri Jul 12 07:26:12 2024
  GitCommit: 94a24974ab345324db1a1489c924af4b89d2d0e9
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.2



### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

No

### Additional environment details

_No response_

### Additional information

_No response_

flouthoc commented 1 month ago

I think this can be easily solved by adding hash of directory to the history, i'd like to propose a solution if @nalind agrees then I can create a patch.

As of now buildah adds history to images for --mount like.

Current

 {
            "created": "2024-08-13T15:07:01.335991879Z",
            "created_by": "/bin/sh -c --mount=type=bind,source=/host/path/,dst=/test,Z ls /test",
            "comment": "FROM docker.io/library/alpine:latest"
        },

Proposed

 {
            "created": "2024-08-13T15:07:01.335991879Z",
            "created_by": "/bin/sh -c --mount=type=bind,source=/home/ar/work/buildah/bin/,dst=/test,Z:<hash-of-the-mounted-directory-or-file> ls /test",
            "comment": "FROM docker.io/library/alpine:latest"
        },

nalind commented 1 month ago

When the source is in the build context or a previous stage, I guess that'll work. When it's in an image, it's going to be cheaper to use the image's digest.

flouthoc commented 1 month ago

I agree, in case of image it can be digest. I can take a stab at it.

containers / podman