buildah bud doesn't use cached layers when combined with multi stage build and --label somelabel=somevalue

[Romain-Geissler-1A]

Description

While testing one of our OCI build tool against both docker and podman, I noticed that the tests of my tool are one order of magnitude slower on podman rather than docker. Trying to find a minimal reproducer, it seems related to multi-stage builds combined with labels passed on the command line (if labels are directly written in the containerfile, it's ok). In the following reproducer, I expect the layer with sleep 3 to be cached after a first run, but it is not, leading to always rebuilding images completely. In my real case, this sleep 3 is actually a dnf install command.

Steps to reproduce the issue:

1. Start a container with the very latest buildah: rgeissler@ncerndobedev6097:~> podman run -t -i --rm --privileged --pull=always quay.io/buildah/upstream
2. Then create a minimal containerfile:

   
   [root@eb194bf8dfa3 /]# mkdir /build-context
   [root@eb194bf8dfa3 /]# cat > /build-context/Dockerfile <<END_OF_DOCKERFILE
   FROM fedora AS useless_stage_1
   FROM useless_stage_1 AS stage_2

RUN sleep 3 END_OF_DOCKERFILE

3. Build this image once, enable layer caching, and with an explicit label provided on the command line. It builds fine, doing a `sleep 3`:

[root@eb194bf8dfa3 /]# buildah bud --layers=true --label somelabel=somevalue /build-context/ [1/2] STEP 1/1: FROM fedora AS useless_stage_1 Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull registry.fedoraproject.org/fedora:latest... Getting image source signatures Copying blob ad5077952f52 done | Copying config 919a420d29 done | Writing manifest to image destination --> 5c4fc42cf62f [2/2] STEP 1/3: FROM 5c4fc42cf62ffa7e1649f932ad3b90bb0f39739531291fd204b2021007f3a4ce AS stage_2 [2/2] STEP 2/3: RUN sleep 3 --> 999f19301ff6 [2/2] STEP 3/3: LABEL "somelabel"="somevalue" [2/2] COMMIT --> f8c8d0a71f64 f8c8d0a71f64f6c14e46c0de8f61bcabfd4c39a71ea8b99cdd26ccaeda765c56

4. Re-run the exact same command, hoping that this time cache is used and `sleep 3` is not run. However it's not using cached layer for the sleep command:

[root@eb194bf8dfa3 /]# buildah bud --layers=true --label somelabel=somevalue /build-context/ [1/2] STEP 1/1: FROM fedora AS useless_stage_1 --> b075ebbee3a6 [2/2] STEP 1/3: FROM b075ebbee3a6450ba3eebbc14319cfe187e4c98dc47c4cb4facb0be7ba798795 AS stage_2 [2/2] STEP 2/3: RUN sleep 3 --> a7223a7828a9 [2/2] STEP 3/3: LABEL "somelabel"="somevalue" [2/2] COMMIT --> 61615826c9b7 61615826c9b7a5091f111b15e22a9ad3178ae565ea6b3a65f8f96f4516eae490

5. Re-running the same build twice, but without explicit `--label` flag on the command line uses a cached build during the second invokation:

[root@eb194bf8dfa3 /]# buildah bud --layers=true /build-context/ [1/2] STEP 1/1: FROM fedora AS useless_stage_1 --> 919a420d29c6 [2/2] STEP 1/2: FROM 919a420d29c6f5ae0bdc8d1872387d3a878d7f69debce9e24f3f2e0506b2ba0d AS stage_2 [2/2] STEP 2/2: RUN sleep 3 [2/2] COMMIT --> f5684459425d f5684459425d3f88134b202c6dece0c0e2a2b91860d3bc29a536ad5b2c138e7a [root@eb194bf8dfa3 /]# buildah bud --layers=true /build-context/ [1/2] STEP 1/1: FROM fedora AS useless_stage_1 --> 919a420d29c6 [2/2] STEP 1/2: FROM 919a420d29c6f5ae0bdc8d1872387d3a878d7f69debce9e24f3f2e0506b2ba0d AS stage_2 [2/2] STEP 2/2: RUN sleep 3 --> Using cache f5684459425d3f88134b202c6dece0c0e2a2b91860d3bc29a536ad5b2c138e7a --> f5684459425d f5684459425d3f88134b202c6dece0c0e2a2b91860d3bc29a536ad5b2c138e7a

6. Adding direction the `LABEL` inside the original containerfile results in the build using cache correctly immediately at the first run:

[root@eb194bf8dfa3 /]# echo 'LABEL "somelabel"="somevalue"' >>/build-context/Dockerfile [root@eb194bf8dfa3 /]# buildah bud --layers=true /build-context/ [1/2] STEP 1/1: FROM fedora AS useless_stage_1 --> 919a420d29c6 [2/2] STEP 1/3: FROM 919a420d29c6f5ae0bdc8d1872387d3a878d7f69debce9e24f3f2e0506b2ba0d AS stage_2 [2/2] STEP 2/3: RUN sleep 3 --> Using cache f5684459425d3f88134b202c6dece0c0e2a2b91860d3bc29a536ad5b2c138e7a --> f5684459425d [2/2] STEP 3/3: LABEL "somelabel"="somevalue" [2/2] COMMIT --> 2d4b8f439951 2d4b8f43995156779b1d852b08f419347d9607e72a77d8e35d1f2c3d80ea8f49

**Describe the results you received:**

Layer cache is not always used.

**Describe the results you expected:**

In the above described scenarios, I would expect that layer cache is always used.

**Output of `rpm -q buildah` or `apt list buildah`:**

buildah-1.31.0-1.20230731174246479315.main.43.g8af2dc4ea.x86_64

**Output of `buildah version`:**

Version: 1.32.0-dev Go Version: go1.20.6 Image Spec: 1.1.0-rc.4 Runtime Spec: 1.1.0 CNI Spec: 1.0.0 libcni Version: image Version: 5.27.0-dev Git Commit: Built: Mon Jul 31 17:47:23 2023 OS/Arch: linux/amd64 BuildPlatform: linux/amd64

**Output of `cat /etc/*release`:**

Fedora release 38 (Thirty Eight) NAME="Fedora Linux" VERSION="38 (Container Image)" ID=fedora VERSION_ID=38 VERSION_CODENAME="" PLATFORM_ID="platform:f38" PRETTY_NAME="Fedora Linux 38 (Container Image)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:38" DEFAULT_HOSTNAME="fedora" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/" SUPPORT_URL="https://ask.fedoraproject.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=38 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=38 SUPPORT_END=2024-05-14 VARIANT="Container Image" VARIANT_ID=container Fedora release 38 (Thirty Eight) Fedora release 38 (Thirty Eight)

**Output of `uname -a`:**

Linux eb194bf8dfa3 5.14.0-162.6.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 30 07:36:03 EDT 2022 x86_64 GNU/Linux

[Romain-Geissler-1A]

Note: in my case I have this issue in real life, using podman 4.4. from RHEL 9.

[flouthoc]

Thanks for reporting I can reproduce this issue.

[flouthoc]

@Romain-Geissler-1A Issue is happening because history is being created incorrectly for the first stage, meanwhile till I diagnose the root cause and create a patch for this a workaround is

FROM fedora AS useless_stage_1
RUN echo "dummy stmt"

FROM useless_stage_1 AS stage_2
RUN sleep 3

[Romain-Geissler-1A]

Thanks for the hint.

Actually I had already updated my build tool (it's a tool that generates Dockerfile automatically, that's why it sometimes generate degenerated cases like this) so that we no longer pass --label flags on the command line, but we directly write LABEL statements directly in the generated dockerfile.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc any update?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.