search

containers / buildah

A tool that facilitates building OCI images.
https://buildah.io
Apache License 2.0
7.51k stars 786 forks source link

Podman / Buildah is not respecting --build-arg param passed in and keep using the old values from its cache #4367

Closed pganeshar closed 2 years ago

pganeshar commented 2 years ago

Description

When I run podman build command with --build-arg passed in, podman / buildah keep ignoring whats passed in and using the old values from its cache instead. I am running this inside a CentOS Stream release 8 rootful container with kernel overlay. I am using the latest podman (4.2.0) and buildah (1.27.0) versions.

Steps to reproduce the issue:

  1. Log into a container
  2. . Run podman buildx build --isolation chroot --jobs 0 --platform linux/amd64 -t test-image --network host --build-arg GIT_HASH=7189d88e75391042f0ba1da774781de720382c6f --build-arg SERVICE_VERSION=6.0.0 .
  3. . Now, run podman buildx build --isolation chroot --jobs 0 --platform linux/amd64 -t test-image --network host --build-arg GIT_HASH=c828b28b538a3ec49c861c2ecf4d2a028d6cf201 --build-arg SERVICE_VERSION=7.0.0 .
  4. Podman will ignore the passed in values which are GIT_HASH=c828b28b538a3ec49c861c2ecf4d2a028d6cf201 and SERVICE_VERSION=7.0.0 and use the values from step 2 instead.

Here is the Dockerfile I used:

FROM public.ecr.aws/docker/library/golang:1.18 as builder1

COPY go.mod /app/go.mod
COPY go.sum /app/go.sum

# internal proxy
ENV GOPROXY=a.b.c
ENV GONOSUMDB=x.y.z
WORKDIR /app
RUN go mod download

ARG GIT_HASH=""
ARG SERVICE_VERSION=""
# pass the git hash and service version to the next stage
ENV GIT_HASH=${GIT_HASH}
ENV SERVICE_VERSION=${SERVICE_VERSION}

# ------------------------------------------------------------------------
#  The second stage container, for building the Go Binary
# ------------------------------------------------------------------------
FROM builder1 AS builder2
COPY . /app
WORKDIR /app/cmd/webapp

RUN echo "Service Version: $SERVICE_VERSION"
RUN echo "Git Hash: $GIT_HASH"

Describe the results you received:

podman buildx build --isolation chroot --jobs 0 --platform linux/amd64 -t test-image --network host --build-arg GIT_HASH=c828b28b538a3ec49c861c2ecf4d2a028d6cf201 --build-arg SERVICE_VERSION=7.0.0 .
[5/5] STEP 1/9: FROM gcr.io/distroless/static:debug
[1/5] STEP 1/11: FROM public.ecr.aws/docker/library/golang:1.18 AS builder1
[3/5] STEP 1/1: FROM public.ecr.aws/docker/library/busybox:latest AS busybox
[1/5] STEP 2/11: COPY go.mod /app/go.mod
--> ff4a8eb070e
--> Using cache 928a281829376ab5417c78ea8447925c4d8e72486709e004359da451682faa82
--> 928a2818293
[1/5] STEP 3/11: COPY go.sum /app/go.sum
--> Using cache 0b0b67a625b0133d5853899d9ace43b5ce23448c6dc69e8c84812388edc2fbe6
--> 0b0b67a625b
[1/5] STEP 4/11: ENV GOPROXY=a.b.c
--> Using cache 85e6eb587b01dfd47a80960f0a2a3909bb98693de75bc32ebe44035c76a94cd4
--> 85e6eb587b0
[1/5] STEP 5/11: ENV GONOSUMDB=x.y.z
--> Using cache 6f4e3cec1921f1322df0b869ac3604b89f5a197b38318073e981020e258ee520
--> 6f4e3cec192
[1/5] STEP 6/11: WORKDIR /app
--> Using cache 42891350cc1f89026c58150e0b695e036ddac151bdbdfc05f0230faf56b0b931
--> 42891350cc1
[1/5] STEP 7/11: RUN go mod download
--> Using cache f2e16e1812bc1c866fa122a4654743fdd8ff692e08eafe4d8c3a202cb2c2cab9
--> f2e16e1812b
[1/5] STEP 8/11: ARG GIT_HASH=""
--> Using cache b94d6a4faf2315de547d7cc7674d386c57b07c0695767044bcdb1c470dee46b2
--> b94d6a4faf2
[1/5] STEP 9/11: ARG SERVICE_VERSION=""
--> Using cache db6ff5b6061c5fd1e640caa5b4b27ee1657ca7acdcf4a7f907f280b268fd23ac
--> db6ff5b6061
[1/5] STEP 10/11: ENV GIT_HASH=${GIT_HASH}
--> Using cache f9f55a9308e5d499a1da2a424e3f260e285cf436019328385e42c367eab91293
--> f9f55a9308e
[5/5] STEP 2/9: COPY --from=busybox /bin/sleep /bin/sleep
[1/5] STEP 11/11: ENV SERVICE_VERSION=${SERVICE_VERSION}
--> Using cache 0e99751d1e15a61b27c579f0e04134f6b19fca4628b1634b585f4b572cd71df4
--> 0e99751d1e1
[2/5] STEP 1/6: FROM 0e99751d1e15a61b27c579f0e04134f6b19fca4628b1634b585f4b572cd71df4 AS builder2
[2/5] STEP 2/6: COPY . /app
--> Using cache 4d2d2924b8d74d7ed5b1cf6a0b7fef5e1aa88708ce8de1a078bd60b1829a07f7
--> 4d2d2924b8d
[5/5] STEP 3/9: USER nonroot:nonroot
--> Using cache 6389e1d18be8f8de500fe377352118cbf5fe36070fa7f4d6a2bb25df713e2284
--> 6389e1d18be
[5/5] STEP 4/9: COPY --from=builder2 --chown=nonroot:nonroot /app/cmd/webapp/app_binary /app/app_binary
--> 4c024023601
[2/5] STEP 3/6: WORKDIR /app/cmd/webapp
--> 5ab5a60afa4
[2/5] STEP 4/6: RUN echo "Service Version: $SERVICE_VERSION"
Service Version: 6.0.0
--> 258de0dfe8e
[2/5] STEP 5/6: RUN echo "Git Hash: $GIT_HASH"
Git Hash: 7189d88e75391042f0ba1da774781de720382c6f

Describe the results you expected: I expected Podman to print the passed in values: Service Version: 7.0.0 Git Hash: c828b28b538a3ec49c861c2ecf4d2a028d6cf201

Output of rpm -q buildah or apt list buildah:

bash-4.4# rpm -q buildah
buildah-1.27.0-2.module_el8.7.0+1216+b022c01d.x86_64

Output of buildah version:

bash-4.4# buildah version
Version:         1.27.0
Go Version:      go1.18.4
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.22.0
Git Commit:
Built:           Wed Sep 21 13:09:15 2022
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of podman version if reporting a podman build issue:

bash-4.4# podman version
Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Built:        Wed Sep 21 13:15:04 2022
OS/Arch:      linux/amd64

*Output of `cat /etc/release`:**

bash-4.4# cat /etc/*release
CentOS Stream release 8
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
CentOS Stream release 8
CentOS Stream release 8

Output of uname -a:

Linux 5.4.209-116.367.amzn2.x86_64 #1 SMP Wed Aug 31 00:09:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf: Note: I have left out all the commented out lines by "#" below:

[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"

[storage.options]
additionalimagestores = [
]

pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}

[storage.options.overlay]
mountopt = "nodev,metacopy=on"
[storage.options.thinpool]

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.27.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.4-1.module_el8.7.0+1216+b022c01d.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: ec95327311bd1f0a118cadc9b24032df370babe4'
  cpuUtilization:
    idlePercent: 99.85
    systemPercent: 0.04
    userPercent: 0.11
  cpus: 48
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: file
  hostname: centos-amd-he-golang-8httq
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.209-116.367.amzn2.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 46397603840
  memTotal: 99656683520
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.5-1.module_el8.7.0+1216+b022c01d.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module_el8.7.0+1216+b022c01d.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 216h 25m 11.00s (Approximately 9.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 536858308608
  graphRootUsed: 49465700352
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 41
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1663766104
  BuiltTime: Wed Sep 21 13:15:04 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0
flouthoc commented 2 years ago

@pganeshar

I am expecting you are only experiencing this with first three steps of stage, and this happens because following steps are cached and does not uses arg so history keeps matching with intermediate image which is already in storage. I suggest using echo before these steps.

FROM builder1 AS builder2
COPY . /app
WORKDIR /app/cmd/webapp

Does changing Containerfile to this fixes issue for you ?

FROM public.ecr.aws/docker/library/golang:1.18 as builder1

COPY go.mod /app/go.mod
COPY go.sum /app/go.sum

# internal proxy
ENV GOPROXY=a.b.c
ENV GONOSUMDB=x.y.z
WORKDIR /app
RUN go mod download

ARG GIT_HASH=""
ARG SERVICE_VERSION=""
# pass the git hash and service version to the next stage
ENV GIT_HASH=${GIT_HASH}
ENV SERVICE_VERSION=${SERVICE_VERSION}

# ------------------------------------------------------------------------
#  The second stage container, for building the Go Binary
# ------------------------------------------------------------------------
FROM builder1 AS builder2
RUN echo "Service Version: $SERVICE_VERSION"
RUN echo "Git Hash: $GIT_HASH"
COPY . /app
WORKDIR /app/cmd/webapp
pganeshar commented 2 years ago

@flouthoc

Thanks very much for your comment and I am very sorry about the late response. Had to deal with different priorities so didn't have a chance to test this out until now.

Yes, your suggestion works. Does doing the echo after FROM builder1 AS builder2breaks the cache?

We actually depend on these values as they get baked into the microservice image. And then Spinnaker uses these to deploy to different environments.

Here is the command we use to run go build:

RUN GO_VERSION=$(grep -e "GO_SDK_URL " ../../go.mod | cut -d' ' -f2) &&\
    SERVICE_VERSION=${SERVICE_VERSION:-$(grep -i "service_version: " ../../settings.yaml | cut -d ' ' -f2 | head -1)} &&\
    CGO_ENABLED=0 go build \
    --ldflags "-s -X go.sdkVersion=$GO_VERSION -X go.serviceVersion=$SERVICE_VERSION -X go.gitHash=$GIT_HASH" \
    -o app_binary

Can I rely on this approach? I have tested a few times and it seemed to have worked every single time, i.e Podman used the values from build args instead of getting them from its cache.

When I searched the net I came across this article: https://stackoverflow.com/questions/35134713/disable-cache-for-specific-run-commands

Do you think its better to use that approach to bust the cache with "date +%s"?

flouthoc commented 2 years ago

@pganeshar Is $GIT_HASH being propagated from the build-arg ? If yes then this should work for busting a cache for specific run instruction where HASH was changed.

I also like the idea of passing date +%s as build-arg and using that to bust cache for a specific run instruction I think both should work here.

pganeshar commented 2 years ago

@flouthoc Yes, GIT_HASH does get passed to podman build command using --build-arg but the weird this is that even this din't bust the cache and podman used previous values from its cache.

For example, I ran release/16.0.0 pipeline first, build ran correctly using the values passed in --build-arg params. I then ran pipeline for release/5.0.0 (which has a different git hash) but podman ran the build using the values from release/16.0.0 instead. So not sure why it used the cache in this scenario?

I am going to do some testing using the UNIX timestamp variable and hoping I get consistent results.

Thanks.

flouthoc commented 2 years ago

For example, I ran release/16.0.0 pipeline first, build ran correctly using the values passed in --build-arg params. I then ran pipeline for release/5.0.0 (which has a different git hash) but podman ran the build using the values from release/16.0.0 instead. So not sure why it used the cache in this scenario?

For this I think there must a case like https://github.com/containers/buildah/issues/4367#issuecomment-1291729248 , but if you can provide a small reproducer I can actually create a working example which will work.

I am going to do some testing using the UNIX timestamp variable and hoping I get consistent results.

Let me know your findings then we can discuss it here.

pganeshar commented 2 years ago

@flouthoc , here the steps to reproduce:

  1. Run the following build:

    podman buildx build --isolation chroot --jobs 0 --platform linux/amd64 -t ECR_URL/sandbox/test-go-microservice:16.0.0-1668700466336-signed-amd64 --network host --build-arg bitbucket_id=**** --build-arg 'bitbucket_token=****' --build-arg GIT_HASH=b8372e2ce8752d2567fd99a91d229a369fd9429e --build-arg SERVICE_VERSION=16.0.0 .
[5/5] STEP 1/9: FROM gcr.io/distroless/static:debug
[1/5] STEP 1/13: FROM public.ecr.aws/docker/library/golang:1.18 AS builder1
[3/5] STEP 1/1: FROM public.ecr.aws/docker/library/busybox:latest AS busybox
Trying to pull public.ecr.aws/docker/library/busybox:latest...
Trying to pull gcr.io/distroless/static:debug...
Trying to pull public.ecr.aws/docker/library/golang:1.18...
Getting image source signatures
Copying blob sha256:3cb762126a62bc9bf7824a3c8d7e44e871fe18cb87876e6734bdd439a8674c54
Copying blob sha256:195ea6a58ca87a18477965a6e6a8623112bde82c5b568a29c56ce4581b6e6695
Copying blob sha256:c85a0be79bfba309d1f05dc40b447aa82b604593531fed1e7e12e4bef63483a5
Copying blob sha256:52908dc1c386fab0271a2b84b6ef4d96205a98a8c8801169554767172e45d8c7
Copying blob sha256:a8ca11554fce00d9177da2d76307bdc06df7faeb84529755c648ac4886192ed1
Copying blob sha256:e4e46864aba2e62ba7c75965e4aa33ec856ee1b1074dda6b478101c577b63abd
Getting image source signatures
Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed
Getting image source signatures
Copying blob sha256:495a88a03ecbbcbfde13499d262e093f95eadd163befc03c106c9a5d7e57eefd
Copying blob sha256:8fdb1fc20e240e9cae976518305db9f9486caa155fd5fc53e7b3a3285fe8a990
Copying config sha256:bc01a3326866eedd68525a4d2d91d2cf86f9893db054601d6be524d5c9d03981
Writing manifest to image destination
Storing signatures
Copying blob sha256:9601c0a52d295d64421ff81e82d2440cfdd3664cb109490ad34cb39f3a007602
--> bc01a332686
Copying config sha256:1367e76602742bca4790f0f8b1954fe4b3d677617326be8e89e724d5083d6bfd
Writing manifest to image destination
Storing signatures
[5/5] STEP 2/9: COPY --from=busybox /bin/sleep /bin/sleep
Copying config sha256:e53ef54a405f8a28710b399b3f4a0ef0350d6d470088cb26d39f70f9a454939f
Writing manifest to image destination
Storing signatures
--> d2ad78284b3
[5/5] STEP 3/9: USER user:user
[1/5] STEP 2/13: COPY go.mod /app/go.mod
--> a0fe83e0a9d
[5/5] STEP 4/9: COPY --from=builder2 --chown=user:user /app/cmd/webapp/app_binary /app/app_binary
--> 09ddf30164a
[1/5] STEP 3/13: COPY go.sum /app/go.sum
--> c75d92b753b
[1/5] STEP 4/13: ENV GOPROXY=https://athens.xxx.yyy.zzz
--> 6e3d3ad566c
[1/5] STEP 5/13: ENV GONOSUMDB=bitbucket.yyy.zzz
--> e55252f0e1e
[1/5] STEP 6/13: WORKDIR /app
--> 082d34e4317
[1/5] STEP 7/13: RUN go mod download
--> 39c7a7bc7da
[1/5] STEP 8/13: ARG GIT_HASH=""
--> 9771a207a83
[1/5] STEP 9/13: ARG SERVICE_VERSION=""
--> bd6ccecac77
[1/5] STEP 10/13: ENV GIT_HASH=${GIT_HASH}
--> 4b32d68bd9a
[1/5] STEP 11/13: ENV SERVICE_VERSION=${SERVICE_VERSION}
--> 21b74e5c4a9
[2/5] STEP 1/6: FROM 43325dab9f6f81ecaf0aa0f20ccf6be344efea7d95450a283fe1f7f52081cee9 AS builder2
[2/5] STEP 2/6: COPY . /app
--> 6431d051bf7
[2/5] STEP 3/6: WORKDIR /app/cmd/webapp
--> 5f92ea21931
[2/5] STEP 4/6: RUN echo "Service Version Stage 2: $SERVICE_VERSION"
Service Version 2: 16.0.0
--> 0cf3816ecf1
[2/5] STEP 5/6: RUN echo "Git Hash Stage 2: $GIT_HASH"
Git Hash 2: b8372e2ce8752d2567fd99a91d229a369fd9429e
--> 19efc03858a
[2/5] STEP 6/6: RUN GO_VERSION=$(grep -e "bitbucket.yyy.zzz/go/go " ../../go.mod | cut -d' ' -f2) &&    SERVICE_VERSION=${SERVICE_VERSION:-$(grep -i "service_version: " ../../settings.yaml | cut -d ' ' -f2 | head -1)} &&    CGO_ENABLED=0 go build     --ldflags "-s -X bitbucket.yyy.zzz/go/go.sdkVersion=$GO_VERSION -X bitbucket.yyy.zzz/go/go.serviceVersion=$SERVICE_VERSION -X bitbucket.yyy.zzz/go/go.gitHash=$GIT_HASH"     -o app_binary
--> 16837f6e7ef
--> a3cecf20f5b

  2. Now, run second build (with different version and git hash):

    podman buildx build --isolation chroot --jobs 0 --platform linux/amd64 -t ECR_URL/sandbox/test-go-microservice:4.0.0-1668547712824-signed-amd64 --network host --build-arg bitbucket_id=**** --build-arg 'bitbucket_token=****' --build-arg GIT_HASH=d0a62e4ea80f56e7b6786f08b6040a5674192436 --build-arg SERVICE_VERSION=4.0.0 .
[5/5] STEP 1/10: FROM gcr.io/distroless/static
[1/5] STEP 1/11: FROM public.ecr.aws/docker/library/golang:1.18 AS builder1
[3/5] STEP 1/1: FROM public.ecr.aws/docker/library/busybox:latest AS busybox
[4/5] STEP 1/1: FROM ECR_URL/common-distroless/secrets-helper:latest AS secrets
Trying to pull gcr.io/distroless/static:latest...
Trying to pull ECR_URL/common-distroless/secrets-helper:latest...
Getting image source signatures
Copying blob sha256:863c7bd51caa4c9419552b92b2c0612f3a168ae089ee7e40d42f5c3a36c8ffb9
Copying config sha256:390e1e47e10d9e6e16f4e06298c941dfa6f4f2a3a3140685b1eeeb8aa361bb10
Writing manifest to image destination
Storing signatures
--> 390e1e47e10
--> bc01a332686
[1/5] STEP 2/11: COPY go.mod /app/go.mod
--> Using cache a2fd94a30c717cd18f00b4b3e47f1f6d716c05b272b0d8d9b888d643828af0ae
--> a2fd94a30c7
[1/5] STEP 3/11: COPY go.sum /app/go.sum
--> Using cache 779e6a99b3b836485fa5ebb8fef3db3072e562926b32e278ae72c4453df54e51
--> 779e6a99b3b
[1/5] STEP 4/11: ENV GOPROXY=https://athens.xxx.yyy.zzz
--> Using cache 8aac32ffcf1ad020069841c477af18f3213191095071f710869ddb05190744e2
--> 8aac32ffcf1
[1/5] STEP 5/11: ENV GONOSUMDB=bitbucket.yyy.zzz
--> Using cache f0e0d1525e256eada80d96827b8de6d2354f241b43c3900d32821308eebb3b49
--> f0e0d1525e2
[1/5] STEP 6/11: WORKDIR /app
--> Using cache beabd5ced74997c382eb7c553ff3882cdb75d60778b820f912c2cfa306f14c4f
--> beabd5ced74
[1/5] STEP 7/11: RUN go mod download
--> Using cache 8a293c91a5138c514cb9d746d752c8def8b0b9c3066f5b9085d0e867bc174e5e
--> 8a293c91a51
[1/5] STEP 8/11: ARG GIT_HASH=""
--> Using cache 71b61ae74514a84897f126e2c01c771f56432f80367945fffacec51f2d70e4be
--> 71b61ae7451
[1/5] STEP 9/11: ARG SERVICE_VERSION=""
--> Using cache a88cfbe7c8d1c0fa67aff90ca41273b2f9adc4e9e2cd188b1e20b58264a81517
--> a88cfbe7c8d
[1/5] STEP 10/11: ENV GIT_HASH=${GIT_HASH}
--> Using cache e083d003770bb087a588534b6fcdfdfe57a8b77fc039cc158c95f7118454b46e
--> e083d003770
[1/5] STEP 11/11: ENV SERVICE_VERSION=${SERVICE_VERSION}
--> Using cache 43d2bf8fd5e7c3593dae47693b8996c8d07cb7d44a13776b8efadbed6eb51206
--> 43d2bf8fd5e
[2/5] STEP 1/6: FROM 43d2bf8fd5e7c3593dae47693b8996c8d07cb7d44a13776b8efadbed6eb51206 AS builder2
[2/5] STEP 2/6: COPY . /app
Getting image source signatures
Copying blob sha256:8fdb1fc20e240e9cae976518305db9f9486caa155fd5fc53e7b3a3285fe8a990
Copying config sha256:21e2f91908c8fe969cf3b5e2f9d0aa4afdea8a9aabf679e722c752f66fb20d0f
Writing manifest to image destination
Storing signatures
[5/5] STEP 2/10: COPY --from=busybox /bin/sleep /bin/sleep
--> 3abbee42f13
[2/5] STEP 3/6: WORKDIR /app/cmd/webapp
--> 9e81f941454
[2/5] STEP 4/6: RUN echo "Service Version Stage 2: $SERVICE_VERSION"
--> 403fa8477fe
[5/5] STEP 3/10: COPY --from=secrets /bin/ /bin/
Service Version Stage 2: 16.0.0
--> 6a32a30caf3
[5/5] STEP 4/10: USER user:user
--> 69a0e6285ae
[2/5] STEP 5/6: RUN echo "Git Hash Stage 2: $GIT_HASH"
--> e6551ba3639
[5/5] STEP 5/10: COPY --from=builder2 --chown=user:user /app/cmd/webapp/app_binary /app/app_binary
Git Hash Stage 2: b8372e2ce8752d2567fd99a91d229a369fd9429e
--> 5b360770359
[2/5] STEP 6/6: RUN GO_VERSION=$(grep -e "bitbucket.yyy.zzz/go/go " ../../go.mod | cut -d' ' -f2) &&    SERVICE_VERSION=${SERVICE_VERSION:-$(grep -i "service_version: " ../../settings.yaml | cut -d ' ' -f2 | head -1)} &&    CGO_ENABLED=0 go build     --ldflags "-s -X bitbucket.yyy.zzz/go/go.sdkVersion=$GO_VERSION -X bitbucket.yyy.zzz/go/go.serviceVersion=$SERVICE_VERSION -X bitbucket.yyy.zzz/go/go.gitHash=$GIT_HASH"     -o app_binary
--> 7c05726cf3f
--> 2a8ced9e014

As you can see on the second build (#2), it should have used SERVICE_VERSION=4.0.0 and GIT_HASH=d0a62e4ea80f56e7b6786f08b6040a5674192436 but instead Podman has used the values from step 1.

Here is the Dockerfile:

# ------------------------------------------------------------------------
#  The first stage container, for setting up and downloading dependencies
# ------------------------------------------------------------------------
FROM public.ecr.aws/docker/library/golang:1.18 as builder1
# only copy the go.mod and go.sum for better docker caching in first stage
COPY go.mod /app/go.mod
COPY go.sum /app/go.sum
# use our internal athens proxy
ENV GOPROXY=https://athens.xxx.yyy.zzz
ENV GONOSUMDB=bitbucket.yyy.zzz
WORKDIR /app
RUN go mod download

ARG GIT_HASH=""
ARG SERVICE_VERSION=""
# pass the git hash and service version to the next stage
ENV GIT_HASH=${GIT_HASH}
ENV SERVICE_VERSION=${SERVICE_VERSION}

# ------------------------------------------------------------------------
#  The second stage container, for building the Go Binary
# ------------------------------------------------------------------------
FROM builder1 AS builder2
COPY . /app
WORKDIR /app/cmd/webapp

RUN echo "Service Version Stage 2: $SERVICE_VERSION"
RUN echo "Git Hash Stage 2: $GIT_HASH"

RUN GO_VERSION=$(grep -e "bitbucket.yyy.zzz/go/go " ../../go.mod | cut -d' ' -f2) &&\
    SERVICE_VERSION=${SERVICE_VERSION:-$(grep -i "service_version: " ../../settings.yaml | cut -d ' ' -f2 | head -1)} &&\
    CGO_ENABLED=0 go build \
    --ldflags "-s -X bitbucket.yyy.zzz/go/go.sdkVersion=$GO_VERSION -X bitbucket.yyy.zzz/go/go.serviceVersion=$SERVICE_VERSION -X bitbucket.yyy.zzz/go/go.gitHash=$GIT_HASH" \
    -o application_binary

FROM public.ecr.aws/docker/library/busybox:latest as busybox

# ------------------------------------------------------------------------
#  The third stage container, for running the application
# ------------------------------------------------------------------------
FROM gcr.io/distroless/static

USER user:user
COPY --from=builder2 --chown=user:user /app/cmd/webapp/application_binary /app/app_binary
COPY --from=builder2 --chown=user:user /app/settings.yaml /app/settings.yaml
COPY --from=builder2 --chown=user:user /app/settings_prod.yaml /app/settings_prod.yaml
COPY --from=builder2 --chown=user:user /app/settings_lab.yaml /app/settings_lab.yaml

WORKDIR /app
ENTRYPOINT ["/app/application_binary"]

Thanks.

flouthoc commented 2 years ago

Hi @pganeshar , I was able to reproduce it and found the problem. Second stage used first one as base image and never honored the CLI args. Could you try this modified Dockerfile it should make it work for your use-case.

# ------------------------------------------------------------------------
#  The first stage container, for setting up and downloading dependencies
# ------------------------------------------------------------------------
FROM public.ecr.aws/docker/library/golang:1.18 as builder1
# only copy the go.mod and go.sum for better docker caching in first stage
COPY go.mod /app/go.mod
COPY go.sum /app/go.sum
# use our internal athens proxy
ENV GOPROXY=https://athens.xxx.yyy.zzz
ENV GONOSUMDB=bitbucket.yyy.zzz
WORKDIR /app
RUN go mod download

ARG GIT_HASH=""
ARG SERVICE_VERSION=""
# pass the git hash and service version to the next stage
ENV GIT_HASH=${GIT_HASH}
ENV SERVICE_VERSION=${SERVICE_VERSION}
RUN echo "Service Version Stage 1: $SERVICE_VERSION"
RUN echo "Git Hash Stage 1: $GIT_HASH"

# ------------------------------------------------------------------------
#  The second stage container, for building the Go Binary
# ------------------------------------------------------------------------
FROM builder1 AS builder2
COPY . /app
WORKDIR /app/cmd/webapp

ARG GIT_HASH=""
ARG SERVICE_VERSION=""
# pass the git hash and service version to the next stage
ENV GIT_HASH=${GIT_HASH}
ENV SERVICE_VERSION=${SERVICE_VERSION}
RUN echo "Service Version Stage 2: $SERVICE_VERSION"
RUN echo "Git Hash Stage 2: $GIT_HASH"

RUN GO_VERSION=$(grep -e "bitbucket.yyy.zzz/go/go " ../../go.mod | cut -d' ' -f2) &&\
    SERVICE_VERSION=${SERVICE_VERSION:-$(grep -i "service_version: " ../../settings.yaml | cut -d ' ' -f2 | head -1)} &&\
    CGO_ENABLED=0 go build \
    --ldflags "-s -X bitbucket.yyy.zzz/go/go.sdkVersion=$GO_VERSION -X bitbucket.yyy.zzz/go/go.serviceVersion=$SERVICE_VERSION -X bitbucket.yyy.zzz/go/go.gitHash=$GIT_HASH" \
    -o application_binary

FROM public.ecr.aws/docker/library/busybox:latest as busybox

# ------------------------------------------------------------------------
#  The third stage container, for running the application
# ------------------------------------------------------------------------
FROM gcr.io/distroless/static

USER user:user
COPY --from=builder2 --chown=user:user /app/cmd/webapp/application_binary /app/app_binary
COPY --from=builder2 --chown=user:user /app/settings.yaml /app/settings.yaml
COPY --from=builder2 --chown=user:user /app/settings_prod.yaml /app/settings_prod.yaml
COPY --from=builder2 --chown=user:user /app/settings_lab.yaml /app/settings_lab.yaml

WORKDIR /app
ENTRYPOINT ["/app/application_binary"]
flouthoc commented 2 years ago

Since I was able to reproduce and make it work I have closed the issue with the fix suggested above but if you feel something is unresolved we can re-open this :)

flouthoc commented 2 years ago

Another simple example for someone who wants to try this out with a simpler file.

FROM alpine as one
RUN echo hello
RUN touch hey
ARG CACHEBUST=1
ENV CACHEBUST=${CACHEBUST}
RUN echo "cache_burst_1: $CACHEBUST"

FROM one as second
RUN echo starting second stage
ARG CACHEBUST=1
ENV CACHEBUST=${CACHEBUST}
RUN echo "cache_burst_2: $CACHEBUST"
COPY --from=0 hey .

@pganeshar If solution in https://github.com/containers/buildah/issues/4367#issuecomment-1340585567 works for you I think this problem can be a really good blog material about busting inline cache in buildah using --build-arg