kaniko cache causes binary not to be rebuilt

Our initial image building code enabled the kaniko caching flag. https://github.com/GoogleContainerTools/kaniko?tab=readme-ov-file#caching)

I observed a case where my build succeeded but the binary was a very old version. My Dockerfile looked like the one below.

ARG BUILD_IMAGE=golang:1.21
ARG RUNTIME_IMAGE=cgr.dev/chainguard/static:latest
FROM ${BUILD_IMAGE} as builder

# Build Args need to be after the FROM stage otherwise they don't get passed through
ARG VERSION=unknown
ARG DATE=unknown
ARG COMMIT=unknown

WORKDIR /workspace/

COPY . /workspace

WORKDIR /workspace/kubedr

## Build
# The LDFLAG can't be specified multiple times so we use an environment variable to build it up over multiple lines
RUN LDFLAGS="-s -w -X github.com/sailplaneai/code/kubedr/cmd/commands.version=${VERSION}" && \
    LDFLAGS="${LDFLAGS} -X github.com/sailplaneai/code/kubedr/cmd/commands.commit=${COMMIT}" && \
    LDFLAGS="${LDFLAGS} -X github.com/sailplaneai/code/kubedr/cmd/commands.date=${DATE}" && \
    CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on \
    go build \
    -ldflags "${LDFLAGS}" \
    -a -o kp github.com/sailplaneai/code/kubedr/cmd

FROM ${RUNTIME_IMAGE}

COPY --from=builder /workspace/kubedr/kp /

ENTRYPOINT ["/kp"]

Here's a link to the build logs https://cloudlogging.app.goo.gl/JtV6rsU5mnmvVFyV8

downloaded-logs-20240131-173727.csv

Also attaching a CSV

I can't tell from the logs where kaniko had a cache hit.

jlewi / hydros

kaniko cache causes binary not to be rebuilt #74