Github actions cache (experimental) still seems to cache python packages after docker layer change

[sarahcthekey]

We have a workflow which includes the following step to build and push our image - docker/build-push-action@v3:

      - name: Build and push
        uses: docker/build-push-action@v3
        with:
          push: true
          tags: ${{ inputs.image_name && format(',{0}:{1}', inputs.image_name, steps.tags.outputs.tag1) || '' }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          provenance: false
          build-args: |
            ${{ secrets.build-args }}
            ${{ inputs.build-args }}
          secrets: |
            ${{secrets.build-secrets}}
          ssh: |
            default=${{ env.SSH_AUTH_SOCK }}

In the Dockerfile we install a few private python packages, here's a shortened version of our Dockerfile showing relevant bits:

FROM python:3.10

RUN pip install -U pip \
    && pip install 'poetry>=1.0.0' \
    && poetry config virtualenvs.create false \
    && poetry install $(test "$BUILD_ENV" == ci && echo "--no-dev") --no-interaction --no-ansi

ARG GIT_HASH

# Install python packages
RUN --mount=type=ssh \
    mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts \
    && poetry run pip install "git+ssh://git@github.com/**REDACTED**/python-files.git#egg=**REDACTED**&subdirectory=**REDACTED**" \
    && poetry run pip install "git+ssh://git@github.com/**REDACTED**/python-files.git#egg=**REDACTED**&subdirectory=**REDACTED**"

CMD python main.py

We use ARG GIT_HASH which will change on each github branch build, this has been added to trigger a cache miss when installing the python packages immediately after as per the official docs - If a layer changes, all other layers that come after it are also affected. When the layer with the ARG GIT_HASH command gets invalidated, all layers that follow will need to run again, too, to ensure a fresh up to date copy of the packages are installed rather than use the cache.

However this doesn't appear to be working and the image is still being built using older versions of the packages.

It only works if I completely comment out the cache-from and cache-to lines to stop using the github actions cache.

Snippet from Docker build log on github actions:

#12 [5/7] RUN --mount=type=ssh     mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts && poetry run pip install "git+ssh://git@github.com/**REDACTED**/python-files.git#egg=**REDACTED**&subdirectory=**REDACTED**" && poetry run pip install "git+ssh://git@github.com/**REDACTED**/python-files.git#egg=**REDACTED**&subdirectory=**REDACTED**" && poetry run pip install "git+ssh://git@github.com/**REDACTED**/
#12 extracting sha256:7d300de960bdd6899dd0ea1687b4d4ab68198341d0e60957463099eafd9e1eab done
#12 extracting sha256:0fc41ee805e3e6029f57a8202d1dcb6a5b15701070236117318a4ff67c0735f2
#12 extracting sha256:0fc41ee805e3e6029f57a8202d1dcb6a5b15701070236117318a4ff67c0735f2 0.2s done
#12 DONE 19.7s

#13 [6/7] COPY app /app
#13 DONE 1.8s

#14 [7/7] WORKDIR /app
#14 DONE 0.0s

#15 exporting to image
#15 exporting layers 0.1s done

Any ideas why this could be?

[tonistiigi]

Please provide a runnable reproducer for this case. I couldn't reproduce similar case locally:

FROM alpine
RUN apk add git
ARG GIT_HASH
RUN env > snapshot

docker buildx create --name=issue4084 --bootstrap
export BUILDX_BUILDER=issue4084
docker buildx build --cache-to type=local,dest=../cache .

# fully cached
docker buildx prune -a
docker buildx build --cache-from type=local,src=../cache .

# last step not cached
docker buildx prune -a
docker buildx build --cache-from type=local,src=../cache --build-arg GIT_HASH=foo .

 => CACHED [2/3] RUN apk add git                                                                                                                             0.3s
 => => sha256:8c6d1654570f041603f4cef49c320c8f6f3e401324913009d92a19132cbf1ac0 3.33MB / 3.33MB                                                               0.0s
 => => extracting sha256:8c6d1654570f041603f4cef49c320c8f6f3e401324913009d92a19132cbf1ac0                                                                    0.1s
 => => sha256:afa2557f73df36a5cce15e5b68a35d5db4d469a75e1c3a8927167abafef02786 7.14MB / 7.14MB                                                               0.1s
 => => extracting sha256:afa2557f73df36a5cce15e5b68a35d5db4d469a75e1c3a8927167abafef02786                                                                    0.1s
 => [3/3] RUN env > snapshot                                                                                                                                 0.1s