The sha256 of registry cache config are mismatched for rerun multi-stage config.

[huxuan]

Contributing guidelines

• [X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• [X] The documentation does not mention anything about my problem
• [X] There are no open or closed issues that are related to my problem

Description

The logs show that the sha256 for all layers for the registry cache are the same except for config.

Expected behaviour

The sha256 for the registry cache config should be the same.

Actual behaviour

• The sha256 for the registry cache config are mismatched which leads to untaged packages.

Repository URL

https://github.com/huxuan/ss-python

Workflow run URL

/actions/runs/7709174339/job/21009975797

YAML workflow

container-publish:
    needs: release-publish
    permissions:
      contents: read
      packages: write
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          password: ${{ secrets.GITHUB_TOKEN }}
          registry: ghcr.io
          username: ${{ github.actor }}
      - env:
          SOURCE_DATE_EPOCH: 0
        uses: docker/build-push-action@v5
        with:
          build-args: |
            PYTHON_VERSION=${{ matrix.python-version }}
          cache-from: |
            type=registry,ref=ghcr.io/${{ github.repository }}:build-cache-dev-py${{ matrix.python-version }}
          cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:build-cache-dev-py${{ matrix.python-version }},mode=max
          context: .
          file: .devcontainer/prebuild/.devcontainer/Dockerfile
          provenance: false
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:dev-py${{ matrix.python-version }}
            ghcr.io/${{ github.repository }}:dev-py${{ matrix.python-version }}-${{ github.ref_name }}
          target: dev
      - env:
          SOURCE_DATE_EPOCH: 0
        uses: docker/build-push-action@v5
        with:
          build-args: |
            PYTHON_VERSION=${{ matrix.python-version }}
            PDM_BUILD_SCM_VERSION=${{ github.ref_name }}
          cache-from: |
            type=registry,ref=ghcr.io/${{ github.repository }}:build-cache-dev-py${{ matrix.python-version }}
            type=registry,ref=ghcr.io/${{ github.repository }}:build-cache-prod-py${{ matrix.python-version }}
          cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:build-cache-prod-py${{ matrix.python-version }},mode=max
          context: .
          file: .devcontainer/prebuild/.devcontainer/Dockerfile
          provenance: false
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:prod-py${{ matrix.python-version }}
            ghcr.io/${{ github.repository }}:prod-py${{ matrix.python-version }}-${{ github.ref_name }}
          target: prod
    strategy:
      matrix:
        python-version:
          - '3.8'
          - '3.9'
          - '3.10'
          - '3.11'
          - '3.12'

Workflow logs

Attempt 1: https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009847395

#22 exporting cache to registry
#22 preparing build cache for export
#22 writing layer sha256:0a329b671abfc277cf19a2670d019384c564e146c9c5ed1bb97d173[377](https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009847395#step:6:382)497d6e
#22 writing layer sha256:0a329b671abfc277cf19a2670d019384c564e146c9c5ed1bb97d173377497d6e 0.2s done
#22 writing layer sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad[378](https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009847395#step:6:383)ff6dbc9e751d5d4
#22 writing layer sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad378ff6dbc9e751d5d4 0.1s done
#22 writing layer sha256:1c74526957fc2157e8b0989072dc99b9582b398c12d1dcd40270fd76231bab0c
#22 writing layer sha256:1c74526957fc2157e8b0989072dc99b9582b398c12d1dcd40270fd76231bab0c 0.1s done
#22 writing layer sha256:20f18e486fc01f87b6b311be20ae9db9cb82df60c14ad1783b86d5fa2ba623f8
#22 writing layer sha256:20f18e486fc01f87b6b311be20ae9db9cb82df60c14ad1783b86d5fa2ba623f8 0.2s done
#22 writing layer sha256:26916576c92c435f7441cb7490dd537040d28b42ab2719ea02ba909f0985a57c
#22 writing layer sha256:26916576c92c435f7441cb7490dd537040d28b42ab2719ea02ba909f0985a57c 0.2s done
#22 writing layer sha256:2878b437a07192a6cf2b2075786d0a7922b16f1cee9dad8a5f305deef52e6253
#22 writing layer sha256:2878b437a07192a6cf2b2075786d0a7922b16f1cee9dad8a5f305deef52e6253 1.3s done
#22 writing layer sha256:2f44b7a888fa005d07c031d3cfad2a1c0344207def2ab9dbb97712425ff812c1
#22 writing layer sha256:2f44b7a888fa005d07c031d3cfad2a1c0344207def2ab9dbb97712425ff812c1 0.1s done
#22 writing layer sha256:30d85599795460b2d9d24c6b87c53ec60555b601705cc83bea31632240500980
#22 writing layer sha256:30d85599795460b2d9d24c6b87c53ec60555b601705cc83bea31632240500980 0.1s done
#22 writing layer sha256:382cd2159c71b198be5533c09c741cdfecf7dc5618feb5637bbe5f036f73d89f
#22 writing layer sha256:382cd2159c71b198be5533c09c741cdfecf7dc5618feb5637bbe5f036f73d89f 0.1s done
#22 writing layer sha256:396f0fee52796a6a98fc294805b23c509beb8cd0b4556e5568b4298a0be6e3c8
#22 writing layer sha256:396f0fee52796a6a98fc294805b23c509beb8cd0b4556e5568b4298a0be6e3c8 0.1s done
#22 writing layer sha256:3f00b3697662aa214d22bb76bbbf5fa4d80f7ba9ca9f9076a9440d53bf529b83
#22 writing layer sha256:3f00b3697662aa214d22bb76bbbf5fa4d80f7ba9ca9f9076a9440d53bf529b83 0.2s done
#22 writing layer sha256:4acda0384fa4980f4c24482858b8894f5711c36ca9269d50fbbd88bbf74de8a4
#22 writing layer sha256:4acda0384fa4980f4c24482858b8894f5711c36ca9269d50fbbd88bbf74de8a4 0.1s done
#22 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#22 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.1s done
#22 writing layer sha256:69c6a95e6fa75b2a15dcc1d93e96f2b16e96be599a4c74574ab1c63126e25f97
#22 writing layer sha256:69c6a95e6fa75b2a15dcc1d93e96f2b16e96be599a4c74574ab1c63126e25f97 0.1s done
#22 writing layer sha256:75e2b45cbee50cea4b3ed4f4fe167ad5994622d77a54adde89adcfeefa3c229a
#22 writing layer sha256:75e2b45cbee50cea4b3ed4f4fe167ad5994622d77a54adde89adcfeefa3c229a 0.1s done
#22 writing layer sha256:76530ff9b004d5b0ad94cc95e54ad2749c0deb9d3d105d15b84049b006f9abed
#22 writing layer sha256:76530ff9b004d5b0ad94cc95e54ad2749c0deb9d3d105d15b84049b006f9abed 0.1s done
#22 writing layer sha256:76c111f84668456d1a37f894e2a1a6fc655cecf002d8b3f7934f4ffc14231f72
#22 writing layer sha256:76c111f84668456d1a37f894e2a1a6fc655cecf002d8b3f7934f4ffc14231f72 0.1s done
#22 writing layer sha256:8ea4555671ddb9b06b855c89880e2946201b5ced31dbf1ce8f5b872b55b224e7
#22 writing layer sha256:8ea4555671ddb9b06b855c89880e2946201b5ced31dbf1ce8f5b872b55b224e7 0.1s done
#22 writing layer sha256:ad5739181616b815fae7edc6bba689496674acbcf44e48a57fc7cc13a[379](https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009847395#step:6:384)b3a2
#22 writing layer sha256:ad57[391](https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009847395#step:6:396)81616b815fae7edc6bba689496674acbcf44e48a57fc7cc13a379b3a2 0.1s done
#22 writing layer sha256:b3ba329ce47347173b038e63b8c9a78f7cf20324c2377dc3a0904837cfbe2e23
#22 writing layer sha256:b3ba329ce47347173b038e63b8c9a78f7cf20324c2377dc3a0904837cfbe2e23 0.1s done
#22 writing layer sha256:c12af4d99b224c7583539128a04c9[395](https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009847395#step:6:400)7b1908684e50013d8b58a8e2988d7b9f
#22 writing layer sha256:c12af4d99b224c7583539128a04c93957b1908684e50013d8b58a8e2988d7b9f 0.1s done
#22 writing layer sha256:f4cb18646a15052fbff9e6a7f2af27b73ce3033e9e973709fedf06058817f092
#22 writing layer sha256:f4cb18646a15052fbff9e6a7f2af27b73ce3033e9e973709fedf06058817f092 0.1s done
#22 writing layer sha256:fa8b5ed51b617f793487eb8b5c0d947553c6d649eabd5615e35795e71c70feb2
#22 writing layer sha256:fa8b5ed51b617f793487eb8b5c0d947553c6d649eabd5615e35795e71c70feb2 0.1s done
#22 writing layer sha256:ffaf3b259c1baad7a193f840f1f0506f9debd25f0ba8d9de0033a61553d0f53d
#22 writing layer sha256:ffaf3b259c1baad7a193f840f1f0506f9debd25f0ba8d9de0033a61553d0f53d 0.1s done
#22 writing config sha256:20bd3b4cbe80bae9a81f88dade1f0f6a84ff8ae28a22ceaeec893b4ad92d21a6
#22 writing config sha256:20bd3b4cbe80bae9a81f88dade1f0f6a84ff8ae28a22ceaeec893b4ad92d21a6 0.9s done
#22 writing cache manifest sha256:11dd2cfbbd2816313b0fe862ee8c8cead80984e3d1a8e7b274e6197877ae815b
#22 preparing build cache for export 6.8s done
#22 writing cache manifest sha256:11dd2cfbbd2816313b0fe862ee8c8cead80984e3d1a8e7b274e6197877ae815b 0.7s done
#22 DONE 6.8s

Attempt 2: https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009975797

#22 exporting cache to registry
#22 preparing build cache for export
#22 writing layer sha256:0a329b671abfc277cf19a2670d019384c564e146c9c5ed1bb97d173377497d6e
#22 writing layer sha256:0a329b671abfc277cf19a2670d019384c564e146c9c5ed1bb97d173377497d6e 0.1s done
#22 writing layer sha256:1b13d4e1a46e5e969702ec92b7c787c1b6891bff7c21ad378ff6dbc9e751d5d4 0.1s done
#22 writing layer sha256:1c74526957fc2157e8b0989072dc99b9582b398c12d1dcd40270fd76231bab0c
#22 writing layer sha256:1c74526957fc2157e8b0989072dc99b9582b398c12d1dcd40270fd76231bab0c 0.1s done
#22 writing layer sha256:20f18e486fc01f87b6b311be20ae9db9cb82df60c14ad1783b86d5fa2ba623f8 0.1s done
#22 writing layer sha256:26916576c92c435f7441cb7490dd537040d28b42ab2719ea02ba909f0985a57c
#22 writing layer sha256:26916576c92c435f7441cb7490dd537040d28b42ab2719ea02ba909f0985a57c 0.1s done
#22 writing layer sha256:2878b437a07192a6cf2b2075786d0a7922b16f1cee9dad8a5f305deef52e6253 0.1s done
#22 writing layer sha256:2f44b7a888fa005d07c031d3cfad2a1c0344207def2ab9dbb97712425ff812c1
#22 writing layer sha256:2f44b7a888fa005d07c031d3cfad2a1c0344207def2ab9dbb97712425ff812c1 0.1s done
#22 writing layer sha256:30d85599795460b2d9d24c6b87c53ec60555b601705cc83bea31632240500980 0.0s done
#22 writing layer sha256:382cd2159c71b198be5533c09c741cdfecf7dc5618feb5637bbe5f036f73d89f
#22 writing layer sha256:382cd2159c71b198be5533c09c741cdfecf7dc5618feb5637bbe5f036f73d89f 0.1s done
#22 writing layer sha256:396f0fee52796a6a98fc294805b23c509beb8cd0b4556e5568b4298a0be6e3c8 0.1s done
#22 writing layer sha256:3f00b3697662aa[214](https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009975797#step:6:219)d22bb76bbbf5fa4d80f7ba9ca9f9076a9440d53bf529b83
#22 writing layer sha256:3f00b3697662aa214d22bb76bbbf5fa4d80f7ba9ca9f9076a9440d53bf529b83 0.1s done
#22 writing layer sha256:4acda0384fa4980f4c24482858b8894f5711c36ca9269d50fbbd88bbf74de8a4 0.0s done
#22 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#22 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.1s done
#22 writing layer sha256:69c6a95e6fa75b2a15dcc1d93e96f2b16e96be599a4c74574ab1c63126e25f97 0.1s done
#22 writing layer sha256:75e2b45cbee50cea4b3ed4f4fe167ad5994622d77a54adde89adcfeefa3c229a
#22 writing layer sha256:75e2b45cbee50cea4b3ed4f4fe167ad5994622d77a54adde89adcfeefa3c229a 0.1s done
#22 writing layer sha256:76530ff9b004d5b0ad94cc95e54ad2749c0deb9d3d105d15b84049b006f9abed 0.1s done
#22 writing layer sha256:76c111f84668456d1a37f894e2a1a6fc655cecf002d8b3f7934f4ffc14231f72
#22 writing layer sha256:76c111f84668456d1a37f894e2a1a6fc655cecf002d8b3f7934f4ffc14231f72 0.1s done
#22 writing layer sha256:8ea4555671ddb9b06b855c89880e2946201b5ced31dbf1ce8f5b872b55b224e7 0.1s done
#22 writing layer sha256:ad5739181616b815fae7edc6bba689496674acbcf44e48a57fc7cc13a379b3a2
#22 writing layer sha256:ad5739181616b815fae7edc6bba689496674acbcf44e48a57fc7cc13a379b3a2 0.1s done
#22 writing layer sha256:b3ba329ce47347173b038e63b8c9a78f7cf20324c2377dc3a0904837cfbe2e23 0.1s done
#22 writing layer sha256:c12af4d99b224c7583539128a04c93957b1908684e50013d8b58a8e2988d7b9f
#22 writing layer sha256:c12af4d99b224c7583539128a04c93957b1908684e50013d8b58a8e2988d7b9f 0.1s done
#22 writing layer sha256:f4cb18646a15052fbff9e6a7f2af27b73ce3033e9e973709fedf06058817f092 0.1s done
#22 writing layer sha256:fa8b5ed51b617f793487eb8b5c0d947553c6d649eabd5615e35795e71c70feb2
#22 writing layer sha256:fa8b5ed51b617f793487eb8b5c0d947553c6d649eabd5615e35795e71c70feb2 0.1s done
#22 writing layer sha256:ffaf3b259c1baad7a193f840f1f0506f9debd25f0ba8d9de0033a61553d0f53d 0.1s done
#22 writing config sha256:336be0cc3b08f2c94c32c45c4234ae9075cb071431e19c4787dc7e51e1a5b54d
#22 writing config sha256:336be0cc3b08f2c94c32c45c4234ae9075cb071431e19c4787dc7e51e1a5b54d 0.8s done
#22 writing cache manifest sha256:fdd22d7352e00b75d70666789bd229fa24f5afebb9a[220](https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009975797#step:6:225)def5db47f648d3f50c
#22 preparing build cache for export 3.6s done
#22 writing cache manifest sha256:fdd22d7352e00b75d70666789bd[229](https://github.com/huxuan/ss-python/actions/runs/7709174339/job/21009975797#step:6:234)fa24f5afebb9a220def5db47f648d3f50c 0.9s done
#22 DONE 3.6s

BuildKit logs

No response

Additional info

1. To make everything reproducible as much as possible, both

   env:
    SOURCE_DATE_EPOCH: 0

   and

   with:
    provenance: false

   are set according to https://github.com/docker/build-push-action/issues/994 and https://github.com/docker/build-push-action/issues/894

2. The sha256 of layers and config for single-stage are the same both for the resulting image and registry cache.
3. The sha256 of layers and config for multi-stage are the same only for the resulting image.
4. The cache overall works as expected, the only side effects are untaged packages for the registry caches.

[crazy-max]

can you post in #994 instead as it seems a duplicate of it? Thanks.