Cached repo/submodules not working with build-push-action

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

[ ] The documentation does not mention anything about my problem
[ ] There are no open or closed issues that are related to my problem

Description

I cache my repository submodules (as those rarely change), but docker/build-push-action@v5 attempts to fetch all branches and submodules by itself.. It fails, because if submodules are cached, checkout with ssh-key is never used.

Not sure why buil-push-action would attempt to fetch everything by itself? Is it possible to disable that?

P.S. Same workflow runs fine when I use docker/bake-action instead.

Expected behaviour

build-push-action should not be responsible for handling repository checkout.

Actual behaviour

build-push-action is attempting to checkout repo when its not intended.

Repository URL

No response

Workflow run URL

No response

YAML workflow

- name: Create private key to access private repos.
      uses: webfactory/ssh-agent@v0.8.0
      with:
        ssh-private-key: ${{ inputs.SSH_PRIVATE_KEY }}
    - name: Checkout monodoo
      uses: actions/checkout@v3
    - name: Get submodules cache key
      id: submodules-key
      uses: ./.github/actions/lstree
      with:
        paths: src/extra,src/monodoo/odoo
    - name: Cache Submodules
      id: cache-submodule
      uses: actions/cache@v3
      with:
        path: |
          src/extra
          src/monodoo/odoo
        key: ${{ steps.submodules-key.outputs.name }}
    - name: Checkout submodules
      if: steps.cache-submodule.outputs.cache-hit != 'true'
      uses: actions/checkout@v3
      with:
        ssh-key: ${{ inputs.SSH_PRIVATE_KEY }}
        submodules: recursive
    - name: Login to docker registry
      uses: docker/login-action@v3
      with:
        registry: ${{ inputs.registry }}
        username: ${{ github.actor }}
        password: ${{ inputs.GITHUB_TOKEN }}
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3
    - name: Build and push
      uses: docker/build-push-action@v5
      with:
        push: true
        tags: ghcr.io/myorg/demo:${{ inputs.tag }}
        build-contexts: |
          project-src=src/projects/demo
          extra-src=src/extra
        cache-from: |
          type=gha,scope=demo
        cache-to: |
          type=gha,mode=max,scope=demo

Workflow logs

When buildx runs in workflow:

Buildx version
/usr/bin/docker buildx build --build-context project-src=src/projects/demo --build-context extra-src=src/extra --cache-from type=gha,scope=demo --cache-to type=gha,mode=max,scope=demo --iidfile /tmp/docker-actions-toolkit-QhvhJZ/iidfile --provenance mode=min,inline-only=true,builder-id=https://github.com/myorg/monodoo/actions/runs/6720746685 --secret id=GIT_AUTH_TOKEN,src=/tmp/docker-actions-toolkit-QhvhJZ/tmp-636197-Voezr4RKDVzb --tag ghcr.io/myorg/demo:cmt-123123123123 --metadata-file /tmp/docker-actions-toolkit-QhvhJZ/metadata-file --push https://github.com/myorg/monodoo.git#3242423423423sfsdfsdfsf

Then it tries to fetch everything:

#0 0.039 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
#0 0.051 fatal: Not a valid object name 97071de07858cdf224ca8e5df8bfa9dbf2d34339^{commit}
...
...
#1 2.309 git@github.com: Permission denied (publickey).
#1 2.311 fatal: Could not read from remote repository.

BuildKit logs

No response

Additional info

No response

docker / build-push-action