search

docker / build-push-action

GitHub Action to build and push Docker images with Buildx
https://github.com/marketplace/actions/build-and-push-docker-images
Apache License 2.0
4.42k stars 561 forks source link

[Bug] buildx failed with: ERROR: failed to solve: failed to read dockerfile: failed to load cache key: repository does not contain ref #1222

Closed chandanpg closed 2 months ago

chandanpg commented 3 months ago

Contributing guidelines

I've found a bug, and:

Description

I noticed recently that we are getting error in CICD when we are calling this action. This error is coming when you raise a Pull request and you merge it immediately. You Docker Build Process(which is calling this action) during Pull Request would fail with below error.

Error: buildx failed with: ERROR: failed to solve: failed to read dockerfile: failed to load cache key: repository does not contain ref refs/pull/9/merge, output: ""

Expected behaviour

Job should go to success even when your Pull Request is merged/closed.

Actual behaviour

Job is failing when Pull Request is merged immediately

Repository URL

No response

Workflow run URL

No response

YAML workflow

- name: Build
      id: build
      uses: docker/build-push-action@v6
      with:
        push: ${{ github.event_name != 'pull_request' }}
        network: host
        file: Dockerfile
        provenance: false
        github-token: ${{ inputs.TOKEN }}
        secret-files: |
            npmrc=${{ env.NPMRC_PATH }}
        secrets: |
            "TOKEN=${{ inputs.TOKEN }}"
        build-args: |
            "GHA_RUN_NUMBER=${{ inputs.GHA_RUN_NUMBER }}"
            ${{ inputs.DOCKER_ARGS }}
        tags: |
            "us-east4-docker.pkg.dev/${{ inputs.GCP_PROJECT_ID }}/${{ env.REPO_NAME }}/${{ inputs.IMAGE_NAME }}:latest"

Workflow logs

No response

BuildKit logs

No response

Additional info

No response

crazy-max commented 2 months ago

Can you provide workflow logs and BuildKit logs please?

chandanpg commented 2 months ago

Hi, I have attached the logs as well

crazy-max commented 2 months ago

Hi, I have attached the logs as well

I don't see them

chandanpg commented 2 months ago

Looks like Earlier did not upload But i have done now.

3CI Build.zip

crazy-max commented 2 months ago

This error is coming when you raise a Pull request and you merge it immediately

I'm not quite sure what that means. Does the PR got merged but workflow is still running? In that case it might be that the ref get removed when merged and therefore BuildKit is not able to fetch?

Looking at the workflow logs, I see the checkout action uses another ref:

2024-09-02T07:47:05.1328165Z ##[group]Fetching the repository
2024-09-02T07:47:05.1336097Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +c5890f87dd023da50edd68c661f16134495e8cb5:refs/remotes/pull/9/merge
2024-09-02T07:47:05.3928376Z From https://github.com/my-org/my-repo
2024-09-02T07:47:05.3930879Z  * [new ref]         c5890f87dd023da50edd68c661f16134495e8cb5 -> pull/9/merge
...
2024-09-02T07:47:05.4042568Z ::group::Checking out the ref
2024-09-02T07:47:05.4043346Z ##[group]Checking out the ref
2024-09-02T07:47:05.4044939Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/9/merge
2024-09-02T07:47:05.4181348Z Note: switching to 'refs/remotes/pull/9/merge'.

I wonder if we should do something similar. I'm taking a look.

crazy-max commented 2 months ago

In the meantime you can use the path context: https://github.com/docker/build-push-action/?tab=readme-ov-file#path-context

    - name: Build
      id: build
      uses: docker/build-push-action@v6
      with:
        context: .
        push: ${{ github.event_name != 'pull_request' }}
        network: host
        file: Dockerfile
        provenance: false
        github-token: ${{ inputs.TOKEN }}
        secret-files: |
            npmrc=${{ env.NPMRC_PATH }}
        secrets: |
            "TOKEN=${{ inputs.TOKEN }}"
        build-args: |
            "GHA_RUN_NUMBER=${{ inputs.GHA_RUN_NUMBER }}"
            ${{ inputs.DOCKER_ARGS }}
        tags: |
            "us-east4-docker.pkg.dev/${{ inputs.GCP_PROJECT_ID }}/${{ env.REPO_NAME }}/${{ inputs.IMAGE_NAME }}:latest"
chandanpg commented 2 months ago

This error is coming when you raise a Pull request and you merge it immediately

I'm not quite sure what that means. Does the PR got merged but workflow is still running? In that case it might be that the ref get removed when merged and therefore BuildKit is not able to fetch?

Looking at the workflow logs, I see the checkout action uses another ref:

2024-09-02T07:47:05.1328165Z ##[group]Fetching the repository
2024-09-02T07:47:05.1336097Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +c5890f87dd023da50edd68c661f16134495e8cb5:refs/remotes/pull/9/merge
2024-09-02T07:47:05.3928376Z From https://github.com/my-org/my-repo
2024-09-02T07:47:05.3930879Z  * [new ref]         c5890f87dd023da50edd68c661f16134495e8cb5 -> pull/9/merge
...
2024-09-02T07:47:05.4042568Z ::group::Checking out the ref
2024-09-02T07:47:05.4043346Z ##[group]Checking out the ref
2024-09-02T07:47:05.4044939Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/9/merge
2024-09-02T07:47:05.4181348Z Note: switching to 'refs/remotes/pull/9/merge'.

I wonder if we should do something similar. I'm taking a look.

Yes correct. PR is getting merged immediately after opening the Pull request. When Pull Request is merged, Workflow for Pull request is still running .

chandanpg commented 2 months ago

@crazy-max Can you please prioritize this as it impacting many users

crazy-max commented 2 months ago

So after investigating it seems the actions/checkout uses a different refspec for pull requests:

/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +9c36961ff5326ad03565ce4e13d9ec9cc21c007c:refs/remotes/pull/37/merge
  From https://github.com/docker/test-docker-action
   * [new ref]         9c36961ff5326ad03565ce4e13d9ec9cc21c007c -> pull/37/merge
...
/usr/bin/git sparse-checkout disable
/usr/bin/git config --local --unset-all extensions.worktreeConfig
/usr/bin/git checkout --progress --force refs/remotes/pull/37/merge
  Note: switching to 'refs/remotes/pull/37/merge'.

  You are in 'detached HEAD' state. You can look around, make experimental
  changes and commit them, and you can discard any commits you make in this
  state without impacting any branches by switching back to a branch.

  If you want to create a new branch to retain commits you create, you may
  do so (now or later) by using -c with the switch command. Example:

    git switch -c <new-branch-name>

  Or undo this operation with:

    git switch -

  Turn off this advice by setting config variable advice.detachedHead to false

  HEAD is now at 9c36961 Merge 9f57cef2960d8343491fd84d8cd8347780291eae into 97e7623b357e0f30b487846b976c2355f91d0245

It uses a refspec +9f57cef2960d8343491fd84d8cd8347780291eae:refs/remotes/pull/37/merge to fetch the merged commit (9f57cef2960d8343491fd84d8cd8347780291eae) and then updates the local reference refs/remotes/pull/9/merge with this commit. The + sign forces the update even if it results in a non-fast-forward update.

In our case we are using the provided github.ref from github context, here refs/pull/37/merge:

#1 [internal] load git source https://github.com/docker/test-docker-action.git#refs/pull/37/merge
#1 0.030 Initialized empty Git repository in /var/lib/docker/overlay2/mlza1t7j8fmpzcahtgc8ir5wo/diff/
#1 ERROR: repository does not contain ref refs/pull/37/merge, output: ""

I tried fetching directly from commit sha as well but didn't work:

      -
        name: Build
        id: build
        uses: docker/build-push-action@master
        with:
          context: "${{ github.server_url }}/${{ github.repository }}.git#${{ github.sha }}"
          file: ./hello.Dockerfile
#1 [internal] load git source https://github.com/docker/test-docker-action.git#9c36961ff5326ad03565ce4e13d9ec9cc21c007c
#1 0.030 Initialized empty Git repository in /var/lib/docker/overlay2/aygqug38q2r4ce2mf9gqwqcmh/diff/
#1 0.034 fatal: Not a valid object name 9c36961ff5326ad03565ce4e13d9ec9cc21c007c^{commit}
#1 0.105 fatal: could not read Username for 'https://github.com/': terminal prompts disabled
#1 ERROR: failed to fetch remote https://github.com/docker/test-docker-action.git: git error: exit status 128
stderr:
fatal: could not read Username for 'https://github.com/': terminal prompts disabled

I'm not quite sure how we can handle that in Git source https://github.com/moby/buildkit/blob/master/source/git/source.go atm :thinking: Might require changes around https://github.com/moby/buildkit/blob/185751bdc010f1f8bf1192d6e9b4ec19552d5741/source/git/source.go#L368-L405

Can you please prioritize this as it impacting many users

We only have a single report from you afaik but please share info if there are more people impacted.

Do you have limitations using the path context vs git context in the meantime as shown in https://github.com/docker/build-push-action/issues/1222#issuecomment-2324893530?

crazy-max commented 2 months ago

So after investigating

Was testing on private repo my bad so here is other investigation with a public repo: https://github.com/crazy-max/buildx-buildkit-tests/actions/runs/11073531258

The actions/checkout uses a different refspec for pull requests: https://github.com/crazy-max/buildx-buildkit-tests/actions/runs/11073531258/job/30770257801#step:4:53

image

It uses a refspec +3231c3fa12afbe81fbb290db15a994004a53ec4a:refs/remotes/pull/1/merge to fetch the merged commit (3231c3fa12afbe81fbb290db15a994004a53ec4a) and then updates the local reference refs/remotes/pull/1/merge with this commit. The + sign forces the update even if it results in a non-fast-forward update.

In our case we are using the provided github.ref from github context, here refs/pull/1/merge: https://github.com/crazy-max/buildx-buildkit-tests/actions/runs/11073531258/job/30770258123#step:4:137

image

I tried fetching directly from commit sha as well but unable to read tree: https://github.com/crazy-max/buildx-buildkit-tests/actions/runs/11073531258/job/30770257444#step:4:138

image

I'm not quite sure how we can handle that in Git source https://github.com/moby/buildkit/blob/master/source/git/source.go atm :thinking: Might require changes around https://github.com/moby/buildkit/blob/185751bdc010f1f8bf1192d6e9b4ec19552d5741/source/git/source.go#L368-L405

What seems to work is using the ref that points to the latest commit on the source branch of the pull request with refs/pull/1/head: https://github.com/crazy-max/buildx-buildkit-tests/actions/runs/11073531258/job/30770257072#step:5:140

Which is solved as https://github.com/crazy-max/buildx-buildkit-tests/actions/runs/11073531258/job/30770257072#step:5:145

#1 0.167 5f416824ecb5b62f3dc2c4016277be39f672c548   refs/pull/1/head

And this commit 5f416824ecb5b62f3dc2c4016277be39f672c548 matches from actions/checkout : https://github.com/crazy-max/buildx-buildkit-tests/actions/runs/11073531258/job/30770257801#step:4:80

   HEAD is now at 3231c3f Merge 5f416824ecb5b62f3dc2c4016277be39f672c548 into 2324e23e7a8dd2365c8e6b4003b74c57baa56c21

image

crazy-max commented 2 months ago

@chandanpg Should be fixed with latest release