Error while checking out: "fatal: reference is not a tree"

[jpkrohling]

I'm sorry if this is not the right place for this issue, but it's where it looked most appropriate to me.

When re-triggering a workflow for the PR #620 in the Jaeger Operator, the checkout step fails with:

git checkout --progress --force 76143e9bd1c40fc0b71fd45a987c80b9db0a9096
##[error]fatal: reference is not a tree: 76143e9bd1c40fc0b71fd45a987c80b9db0a9096
Removed matchers: 'checkout-git'
##[remove-matcher owner=checkout-git]
##[error]Git checkout failed with exit code: 128
##[error]Exit code 1 returned from process: file name '/home/runner/runners/2.157.0/bin/Runner.PluginHost', arguments 'action "GitHub.Runner.Plugins.Repository.CheckoutTask, Runner.Plugins"'.

I can confirm that the commit exists in the fork, so, I'm not sure what the action is complaining about:

$ git remote add rubenvp8510 https://github.com/rubenvp8510/jaeger-operator
$ git fetch rubenvp8510 
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 53 (delta 41), reused 43 (delta 41), pack-reused 9
Unpacking objects: 100% (53/53), done.
From https://github.com/rubenvp8510/jaeger-operator
 * [new branch]        Issue-304            -> rubenvp8510/Issue-304
 * [new branch]        Issue-399            -> rubenvp8510/Issue-399
 * [new branch]        Issue-442            -> rubenvp8510/Issue-442
 * [new branch]        Issue-443            -> rubenvp8510/Issue-443
 * [new branch]        Issue-557            -> rubenvp8510/Issue-557
 * [new branch]        Issue-568            -> rubenvp8510/Issue-568
 * [new branch]        Issue-598            -> rubenvp8510/Issue-598
 * [new branch]        master               -> rubenvp8510/master
 * [new branch]        revert-391-streame2e -> rubenvp8510/revert-391-streame2e
 * [new branch]        token-propagation    -> rubenvp8510/token-propagation

$ git checkout --progress --force 76143e9bd1c40fc0b71fd45a987c80b9db0a9096
Note: checking out '76143e9bd1c40fc0b71fd45a987c80b9db0a9096'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 76143e9b add resource limits for spark dependencies cronjob

[TingluoHuang]

This is really weird, looks like the wrong GITHUB_REF and GITHUB_SHA get send to the runner, so the checkout action does not checkout the PR merge ref, instead, it tries to checkout the SHA from the PR source branch, which is wrong.

I can confirm the problem in my repo as well:

GITHUB_EVENT_NAME=pull_request
GITHUB_REF=refs/heads/PR_test   <-- this should be something like refs/pull/123/merge
GITHUB_REPOSITORY=bbq-beets/ting-test
GITHUB_SHA=4ba7f853093a0989aece593b33de5e8dec7ce579

looks like this happened recently since the PR run I have last night is able to get the right merge ref.

GITHUB_EVENT_NAME=pull_request
GITHUB_REF=refs/pull/6/merge
GITHUB_REPOSITORY=bbq-beets/ting-test
GITHUB_SHA=d9fd1a7c30b9349606d0db4b9f777bcbbb4376e4

@chrispat

[TingluoHuang]

@jpkrohling thanks for reporting, we are investigating.

[TingluoHuang]

Looks like the wrong REF only happened when we try to re-run a failed workflow.

[cdb]

At the moment, the payloads on re-run are currently not exactly the same as the original run, which is not what users are expecting of course. @iheanyi is working on fixing this as part of in the PR he just referenced ^

[jpkrohling]

Any news on this one? Sometimes, I have the feeling that it's fixed, but then, recent re-checks fail with the same symptom...

[soluchok]

@iheanyi any luck with this?

[iheanyi]

@soluchok Apologies, we're still working on a fix for this! I was on vacation, so I'll hopefully get one in soon and update this ticket accordingly.

[runspired]

@iheanyi I'm hitting this too, but in a slightly different pattern.

Example failed run: https://github.com/emberjs/data/pull/6655/checks?check_run_id=285284611

This is a workflow that compares the built assets of the PR to master, so it does some git maneuvering:

• checkout master, run build
• checkout $GITHUB_SHA, run build again

About 3-5% of the time we hit this, re-running the check it will be fine.

For instance in the above failure the sha was 8343c5c56fbf31675b7557155849fcc07ef91bbd which didn't match anything I could find. It may have been a stale sha for the merge commit.

When rerunning the sha became: b8ed7472261b5984b431806beac6f33fb0c2de0e.

In this particular failure a force-push was involved but I've hit it at around the same rate without force-push as well. I suspect this was probably a stale sha.

[iheanyi]

@runspired Is this for re-triggering a workflow or just a regular run?

[runspired]

@iheanyi regular run. I traced down the cause I think (in the twitter thread I added you to)

[runspired]

@iheanyi thread: https://twitter.com/Runspired/status/1191746539372212224

Roughly I believe the root of the issue is a race condition between when @actions/checkout triggers the checkout and when the merge commit has been created in the repo.

If the checkout occurs before the commit is created, then checkout results in creating the merge commit itself which results in a different SHA than the SHA that is used for $GITHUB_SHA

For some reason this condition is far more common when it is a new commit to an open PR.

I suspect roughly the flow behind the scenes is (in order of A -> G)

success

- A workflow triggered
  - B create merge commit
  - C populate GITHUB_SHA
  - D push merge commit and branch to repository
     - F repository receives new merge commit
  - E begin workflow
     - G checkout PR branch on repository

failure

- A workflow triggered
  - B create merge commit
  - C populate GITHUB_SHA
  - D push merge commit and branch to repository
     - G repository receives new merge commit
  - E begin workflow
     - F checkout PR branch on repository

[runspired]

Also if anyone else hits this I fixed this in my flows by using git rev-parse HEAD to access the true SHA, and if I need to checkout another commit and come back I stash that SHA into a local file like this:

          sha=$(git rev-parse --short=8 HEAD)
          echo "HEAD sha=$sha"
          echo "GITHUB_SHA sha=$GITHUB_SHA"
          mkdir -p tmp
          echo $sha > tmp/sha-for-check.txt

Which I can then access in later steps via

sha=$(cat tmp/sha-for-check.txt)

[ramonmedeiros]

Just got the same error:

My PR HEAD is 60f77e1dbfcc35b60992f25edc89f9bda98c539b

But github.sha is a5adf7586c64f47a1bee01b449f663f8c7c840a4

https://github.com/ramonmedeiros/kimchi/pull/1/commits/60f77e1dbfcc35b60992f25edc89f9bda98c539b

[ericsciple]

@ramonmedeiros in your case, is a5adf758 the merge commit? For PRs, the action checks-out the merge commit by default. If you want to checkout the PR HEAD commit, you can do this

[ericsciple]

@iheanyi do you know whether the fix has rolled out now for the race condition?

[iheanyi]

@ericsciple The first part has been addressed, the re-run errors that is. I'm unsure if the other issues are still occurring.

[ericsciple]

Closing since I think this is fixed by actions/checkout@v2. V2 fetches a specific SHA and retries with a few delays between before failing. Whereas V1 fetched the merge PR ref.

Reopen if still occurring. Noise has gone down since last comments in November. V2 preview was early December, and then GA late December so more evidence this is fixed now.

[sebdau]

I'm seeing this error, too. Our build is not yet using YAML but classic UI-style release pipe. According to the logs, this is still using V1 git task. Is there a way to change the task version?

https://msazure.visualstudio.com/One/_build/results?buildId=28043911&view=logs&j=92a4755a-6173-591e-5c07-8dbe1165e937&t=ead66276-ec45-5be0-fa6c-361c94a3aa0c&l=5

Thx, Seb

[ericsciple]

@sebdau sorry but that is the Azure DevOps product. This repo is for GitHub Actions :(

[localheinz]

As one can see in https://github.com/sebastianbergmann/phpunit/blob/94821def76750d335505c384d05fcb3394130615/.github/workflows/ci.yml#L52-L74, we are using actions/checkout@v2 in a job:

  backward-compatibility:
    name: Backward Compatibility

    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Fetch tags
        run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*

      - name: Install PHP with extensions
        uses: shivammathur/setup-php@v2
        with:
          php-version: 7.4
          coverage: none
          extensions: intl

      - name: Run roave/backward-compatibility-check
        run: ./tools/roave-backward-compatibility-check --from=9041d82604effd0d0ab59ce67eafbc3462a2ba83

As one can also see in https://github.com/sebastianbergmann/phpunit/runs/650379867#step:5:22, running a tool that checks out a different commit (which should be available) fails with

fatal: reference is not a tree: 9041d82604effd0d0ab59ce67eafbc3462a2ba83     

Why?

[aivenkimmob]

Our build is still affected by this issue, and v2 version doesn't seem to do a difference. Similar to what @localheinz commented, we also use fetch-depth: 0 configuration which could be the common denominator with our issues.

[wl2776]

We also see this bug. I'd like to assist in fixing it ASAP by providing all necessary information for tracking down the issue.

Versions of the software we use:

GitHub Enterprise Server 2.20.19 Jenkins 2.249.3 Plugins: Git 4.4.5 Git client 3.5.1 Github API 1.116 GitHub Branch Source 2.9.1 Github checks 1.0.6 Github integration 0.2.8 Github 1.32.0 GitHub Pipeline for Blue Ocean 1.24.3

[chrispat]

@wl2776 this repo is for GitHub Actions. If you are seeing bugs in various Jenkins plugins please report those there.

[TomasHubelbauer]

The last comment is from half a year ago so I'd like to update and say this issue is indeed still happening. I can't see why it is closed, because the "145 hidden items - Load more items" button doesn't work, but if it was fix, looks like the fix is incomplete or there are multiple ways to trigger this issue.

[GerHobbelt]

Ran into this issue as I've got the same error in my github action.

For me it was resolved today by running git pull --all in the action shell script before doing anything else with the git repository.

Reasoning: seems like github creates a shallow-depth git repo by default as the git branch -a command only listed the master branch (plus the remotes/origin/master one) while the repo had at least one other branch. Only after running git pull --all as part of the action code did that other branch show up and did my other git commands start to behave as expected.

HTH.

Action YML:

name: learn-github-actions
on: [push]
jobs:
  fix-illegal-names:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: bash ./fix-bad-names.sh

shell script in repo root:

ls -lr
git config user.email github-action@hobbelt.com 
git config user.name Github-Action
git pull --all
# show the branches known to git: allows visual check in 'actions' web page
git branch -a
# create local branch when running the action
git checkout -b clean-marker --track remotes/origin/clean-marker
# `ls -lr` for visual feedback and check to see if checkout command succeeded indeed
ls -lr

# do some work on that branch (anything) -- instead of working on HEAD
echo "replace : colons in filenames"
find . -name '*:*' -print | sed -e 's/\(.*\)/mv "\1" @@ "\1"/' -e 's/@@ \(.*\):\(.*\)/\1_\2/g' > rn_us.sh
chmod a+x rn_us.sh
cat rn_us.sh
./rn_us.sh

ls -lr
find . -name '*_*' -print | grep -v '\.git'
find . -name '*_*' -print | grep -v '\.git' | xargs -n 1 -d '\n' git add 

# push back the edits into my own github repo
git commit -a -m "fixed file names for Windows compatibility"
git push --all

NOTE: I suspect the shallow repo copy as any git checkout would fail, even when specifying commit hashes instead of branch names, and that also when the commit was a close predecessor of the HEAD commit -- only spot checked this with a distance of about 5, had an idea and moved on that: that's where I came up with the git pull --all which did fix the problem for me at least.

[tornvallalexander]

Looks like the wrong REF only happened when we try to re-run a failed workflow.

This seems to be it in my case - made a dummy commit and now the workflow is running fine. Using Amplify though.

[barracuda156]

Looks like still unfixed.