Add support for fetching unadvertised SHA-1 (allow-tip-sha1-in-want capability) #11958

Open JustAnotherArchivist opened 4 years ago

I'd like to see the allow-tip-sha1-in-want capability added to Gitea's Git server so that objects that aren't advertised in upload-pack can be fetched by a client. The primary use case that I'm aware of is fetching commits that were on branches that have been force-pushed over. In the context of pull requests, these previous commits can be important to understand the review process, for example. GitHub and GitLab.com (haven't tested GitLab CE) support this.

For those who may not be that familiar with how a git clone works, here's a quick summary: the client connects to the server and executes git upload-pack. This returns the server's capabilities and a list of refs. The client then determines which of these refs they want and sends a series of want <SHA-1> lines to the server of the objects (and their history) it wants. The server then returns a packfile with those objects, and the client creates the desired local directory structure.

However, even if the client fetches everything advertised by the server (e.g. with --mirror), this only returns reachable commits. An example of unreachable commits are branches that were force-pushed over. Most notably, that is common in pull requests when major refactors of the PR code are necessary to keep the history clean after the merge. As long as garbage collection hasn't been run on the server, those commits still exist, and they can be fetched if the server supports the allow-tip-sha1-in-want: git fetch remote sha1 will attempt to fetch the commit with hash sha1 from the remote remote.

Example using GitHub and the recent PR #11032:

git clone --mirror https://github.com/go-gitea/gitea.git
cd gitea.git
git show e4c89ba252341b8222dd72389b7407c3a0e05c99 → fatal: bad object e4c89ba252341b8222dd72389b7407c3a0e05c99
git fetch origin e4c89ba252341b8222dd72389b7407c3a0e05c99
git show e4c89ba252341b8222dd72389b7407c3a0e05c99 → commit e4c89ba252341b8222dd72389b7407c3a0e05c99 etc.

As of version 1.11.6 and the version deployed on gitea.com (see below), Gitea does not support step 4 (error: Server does not allow request for unadvertised object <SHA-1>), which amounts to a want e4c89ba252341b8222dd72389b7407c3a0e05c99 request from the client where that commit is not reachable from any ref.

Example using https://gitea.com/gitea/homebrew-gitea/pulls/94:

git clone --mirror https://gitea.com/gitea/homebrew-gitea.git
cd homebrew-gitea.git
git show 238dfca61957cd2b8b7c690a1ccea6481104c5b3 → bad object (as expected)
git fetch origin 238dfca61957cd2b8b7c690a1ccea6481104c5b3 → error: Server does not allow request for unadvertised object 238dfca61957cd2b8b7c690a1ccea6481104c5b3

Since Gitea essentially just delegates the HTTP requests handling to Git (after auth checks etc.) in routers/repo/http.go, I believe the only change that would be needed to support this would be enabling uploadpack.allowTipSHA1InWant in the repositories.

On a related note, supporting allow-reachable-sha1-in-want aka uploadpack.allowReachableSHA1InWant may also be useful.

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

This needs git v2.5.0

This is simply an option in .gitconfig or git config files in the repositories. Git capabilities are passed through by Gitea unaltered.

If you set uploadpack.allowTipSHA1InWant, uploadpack.hideRefs, uploadarchive.allowUnreachable etc. appropriately in git you can do this.

I'm seeing this error too in some gitea actions pipelines when the CI tries to run actions/checkout@v3 on a merge commit on the main branch after a pull request merge.

It's interesting because some pipelines work and others don't, it looks there is a race condition.

Any idea on how to solve this?

You can set your own git config options by : Support changing git config through app.ini #24860

https://docs.gitea.com/administration/config-cheat-sheet#git---config-options-gitconfig

https://github.com/go-gitea/gitea/blob/ec1feedbf582b05b6a5e8c59fb2457f25d053ba2/custom/conf/app.example.ini#L712C6-L712C6

I'm seeing this error too in some gitea actions pipelines when the CI tries to run actions/checkout@v3 on a merge commit on the main branch after a pull request merge.

It's interesting because some pipelines work and others don't, it looks there is a race condition.

Any idea on how to solve this?

If I understand correctly, the checkout action shouldn't depend on this "unadvertised SHA-1" feature.

What's the difference of these PRs? Maybe branch of the merged PR has been deleted so the CI are not able to checkout.

For this issue, I think it can be closed.

I'm not sure whether the CI situation is related. But the original issue, i.e. enabling allowTipSHA1InWant (and possibly the other two options mentioned above) by default to allow fetching unadvertised commits, just like GitHub and GitLab do, is not resolved. I understand that it can be done by changing the configuration, but it should be on by default. Please reopen.

Sorry for the delay @wxiaoguang . The workflow that sometimes seems to show this issue is:

Make a Pull request (everything works fine)
Merge the PR
The branch is deleted by default on merge as specified in the repository configuration
A new workflow run is triggered on the commit on main

The checkout of that run fails when pulling from main

::group::Fetching the repository
[command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin   +2dee84d8629c61d22ee57a7536fc58bf635c40e5:refs/remotes/origin/main
error: Server does not allow request for unadvertised object 2dee84d8629c61d22ee57a7536fc58bf635c40e5
The process '/usr/bin/git' failed with exit code 128

I'm guessing that the jobs that don't fail are the ones that the checkout phase of the main job is run before the automatic deletion of the pull request branch

Fetching the whole history with fetch-depth: 0 worked for us:

      - name: Checkout the codebase
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

Also getting this issue with actions/checkout@v3 using the demo.yaml workflow.

2024-03-06 10:18:08 [Gitea Actions Demo/Explore-Gitea-Actions]   | ::group::Fetching the repository
2024-03-06 10:18:08 [Gitea Actions Demo/Explore-Gitea-Actions]   | [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +1cb8e8e887467799ae13c4cf3a5efeba9e051d8c:refs/remotes/origin/actions
2024-03-06 10:18:09 [Gitea Actions Demo/Explore-Gitea-Actions]   | error: Server does not allow request for unadvertised object 1cb8e8e887467799ae13c4cf3a5efeba9e051d8c
2024-03-06 10:18:09 [Gitea Actions Demo/Explore-Gitea-Actions]   | The process '/usr/bin/git' failed with exit code 128
2024-03-06 10:18:09 [Gitea Actions Demo/Explore-Gitea-Actions]   | Waiting 13 seconds before trying again

Actions run fine at first, but if I go back to old commits and try re-run the action it fails with this Server does not allow request for unadvertised object error. I'm on the same branch and haven't done any PRs, I'm simply a number of commits ahead - but I guess these commits are now somehow 'unreachable'? I've tested using with: fetch-depth: 0 and this does solve the problem. But I feel by default everything should be configured nicely to work - especially when uses: actions/checkout@v3 is part of the demo workflow and there is no note to say it can break if you don't use it with fetch-depth: 0 . It can be puzzling to the end user if the demo workflow which used to run fine suddenly stops working. So I would support @JustAnotherArchivist proposal to make the necessary configs enabled by default.

We close issues that need feedback from the author if there were no new comments for a month. :tea:

go-gitea / gitea

Add support for fetching unadvertised SHA-1 (allow-tip-sha1-in-want capability) #11958