search

go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
44.16k stars 5.41k forks source link

Merging pull requests on repository with LFS not working #4463

Closed severgun closed 5 years ago

severgun commented 6 years ago

Description

Merging pull requests from web interface not working on repository with LFS. I can commit, merge and push branches on local machine with git CLI or Github for Windows, but not in gitea web.

Web Interface just fall to error 500 page after pressing Merge button.

Gitea.log:

2018/07/17 23:07:35 [...routers/repo/pull.go:533 MergePullRequest()] [E] Merge: git clone: Cloning into '/var/lib/gitea/data/tmp/local-repo/merge-192609140.git'... done. Downloading File.bin (11 KB) Error downloading object: File.bin (08afa7b): Smudge error: Error downloading File.bin (08afa7bbdbb8224c12d26dad8bb3641a8c0eacd032c47bba0f91275e0d892fef): batch request: missing protocol: "/home/git/gitea-repositories/liquid/test.git/info/lfs"

Errors logged to /var/lib/gitea/data/tmp/local-repo/merge-192609140.git/.git/lfs/objects/logs/20180717T230735.214492563.log Use git lfs logs last to view the log. error: external filter 'git-lfs filter-process' failed fatal: File.bin: smudge filter lfs failed warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry the checkout with 'git checkout -f HEAD'

Seems like something related to lfs.url on server side, but what and how to fix?

lunny commented 6 years ago

Have you installed git-lfs on your sever?

severgun commented 6 years ago

Yes. Also, broken once LFS pull request broke all webui merges for that repo.

severgun commented 6 years ago

After I add lfs.url = http://94.000.000.000:3000/Liquid/Test.git/info/lfs in .lfsconfig and push it to master I got new error. From batch request: missing protocol: to Merge: git push: Invalid remote name "/home/git/gitea-repositories/liquid/Test.git"

gerroon commented 6 years ago

@severgun

Did you solve this issue? I am facing it as well.

severgun commented 6 years ago

Nope. Tagged as a bug. Waiting for a fix. Seems like gitea now in pre-alpha unstable test because there is dozens of new issues per day.

gerroon commented 6 years ago

@severgun thanks

chowey commented 6 years ago

I'm getting the following error log when merging a pull request with LFS:

2018/09/12 11:24:52 [...routers/repo/pull.go:533 MergePullRequest()] [E] Merge: git push: Invalid remote name "C:\gitea\data\repos\my\repo.git" error: failed to push some refs to 'C:\gitea\data\repos\my\repo.git'

I'm using Gitea on Windows.

Haixing-Hu commented 5 years ago

same problem. any solution?

ghost commented 5 years ago

same problem. OS:centos7

$ git lfs env
git-lfs/2.6.0 (GitHub; linux amd64; go 1.11.1; git 93702d75)
git version 2.16.5

LocalWorkingDir=
LocalGitDir=
LocalGitStorageDir=
LocalMediaDir=lfs/objects
LocalReferenceDirs=
TempDir=lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

gitea.log

2018/11/27 11:19:39 [I] LFS server enabled
2018/11/27 11:20:22 [...routers/repo/pull.go:579 MergePullRequest()] [E] Merge: git merge --no-ff --no-commit [/var/apps/gitea/data/tmp/local-repo/merge-249727395.git]: exec(11:PullRequest.Merge (git merge --no-ff --no-commit): /var/apps/gitea/data/tmp/local-repo/merge-249727395.git) failed: exit status 128(<nil>) stdout:  stderr:
hint: The remote resolves to a file:// URL, which can only work with a
hint: standalone transfer agent.  See section "Using a Custom Transfer Type
hint: without the API server" in custom-transfers.md for details.
Downloading a.zip (34 B)

hint: The remote resolves to a file:// URL, which can only work with a
hint: standalone transfer agent.  See section "Using a Custom Transfer Type
hint: without the API server" in custom-transfers.md for details.
Error downloading object: a.zip (0a12b35): Smudge error: Error downloading a.zip (0a12b35d75289e158ab0d06fe388303c7275c458d51261dc44f747be6dfedc23): batch request: missing protocol: "file:///var/apps/gitea/repositories/developer/lfs-test.git/info/lfs"

Errors logged to /var/apps/gitea/data/tmp/local-repo/merge-249727395.git/.git/lfs/logs/20181127T112022.279642685.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: a.zip: smudge filter lfs failed
 -
hint: The remote resolves to a file:// URL, which can only work with a
hint: standalone transfer agent.  See section "Using a Custom Transfer Type
hint: without the API server" in custom-transfers.md for details.
Downloading a.zip (34 B)

hint: The remote resolves to a file:// URL, which can only work with a
hint: standalone transfer agent.  See section "Using a Custom Transfer Type
hint: without the API server" in custom-transfers.md for details.
Error downloading object: a.zip (0a12b35): Smudge error: Error downloading a.zip (0a12b35d75289e158ab0d06fe388303c7275c458d51261dc44f747be6dfedc23): batch request: missing protocol: "file:///var/apps/gitea/repositories/developer/lfs-test.git/info/lfs"

Errors logged to /var/apps/gitea/data/tmp/local-repo/merge-249727395.git/.git/lfs/logs/20181127T112022.279642685.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: a.zip: smudge filter lfs failed
stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

linghu86 commented 5 years ago

same problem

Begounet commented 5 years ago

same problem

zeripath commented 5 years ago

Which version of Gitea are you using?

Begounet commented 5 years ago

Gitea Version: 1.7.1 Edit: Do you think the issue has been fixed since? Edit2: We just saw that the Gitea version goes from 1.7.1 to 1.8.1 in 5 months! You are so efficient! Our admin has been told about it and will update the version tonight. I will be able to tell you about it tomorrow. Edit3: Got an emergency, we had to delay the update. We will go back to you when it will be done :)

Begounet commented 5 years ago

Ok, so we updated to 1.8.1 but the issue is still present. Edit: the issue occured when I tried a "merge with rebase" but not with a "standard merge". Still investigating on reproducing the issue. Edit: I made a new test by just adding one single 4x4 image (so it is filtered by Git LFS) and the merge didn't work.

zeripath commented 5 years ago

OK so I bet this is same issue as #4760 and #732

zeripath commented 5 years ago

Now I am willing to bet that you have git-lfs version greater than 2.3.4 installed on your server and gitea user has a .gitconfig file containing:

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true

If you remove that that should act as a workaround.

OK so that's not enough. Do your repos have .lfsconfig or is there an additional lfsconfig you're running?

In particular I'm highly suspicious that the there is lfs.storage set.

zeripath commented 5 years ago

I bet if you remove that - the problem will disappear.

zeripath commented 5 years ago

Which is obviously not a complete solution but if it works then I know that I am at least on the right track.

zeripath commented 5 years ago

OK, I think I've looked at this a bit more I think this is due to us previously cloning without setting --shared and without adding an alternative object dir reference for the pulled branch. What doesn't make sense though is that @Begounet has reported this on 1.8.1 - which also shouldn't be cloning without shared repository settings...

@Begounet I wonder can you reproduce this on master? What is your version of git lfs? Mine is: 

:; git lfs version
git-lfs/2.3.4 (GitHub; linux amd64; go 1.8.3)
yasuokav commented 5 years ago

This issue is not related to the version of git-lfs. Gitea does not store the LFS objects in repositories/xxx.git/lfs/objects, but Git transfers the files from the local directory instead of Gitea's LFS server.

local-repo/merge-xxxxx.git/.git:

# git lfs env
Endpoint=/path/repositories/test1.git/info/lfs (auth=none)
Endpoint (head_repo)=/path/repositories/test1.git/info/lfs (auth=none)

Also, I don't think it's a good idea to download the LFS file to the local temp folder and merge the PR.

zeripath commented 5 years ago

@yasuokav it might actually be related to LFS version. For example prior to 11 months ago git LFS would definitely fail on multiple alternative repositories. I don't get this problem on 2.3.4 and I know git-lfs changed their handling of local clones at some point - likely around 2.6. So if you're experiencing this bug would you please tell me what your server's version of git-lfs is? I suspect it is highly pertinent.

Now, I agree that I'm not certain that grabbing the LFS objects in order to do a merge is a good idea. I will look at stopping this. If you are experiencing this bug there are a couple of things I would like to know:

So that's 3 questions but the git-lfs version question is pertinent. So please tell me.

yasuokav commented 5 years ago

@zeripath

  1. git-lfs/2.4.0
  2. It does help
  3. No
zeripath commented 5 years ago

Cool. I can check if I can reproduce this with 2.4 - (looking at the release information for 2.4 they rewrote their file API at that point so that's highly suspicious.)

Ok.

In terms of quick fix for this I would guess that the most sensible thing is for Gitea to have its own .gitconfig in which LFS is not installed. Either that or determine a way to commit without running filters.

The correct answer is to change to an index based merging style - then filters etc don't matter as we determine what goes in to the index. You shouldn't be experiencing this bug on repo edit, or upload is because I rewrote these sections to do their work directly with the index.

zeripath commented 5 years ago

But this at least finally gives those of you experiencing this bug a workaround until I get a fix in.

zeripath commented 5 years ago

Sorry this has taken so long to figure out.

zeripath commented 5 years ago

OK, I've finally had a chance to download git-lfs 2.4.0 and git-lfs 2.7.2 and I still can't reproduce this.

If you can work out a minimal testcase that breaks for you I can try to replicate and figure out exactly what's causing the breakage, however what I might do is add a note to the troubleshooting page.

zeripath commented 5 years ago

I also think we should mark this as a duplicate of #732