Open mbrennan opened 7 years ago
leth
commented
7 years ago As a workaround, you can run the "lfs-test-server" locally for the conversion; then use git config --local --add lfs.url URL, git lfs fetch --all and git lfs push -all to migrate the LFS objects to a new server.
mbrennan
commented
7 years ago Thanks @leth,
I didn't realize the source repository had to be bare, so I did:
git clone --mirror repo.git repo.bare.git
Then I managed to get git-lfs-migrate to run properly with lfs-test-server using the default options. This starts the LFS server on http://localhost:8080. This is the command I invoked:
java -jar git-lfs-migrate.jar -s repo.bare.git -d repo.lfs.git --lfs http://localhost:8080 --glob-file binaries.txt
Everything seemed to work fine. Then to test out the new LFS-based repo I did:
git clone repo.lfs.git repo.lfs.not_bare
When that command tries to checkout the HEAD of master, it fails with this output:
Cloning into 'repo.lfs.not_bare'... done. Downloading subdir/Resources/112_LeftArrowLong_Blue.ico (29.22 KB) Error downloading object: subdir/Resources/112_LeftArrowLong_Blue.ico (d8e5f5b20eff4ab7e1023e33ca69cdbf9023288ef436769f37606e4d88ca9cee)
Errors logged to E:\repo.lfs.not_bare\.git\lfs\objects\logs\20170203T105759.3777981.log Use
git lfs logs lastto view the log. error: external filter git-lfs smudge -- %f failed 2 error: external filter git-lfs smudge -- %f failed fatal: subdir/Resources/112_LeftArrowLong_Blue.ico: smudge filter lfs failed warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry the checkout with 'git checkout -f HEAD'
That log file referenced shows this:
git-lfs/1.5.5 (GitHub; windows amd64; go 1.7.4; git c2dcd6f5) git version 2.10.0.windows.1
$ git-lfs.exe smudge -- subdir/Resources/112_LeftArrowLong_Blue.ico Error downloading object: subdir/Resources/112_LeftArrowLong_Blue.ico (d8e5f5b20eff4ab7e1023e33ca69cdbf9023288ef436769f37606e4d88ca9cee)
Smudge error: Error downloading d8e5f5b20eff4ab7e1023e33ca69cdbf9023288ef436769f37606e4d88ca9cee: http: Post e:///.%5Crepo.lfs.not_bare/info/lfs/objects/batch: unsupported protocol scheme "e": http: Post e:///.%5Crepo.lfs.not_bare/info/lfs/objects/batch: unsupported protocol scheme "e" github.com/git-lfs/git-lfs/errors.newWrappedError C:/Go/src/github.com/git-lfs/git-lfs/errors/types.go:166 github.com/git-lfs/git-lfs/errors.NewSmudgeError C:/Go/src/github.com/git-lfs/git-lfs/errors/types.go:252 github.com/git-lfs/git-lfs/lfs.PointerSmudge C:/Go/src/github.com/git-lfs/git-lfs/lfs/pointer_smudge.go:70 github.com/git-lfs/git-lfs/lfs.(Pointer).Smudge C:/Go/src/github.com/git-lfs/git-lfs/lfs/pointer.go:65 github.com/git-lfs/git-lfs/commands.smudge C:/Go/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:84 github.com/git-lfs/git-lfs/commands.smudgeCommand C:/Go/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:112 github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(Command).execute C:/Go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477 github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute C:/Go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551 github.com/git-lfs/git-lfs/commands.Run C:/Go/src/github.com/git-lfs/git-lfs/commands/run.go:66 main.main C:/Go/src/github.com/git-lfs/git-lfs/git-lfs.go:33 runtime.main C:/Go/src/runtime/proc.go:183 runtime.goexit C:/Go/src/runtime/asm_amd64.s:2086
It seems lfs-test-server isn't compatible with local directory-based repositories. Is this benign? I assumed it was and tried to do this:
cd repo.lfs.git
git lfs fetch --all
git config --local --add lfs.url SSH=git@somebitbucketserver.net:project/repo.git
git remote add origin ssh://git@somebitbucketserver.net:7999/project/repo.git
git lfs push origin --all
This errors with:
open subdir\Resources\112_LeftArrowLong_Blue.ico: The system cannot find the path specified.
Any other approaches anyone can think of?
leth
commented
7 years ago Git LFS urls are always HTTP; by default it will autodiscover the correct url. I'm only suggesting using the url override for the one-off transform, then pushing them across to their ideal server.
Here's roughly how I'm doing my migration, I'm doing all this on linux, so YMMV:
LFS_ADMINUSER=admin LFS_ADMINPASS=password ./lfs-test-server-linux-amd64/lfs-test-server > lfs-server.log &wget -O - --post-data "name=user&password=password" http://admin:password@localhost:8080/mgmt/add--lfs http://user:password@localhost:8080/git add remote my_remote ...git config --local --add lfs.url http://user:password@localhost:8080/git lfs fetch --all my_remotegit config --local --unset lfs.urlgit lfs push --all my_remoteHope that helps!
bozaro
commented
7 years ago If you don't set --lfs and --git option, git-lfs-migrate should create lfs files locally in lfs/object subdirectory inside destination repository. Theoretically you can upload it later by git lfs push --all command (see https://help.github.com/articles/duplicating-a-repository/).
Also you can provide user/password in --lfs key for working with custom authorization.
flipside
commented
7 years ago finally got it working using this to convert and push a local repository! more or less what i did was:
git clone --mirror ./path/to/repo.git
java -jar git-lfs-migrate.jar \ -s repo.git \ -d repo-converted.git \ "*.psd"
cd repo-converted.git
git fsck
git add remote origin https://github.com/user/repo-converted.git
git push origin master or git push --all origin
git lfs push --all origin
pwagland
commented
7 years ago Should this issue be closed? Or is there still some work that needs to happen?
svendhhh
commented
7 years ago Not sure... If it IS all working, I think the instructions/tutorial on how to use git-lfs-migrate should be updated. I'm getting the same issue, and am having to read through this conversation to try to work out how to actually do the conversion. Would be cool if the instructions were a bit more detailed, to help getting the migration working :)
svendhhh
commented
7 years ago I tried to follow the steps given by flipside as what was working:
time java -jar git-lfs-migrate/git-lfs-migrate.jar -s repo.git -d repo-converted.git "*.png" "*.jpg"This seems to run OK. As far as I can tell it's put the binaries into folder called "lfs" inside "lfs-converted-repo.git". Then I try:
cd repo-converted.git
git fsck
git add remote origin https://github.com/user/repo-converted.gitThis last command fails, because repo-converted.git is a bare repository, and you have to run git add remote in a work tree. So I try to first clone it:
git clone repo-converted.git repo-converted-clonedBut this now fails with errors trying to download the tracked files:
$ git clone repo-converted.git repo-converted-cloned
Cloning into 'repo-converted-cloned'...
done.
Downloading android/DeviceUtilityBridgeAndroid/res/drawable-hdpi/ic_launcher.png (9.18 KB)
Error downloading object: android/DeviceUtilityBridgeAndroid/res/drawable-hdpi/ic_launcher.png (28c8991): Smudge error: Error downloading android/DeviceUtilityBridgeAndroid/res/drawable-hdpi/ic_launcher.png (28c89918a8d235fcadf2e83bb14a6d9da9eab9e665857ad43b9f00e038f1f615): batch response: Post p:///repo-converted.git/info/lfs/objects/batch: unsupported protocol scheme "p"Apparently (https://github.com/git-lfs/git-lfs/issues/1207) this is because Git LFS only works when downloading from a server (not from file)... so I guess I'd need to be able to do the push, but I can't do the add of the remote... catch 22? :P
I'm not running the lfs test server, but it didn't seem from that latest post that that was necessary?
svendhhh
commented
7 years ago I now see that the error I was getting was due to the remote command being wrong. It should be:
git remote add origin https://github.com/user/repo-converted.gitOtherwise it was trying to add some (non-existent) files "origin" and "http...", which isn't allowed when not on a work tree.
I then did git push --all origin which worked, but then the git lfs push --all origin complains that:
$ git lfs push --all origin
Error scanning for Git LFS files in the "origin/master" ref
Error: ref 'origin/master' is ambiguousSo, I've gotten a bit further, but not quite there yet :)
Maybe this is the problem:
$ git branch -a
* master
origin/3dBranch
origin/master
remotes/origin/master
remotes/origin/origin/3dBranch
remotes/origin/origin/masterNot sure where the "origin/origin/.." branches came from, but might be the issue...
svendhhh
commented
7 years ago I think I've worked out a solution to the new problem. When I do the conversion I end up with branches master, origin/master and origin/3dBranch. I presume the master one has been added as a default when during the conversion, as it doesn't exist in the bare repo before. So I've renamed the branches and deleted the new one:
git branch -D master
git branch -m origin/master master
git branch -m origin/3dBranch 3dBranchAfter this the rest of flipside's solution works for me (with the "add" and "remote" swapped).
Hello,
I'm trying to use git-lfs-migrate to migrate an internal Subversion repository to Git for use with Atlassian BitBucket 4.11. This repository has a lot of binary files and I'd like to remove them from the git history using git-lfs-migrate. I'm using Windows 7, Java 1.8.0_91, Git LFS 1.5.5, git 2.10.0.windows.1.
In the commands and output below, I've changed the repository name to repo.git, the BitBucket project to 'project', the BitBucket server name to somebitbucketserver.net and the BitBucket IP address to 111.111.111.111 for illustration purposes.
When I issue any of the following commands:
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://somebitbucketserver.net/myproject/repo.git/info/lfs --no-check-certificate --glob-file binaries.txtjava -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://somebitbucketserver.net --no-check-certificate --glob-file binaries.txtjava -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://user:password@somebitbucketserver.net/myproject/repo.git/info/lfs --no-check-certificate --glob-file binaries.txtjava -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs https://user:password@somebitbucketserver.net --no-check-certificate --glob-file binaries.txtI always get this error:
I suspect it has something to do with authentication because in order to view the repository via HTTPS in a browser, I have to authenticated using our third-party SSO service. However, repos are fully accessible via SSH. Which led me to try the following in a git bash session after I added my SSH key to ssh-agent:
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g ssh://git@somebitbucketserver.net:7999/project/repo.git --glob-file binaries.txt --no-check-certificatejava -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g git@somebitbucketserver.net:project/repo.git --glob-file binaries.txt --no-check-certificatejava -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g ssh://git@somebitbucketserver.net:7999/project/repo.git --glob-file binaries.txtjava -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git -g git@somebitbucketserver.net:project/repo.git --glob-file binaries.txtThese commands always gives this error:
I also tried:
java -jar git-lfs-migrate.jar -s repo.git -d repo.lfs.git --lfs ssh://git@somebitbucketserver.net:7999/project/repo.git --glob-file binaries.txt --no-check-certificateWhich errors with:
Is there some way to get this working other than resorting to filter-branch or cluttering my repo with .gitattributes files from bfg --convert-to-git-lfs?
Thanks, Mike