search

rtyley / bfg-repo-cleaner

Removes large or troublesome blobs like git-filter-branch does, but faster. And written in Scala
https://rtyley.github.io/bfg-repo-cleaner/
GNU General Public License v3.0
10.86k stars 535 forks source link

BFG --convert-to-git-lfs does not convert all of the files I would like to convert in a big local repository. #349

Open ArdaS2012 opened 4 years ago

ArdaS2012 commented 4 years ago

The command is : java -jar /d/bfg-1.13.0.jar --convert-to-git-lfs "*.{png,dll,exe,ppt,pptx,doc,docx,docm,dot,dotx,dotm,xls,(... and many more)}" --no-blob-protection "/direction"

It changes some files but for examples some png files are not changed to lfs and I do not know why.

Also how can I modify the script (or the command?) so that the endings of the files (.png) will change also if its like .PNG, or .PnG etc.

I am new to git and the lfs system, so I would appreciate every little help. Thank you very much!

javabrett commented 4 years ago

My understanding is that BFG LFS support is experimental and the LFS team recommend other tools which you should explore.

ArdaS2012 commented 4 years ago

Thank you for the answer. I will look for other tools too, but I would be glad if someone could help me with this problem.

ymartin59 commented 4 years ago

I have found no where information that this feature is experimental. I faces the same issue - after fixing the issue with multiple patterns in git attributes file. On a really large repository (so difficult to investigate), it took me hours to consider I am facing a tool bug. So a basic test with only "*.zip" pattern makes it obvious it is buggy...

I am post-processing a Git repository produced from a Subversion migration, and I would expect to be able to discard files and folders and migrate some patterns in Git LFS in a single pass (and then fix commit notes with revisions details)

nbusseneau commented 3 years ago

To whoever may find this issue in the future: I can attest to this happening with v1.13.0. On a big repo with several years of history and 700K Git objects (as denoted by git rev-list --objects --all | wc -l), it missed about 400 LFS objects on a total of 2600.

A correct result was achieved with git lfs migrate, which seems to be the more modern way to migrate LFS objects anyway, contrary to what the GitLab documentation might imply.

bfg --convert-to-git-lfs "*.{png,gif}"                      # instead of that
git lfs migrate import --everything --include="*.png,*.gif" # use this

 

Also, since it is likely people ending up reading this are unfamiliar with .gitattributes: know that Git filters are case sensitive. Therefore, when tracking files with LFS, it is usually a good idea to use glob patterns in order to make sure you track files in a case insensitive manner:

git lfs track "*.png"         # will only track files ending in ".png"
git lfs track "*.[pP][nN][gG]" # will track both files ending ".png" and ".PNG", as well as any mix of the two

This mechanism is identical for git lfs migrate, e.g.:

git lfs migrate import --everything --include="*.[pP][nN][gG]"`