Add an option to git lfs migrate export to affect all files

[gabrieldevillers]

Hello,

The documentation says "The export command will modify the .gitattributes to set/unset any filepath patterns as given by those flags." but that does not seems to be the case.

To Reproduce

file reproduce.sh:

#! /bin/bash
set -x
git init reproduce_git_lfs_export_bug
cd reproduce_git_lfs_export_bug
git lfs install
git lfs track "*.bin"
git add .gitattributes
git commit -m "Track .bin files using git-lfs"
echo -n $'\x01\x01\x01\x01\x01\x01\x01\x01' > file.bin
git add file.bin
git commit -m "Add a .bin file"
git lfs migrate export --everything --include="*"
cat .gitattributes

Actual result

*.bin filter=lfs diff=lfs merge=lfs -text
* !text !filter !merge !diff

Expected result

* !text !filter !merge !diff

To be precise, I expect git lfs migrate export to rewrite history such that all the .gitattributes files (there can be more than one) seems to have never been impacted by git lfs track.

System environment

OS Name: Microsoft Windows 10 Professionnal OS Version: 10.0.19043 N/A Build 19043 Git for Windows: git version 2.34.1.windows.1 64 bits git lfs version: git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2) Shell: MSYS (from Git for Windows)

git config -l:

diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
core.autocrlf=true
core.fscache=true
core.symlinks=false
credential.helper=manager-core
pull.rebase=false
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
core.autocrlf=false
rebase.missingcommitscheck=error
merge.tool=meld
mergetool.meld.path=C:\Users\devillers\AppData\Local\Programs\Meld\Meld.exe
winupdater.recentlyseenversion=2.32.0.windows.1
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
core.repositoryformatversion=0
core.filemode=false
core.bare=false
core.logallrefupdates=true
core.symlinks=false
core.ignorecase=true
lfs.repositoryformatversion=0

git lfs env:

git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
git version 2.34.1.windows.1

LocalWorkingDir=C:\dev\reproduce_git_lfs_export_bug
LocalGitDir=C:\dev\reproduce_git_lfs_export_bug\.git
LocalGitStorageDir=C:\dev\reproduce_git_lfs_export_bug\.git
LocalMediaDir=C:\dev\reproduce_git_lfs_export_bug\.git\lfs\objects
LocalReferenceDirs=
TempDir=C:\dev\reproduce_git_lfs_export_bug\.git\lfs\tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=C:\dev\reproduce_git_lfs_export_bug\.git\lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=C:/Program Files/Git/mingw64/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

[gabrieldevillers]

This is a problem for me because I am working on a script that mirrors Git branches to SVN, so I need to regularily git lfs migrate export so as not just pushing git lfs metadata files to SVN.

Given the current behavior I have to choose between manually fixing the .gitattributes files (perhaps using filter-branch), or simply ignoring the (somewhat) false positive information given by git status (files previously in git-lfs show up has modified, I need to git checkout -f to checkout something else, and I get Encountered N file(s) that should have been pointers, but weren't: when I checkout back to the branch where files have been removed from git-lfs).

[bk2204]

Hey,

This is behaving as expected. A .gitattributes directive starting with ! does unset the value. It is very difficult to determine whether a pattern subsumes another pattern in the general case, and there are additional specific cases where a pattern may subsume another in a particular repository in one commit but not another.

I think it might be possible for us to add an --all option or such to affect all patterns without --include in the future such that it subsumes all previous LFS patterns, though. That sounds like a nice improvement.

[gabrieldevillers]

Thanks for this answer. I am not sure I understand though:

Does having the following in the .gitattributes ignore the .bin extension ?

*.bin filter=lfs diff=lfs merge=lfs -text
* !text !filter !merge !diff

On the line related to the ".bin" I do not see the ! that you mention, moreover it is the same line as present in the file after git lfs track "*.bin" has been run.

Also I do not understand why this line is added (and why at the bottom of the file ?):

  * !text !filter !merge !diff

[bk2204]

Having those entries in .gitattributes results in a file named foo.bin not being an LFS file. The * entry matches all *.bin files and unsets the filter, merge, text, and diff attributes.

The reason it's added at the end of the file is that in the general case, the --include patterns don't need to cover all existing patterns. You could have patterns in .gitattributes for *.bin and *.dat, and if you run git lfs migrate export --everything --include="*.bin", then the *.dat patterns shouldn't be overridden. Placing the items at the end of the file causes that behavior to work correctly. Patterns that match an existing pattern exactly will be replaced instead of written into the end of the file.

[gabrieldevillers]

Ok, thats clearer thanks. But then do you know why I get a non-clean status on the files that have been moved out of git-lfs ?

[bk2204]

I'm not sure. Can you give me an example of such a path and the .gitattributes file in the revision that's checked out when you see it?

[gabrieldevillers]

Ok, thats clearer thanks. But then do you know why I get a non-clean status on the files that have been moved out of git-lfs ?

Actually I cannot reproduce, I must have been mistaken, sorry! Thanks for your help, and for working on git-lfs which is a very useful tool.

[bk2204]

Glad it's working for you now. If you see more problems, please let us know.

[bk2204]

Ah, wait. I had marked this as an enhancement to add a better flag here, so let me reopen.