git fast-import fails with uppercase/lowercase file name changes in commit

viric commented 8 years ago

[x] I was not able to find an open or closed issue matching what I'm seeing
Setup
Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options
git version 2.10.0.windows.1
sizeof-long: 4
machine: x86_64

The same happens in git-for-cygwin.

Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver
Version 10.0.14393

What options did you set as part of the installation? Or did you choose the defaults?

I used the defaults but "CRLFCommitAsIs"

Any other interesting things about your environment that might be related to the issue you're seeing?

No.

Details

Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

GitForWindows Bash and also using git from cygwin, from its mintty bash terminals.

What commands did you run to trigger this issue? If you can provide a Minimal, Complete, and Verifiable example this will help us understand the issue.

I took a fossil (fossil-scm.org) vcs export in "git fast-import" format, and then I gave it to a just initialised git repository with "git fast-import".

What did you expect to occur after running these commands?

The resulting git repository should have the same commits as if "git fast-import" was run on linux, but in some commits we have lost files. The files were properly referred in the fast-import format. Using "git fast-import" in linux, from the same source, results in all files in all commits correct.

What actually happened instead?

There were commits in the "fast-import" source where a file "ply.h" was removed and a "Ply.h" was added. In the linux "git fast-import" it was well handled, but in GitForWindows or GitForCygwin the commit resulted without "ply.h" or "Ply.h", so the new Ply.h file was missing. And it was also missing from all later commits, of course. Git was completely silent about this lost file. The very same happend with another file that had the same trouble: lowercase removed, added with some uppercase, in the same commit.

So, my workaround was to run "git fast-import" on linux, but I guess it is worth fixing, because "git fast-import" in windows gives the false feeling of properly importing the repository.

If the problem was occurring with a specific repository, can you provide the URL to that repository to help us with testing?

It happened with a non-public repository. I cannot provide it.

dscho commented 7 years ago

This is most likely not specific to Windows (so you may want to report it to the Git mailing list).

As to the non-public repo: providing a minimal, public repo to make it faster to reproduce your issue is in your pest interest, as it increases the chances of somebody else jumping in to help you.

PhilipOakley commented 7 years ago

@dscho, My reading of the description would suggest that it is a type of Windows problem where the fstat (on windows) can't tell that "a file "ply.h" was removed and a "Ply.h" was added" as they are identical as far as Windows is concerned, but distinct on Linux.

How to detect and handle such and issue still may require upstream changes so that the filename change detection is noted at a textual level before the old and new file names are compared case sensitively/insensitively ...

@viric, If the repo is small enough maybe the --anonymize capability could be used. See git fast-export

kgybels commented 7 years ago

I could reproduce this problem:

KGybels@PC-KIM MINGW64 /c/git/bug
$ git --version --build-options
git version 2.14.1.windows.1
built from commit: 82d9b3f3b2407b52251620597d4b14933685459d
sizeof-long: 4
machine: x86_64

KGybels@PC-KIM MINGW64 /c/git/bug
$ ls
bug.sh*

KGybels@PC-KIM MINGW64 /c/git/bug
$ cat ./bug.sh
#!/bin/bash

git init export
cd export
touch foobar.txt
git add foobar.txt
git commit -m v1
git mv foobar.txt FOOBAR.txt
git commit -m v2
cd ..
git init import
(cd export && git fast-export --all) | (cd import && git fast-import)
if [[ $(cd export && git log -1 --format="%H") == $(cd import && git log -1 --format="%H") ]]; then
        echo "Import OK."
else
        echo "Import doesn't match."
fi

KGybels@PC-KIM MINGW64 /c/git/bug
$ ./bug.sh
Initialized empty Git repository in C:/git/bug/export/.git/
[master (root-commit) 41b6b47] v1
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 foobar.txt
[master 0142f51] v2
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename foobar.txt => FOOBAR.txt (100%)
Initialized empty Git repository in C:/git/bug/import/.git/
Unpacking objects: 100% (5/5), done.
C:\Program Files\Git\mingw64\libexec\git-core\git-fast-import.exe statistics:
---------------------------------------------------------------------
Alloc'd objects:       5000
Total objects:            5 (         0 duplicates                  )
      blobs  :            1 (         0 duplicates          0 deltas of          0 attempts)
      trees  :            2 (         0 duplicates          0 deltas of          1 attempts)
      commits:            2 (         0 duplicates          0 deltas of          0 attempts)
      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)
Total branches:           1 (         1 loads     )
      marks:           1024 (         3 unique    )
      atoms:              1
Memory total:          2344 KiB
       pools:          2110 KiB
     objects:           234 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =      65536
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 35184372088832
pack_report: pack_used_ctr            =          1
pack_report: pack_mmap_calls          =          1
pack_report: pack_open_windows        =          0 /          1
pack_report: pack_mapped              =          0 /        373
---------------------------------------------------------------------

Import doesn't match.

The export itself is fine, I tried importing an export created with Git-for-Windows on a linux VM with git 2.7.4, this worked. Running the above script on the same VM prints Import OK.

kgybels commented 7 years ago

Setting git config core.ignorecase false fixes the problem.

Updated script to set ignorecase to false before the import:

#!/bin/bash

git init export
cd export
touch foobar.txt
git add foobar.txt
git commit -m v1
git mv foobar.txt FOOBAR.txt
git commit -m v2
cd ..
git init import
(cd import && git config core.ignorecase false)
(cd export && git fast-export --all) | (cd import && git fast-import)
if [[ $(cd export && git log -1 --format="%H") == $(cd import && git log -1 --format="%H") ]]; then
    echo "Import OK."
else
    echo "Import doesn't match."
fi

Now it prints Import Ok.

I wouldn't expect this setting to change the behavior of fast-import.

PhilipOakley commented 7 years ago

@kgybels, maybe the git config man page needs a tweak (for GfW) so that users know the default here is different?

Changing upstream to mention that some case insensitive systems (e.g. Git for Windows) may set it true by default (IIRC) may help make the 'probe' aspect more understandable.

kgybels commented 7 years ago

@PhilipOakley Maybe the documentation could be improved, however, it seems irrelavant to this bug, I would expect git fast-import to correctly import the changes given to it, independent of the core.ignorecase setting.

It's worth mentioning that the problem can be reproduced on linux, using a case-sensitive filesystem, by setting git config core.ignorecase true. So @dscho was right, this is not a Git-for-Windows specific bug.

PhilipOakley commented 7 years ago

@kgybels, While I'd generally agree, I take a slightly different line of reasoning to get to the various intermediate points and what to do if one stops at those points.

The GfW issues is that Windows is an core.ignorecase true system, and that Git / GfW will detect that difference, and as currently implemented the fast-import then does not produce an identical import.

Now we could stop at that point and say that it is all GfW/Windows fault and require a magic fix in GfW (poorest option)

We could say that the GfW/Windows issue (of core.ignorecase true) is that it's just not obvious and hence the documentation should have a suitable note "at the right place", but that is frought with difficulties regarding the initial RTFM step, and deciding, which part of the manual would actually be read - should it be special to the git help fast-import page? or is it part of the extensive git config man page, or even that the core.ignorecase section (if ever found by the fast-import user) is somewhat unfathomable regarding the Worlds Most Popular File System (TM?).

Or that the git fast-import code should, like you say, be updated to match that of the exporting repo so that filename case changes are handled as per the originating repo [Is that actually true? - does the export care about the FileSystem case, the names (from git) are in the object store trees - maybe some VCS has issues at export, dunno].

So I do see it as related to the Windows environment, but at the same time I see it also as a problem in getting changes included upstream, and often the documentation changes can be even harder than code changes - it's not only a compatibility issue (which hits code) but also a subjective language issue regarding purpose of the man pages - some say that are for reference by those that already know, while others think they should help those that doesn't yet know (educational).

Perhaps @viric could indicate what would have helped avoid this #908 pitfall (i.e. if the docs had been better somewhere, where did they look? Or does the Git code just need to work.

tboegi commented 7 years ago

For the interested reader:

https://public-inbox.org/git/1391346784-11891-1-git-send-email-reubenhwk@gmail.com/

PhilipOakley commented 7 years ago

@tboegi wrote: For the interested reader:

https://public-inbox.org/git/1391346784-11891-1-git-send-email-reubenhwk@gmail.com/

That's a useful link. I think the rule should be at least that Git can round trip any Git repo between different ignorecase File systems and still get back the identical same repo. Yes? It does sound like an upstream problem for that case.

For exporters from other VCS systems there may be extra concerns and documentation issues as noted in one of Peff's replies to that thread (also at https://marc.info/?l=git&m=139134692627739&w=2).

tboegi commented 7 years ago

To come back to the original problem: git -c core.ignorecase=false fast-import should work.

And all this has nothing to do with GfW, as other case-ignoring file systems (e.g. Mac OS) have the same problem.

kgybels commented 7 years ago

@tboegi Thanks for the link, that was an interesting read.

I found more here: http://git.661346.n2.nabble.com/fast-import-should-not-care-about-core-ignorecase-td7622690.html

@PhilipOakley Some information about this case folding could be added to the git fast-import documentation, together with the example use git -c core.ignorecase=false fast-import @tboegi gave to disable it.

PhilipOakley commented 7 years ago

@kybels, I'd agree that the documentation could be improved for this case, mentioning at least Windows and Mac OS as affected systems.

I also have a suspicion that if the repo has had multiple 'core.ignorecase=true/false` transitions that the fast export may not actually contain sufficient information to allow a successful import with either setting. I'm just not sure.

Possibilities include that user A works using true and user B uses false, both have renames on the feature branches they worked on prior to a fetch / merge upstream (so each of the merged branches used a different setting, hiding their renames). The whole repo is then exported, and attempts are made to re-import the repo but neither setting is appropriate (I suspect) to both! It's a rather similar problem (but thankfully far more infrequent) to the EOL setting problem. Improving the documentation should help

git-for-windows / git

git fast-import fails with uppercase/lowercase file name changes in commit #908

Setup

Details