NAR hash mismatch cloning git repository with crlfs introduced in 2.20

[pwaller]

Describe the bug

Something changed in nix 2.20 which resulted in a new narHash for some git repositories/commits. I observe this if I use a nix 2.24 to update my lock file, and then have a different nix 2.18 machine try to use that lock file; for me the remote barfs when running nix flake archive --to file:///... with an error NAR hash mismatch in input. The reproducer below can be used to see that two different nix versions put different nar hashes into the lock file. Additionally, nix store --dump is different for the two paths even though git diff --no-index /nix/store/...A-source /nix/store/....B-source reports the two directories have no diff.

I wasn't trivially able to make a reproducer which produces this message since the error is happening on the remote and it doesn't reproduce locally for some reason, but I share a reproducer below for the divergent nar hashes:

Steps To Reproduce

1. Make a flake.nix:

   {
     inputs.llvm-project.url = "git+ssh://git@github.com/llvm/llvm-project?rev=c79d1fa540390f6e37e1ea326153559eeadd0de6";
     inputs.llvm-project.flake = false;
     outputs = { self, llvm-project }: { packages.x86_64-linux.default = llvm-project; };
   }

2. Run nix flake lock on nix 2.19: time nix run nixpkgs#nixVersions.nix_2_19 -- flake lock
3. Observe input narHash: sha256-WaoRxZ7Fjt4NaK2R+lCRIxo8//fiN79RUr+dov8+ad0= jq -r '.nodes."llvm-project".locked.narHash' flake.lock
4. Delete lockfile and recreate with nix 2.20: rm flake.lock; time nix run nixpkgs#nixVersions.nix_2_20 -- flake lock
5. Observe new input narHash: sha256-0YwZ6TiaALTxEwPAORLmM1G4+Hcq3jgh9XX0ke6EQ4g=.

Note also with nix eval --system x86_64-linux that there are two source directory paths: /nix/store/14ns2sfzd888cr0jghz8hrmh54s7hvjr-source and /nix/store/g8yc866ipcx1qwvn3ca6zbn8885dh5sl-source.

Comparing the nix-store --dump output on these two paths shows that the nars are 40 bytes different in size (it has increased in nix 2.20). bspatch generates a 1,741 byte patch between the two dumps, with the vbindiff output screenshot below.

The latter has an 0D 0A in it and the former has an 0A in it, so it looks like something has gone wrong with newline normalisation.

Expected behavior

I expected the nar hashes to remain the same. ~Especially so since the contents of the resulting source directories on disk appear on the surface to be the same.~ (edit: I wrote this before I had a plausible explanation; I guess the reason is that there are windows newlines in the repository and nix has changed how it operates with git repositories under the hood. I would still hope that changing the nix version doesn't change the nar hash though)

Priorities

Add :+1: to issues you find important.

[roberth]

In Nix 2.20 we have at least partially* solved an impurity where the fetched contents would depend on the local Git configuration with respect to newlines, smudge filters and possibly other impurities.

This was a serious bug in the older versions of Nix. Purity requires that the output content of fetching a source does not depend on local configuration, such as the local git configuration for newline conversions. Otherwise, users or machines with different git configurations will produce different derivations, making them unable to make effective use of a binary cache, not to speak of the possibility that the derivation outputs are meaningfully different.

Unfortunately we can not solve the git configuration impurity without a "Nix version impurity", but at least the latter will eventually solve itself.

*: I believe currently this impurity still exists when using a Git worktree (ie, local, normal, non-bare Git repo), as we are accessing the worktree directly, and Git has applied the described impurities to those files.

[pwaller]

Unfortunately we can not solve the git configuration impurity without a "Nix version impurity"

Is it possible for nix to at least match the default git configuration? I have not configured git in any way with respect to crlfs, so nix has diverged from default git behaviour here at least on the linux platform. That would also help with matching up against normal git worktrees for the majority of users which don't configure the crlf behaviour.

[maddiemort]

We're also seeing this problem, where if a Linux machine on Nix 2.24 updates a git+ssh flake input, the NAR hash will be incorrect when that flake is built from any other machine (mainly Darwin machines on either Nix 2.18 or 2.24). cc @sjbodzo