search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.05k stars 1.47k forks source link

Flake input `git+file:./${submodule}` no longer works #9708

Open bryango opened 8 months ago

bryango commented 8 months ago

Describe the bug

Flake input of a git submodule in a relative subdirectory git+file:./${submodule}

Update: it seems that the regression happens since #9061. An example of this flake ref:

https://github.com/bryango/cheznix/blob/c445c8c82697de2bde53c65e9acacb9790fcd54e/flake.nix#L20-L23

The breakage would be acceptable to me if there is any alternative syntax that works, but currently I could not find anything that can replace git+file:./${submodule}.

Steps To Reproduce

$ nix --version
nix (Nix) 2.19.2

## suppose we have `inputs.nixpkgs.url = git+file:./nixpkgs;`
$ nix flake update nixpkgs
fatal: '/nixpkgs' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
warning: could not read HEAD ref from repo at 'file://file:./nixpkgs', using 'master'
fatal: '/nixpkgs' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error:
       … while updating the lock file of flake 'git+file:///home/$USER/.config/home-manager?ref=refs/heads/master&rev=c445c8c82697de2bde53c65e9acacb9790fcd54e'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'git+file://file:./nixpkgs'

       error: program 'git' failed with exit code 128

Expected behavior

I believe submodule sub-flake is quite common and git+file:./${submodule} should still work.

nix-env --version output

evaluating file '/derivation-internal.nix'
nix-env (Nix) 2.19.2
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/$USER/.config/nix/nix.conf:/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/0k52aibl2flync375nfw9a0kdjzvrzrc-nix-2.19.2/share

Additional context

N/A

Priorities

Add :+1: to issues you find important.

roberth commented 8 months ago

In the upcoming release 2.20, the git fetcher is overhauled, with improved submodule support, but the submodule flag for the fetcher is not enabled by default. @edolstra I vaguely recall that we'd enable submodules by default in fetchTree. Did we decide that? Otherwise, we could solve this flake problem by having a flake meta-attribute and "re-fetch" themselves with changed flags such as submodules = true;.

bryango commented 8 months ago

I tried the latest static binary from hydra which is built from 8e865f3aba526394ca333efe7258bd8db0050fbb, and it has a similar regression:

$ latestStatic=/nix/store/x1vkrxsxakk9wj7x0sydwifqa22jvqpx-nix-static-x86_64-unknown-linux-musl-2.20.0pre20240106_8e865f3
$ nix copy --from https://cache.nixos.org "$latestStatic"
$ nix shell "$latestStatic" ## below is from within the nix shell

$ nix --version
nix (Nix) 2.20.0pre20240106_8e865f3

$ nix flake update nixpkgs
fatal: '/nixpkgs' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
warning: could not read HEAD ref from repo at 'file://file:./nixpkgs', using 'master'
error:
       … while updating the lock file of flake 'git+file:///home/$USER/.config/home-manager?ref=refs/heads/master&rev=c445c8c82697de2bde53c65e9acacb9790fcd54e'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'git+file://file:./nixpkgs'

       error: fetching 'refs/heads/master:refs/heads/master' from 'file://file:./nixpkgs': failed to resolve path 'file://file:./nixpkgs': No such file or directory
bryango commented 8 months ago

I tried a bisect using hydra static builds, and it seems that the breaking change is made:

... which narrows down the commits to this range: https://github.com/NixOS/nix/compare/301623f3a3aa2b9bc6f116b7456f1528bd6bad47...d070d8b7460f412a657745698dba291c66792402. Looks like it is indeed related to fetchTree.

Update: I think it is precisely:

Update: I found that reverting these lines fixes the issue:

But I need to understand why. Then I can make a PR to try to fix it.

simonzkl commented 7 months ago

Before I open a new issue, I also noticed that this broke:

nix build '.?ref=my-branch#my-package'
fatal: ambiguous argument 'my-branch#m': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
error:
       … while fetching the input 'git+file:///home/foo/project?ref=my-branch%23m'

       error: program 'git' failed with exit code 128

Is this the same issue by any chance?

It's possible to work around this issue with:

nix build '.?ref=my-branch&#my-package'
bryango commented 7 months ago 
nix build '.?ref=my-branch#my-package'
fatal: ambiguous argument 'my-branch#m': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
error:
       … while fetching the input 'git+file:///home/foo/project?ref=my-branch%23m'

       error: program 'git' failed with exit code 128

Is this the same issue by any chance?

I think this is a different issue, though it is also related to flake URI parsing.

bryango commented 7 months ago

@roberth @fricklerhandwerk would you be interested in taking a look at #9897? :pleading_face: It is a super simple PR which fixes the issue for the 2.19 series (but not 2.20 yet, due to new complications from submodules & switching to libgit2).