Flake derivation only builds when git source is dirty #8442

Open x10an14 opened 1 year ago

Describe the bug

CAVEAT EMPTOR

THIS IS NOT A "OH NOES; SUBMODULES DOESN'T WORK!!!!" ISSUE Read first two items on below list in their entirety...

The issue at hand w/ git dirty

So, submodules eh? A big pain in nix's proverbial behind for now. Many smart people and issues working on it though, so no worries.
- Any repo referencing a submodule as part of "self", just finds the submodule(s) missing in the resulting /nix/store/XYZ-source derivation. Fine.
What if I just fetch the submodule as a new derivation? Say fetchFromGitHub for instance.
- Then just cp -r over the submodule's files into the src of the "main" derivation at buildPhase?
That works! Buuuut only when git-repo is "dirty"...

Steps To Reproduce

git clone https://git.sr.ht/~x10an14/blog
nix build <-- See me fail!
Add/edit a comment to any line in the flake.nix file
- Or Readme.md, or anything else that has no impact on the zola build (zola static site generator) command for that matter.
See error from step 2 disappear, no matter what you do!
Error from step 2 will re-appear the instant you commit any inconsequential change from step 3.

Expected behavior

Either;

this should never work, even when repo is dirty (because of the manual copy/overwrite during buildphase of the git repo's empty submodule folder)
this should always work, since it works when git repo is dirty

~~nix-env --version output~~ nix-info output ('cause I'm on a flake - joke intended)!

 - system: `"x86_64-linux"`
 - host os: `Linux 6.2.12, NixOS, 23.05 (Stoat), 23.05.20230421.2362848`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `"nixos"`
 - channels(x10an14): `""`
 - nixpkgs: `/nix/store/22z4n4mxs2vz3l3lg41dz3mgnq1d4wxs-source`

Additional context

Add any other context about the problem here.

Priorities

Add :+1: to issues you find important.

EDIT: Fixed git url.

Does the problem persist, if you doe the following:

clone with git clone --recursive to get the submodules populated in advance
also use nix build .?submodules=1 to allow nix to follow and include them?

This are basically the 2 main problems that I see:

You have 2 layers which both need to be aware of the submodule, you have told neither about it.

I was able to repair the submodules URL and get the submodules content and also built using submodules=1:

$ git submodule update --init --recursive
Cloning into '/home/nmelzer/tmp/blog/themes/shadharon'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (4/4), done.
remote: Total 6 (delta 4), reused 4 (delta 4), pack-reused 2
Unpacking objects: 100% (6/6), 938 bytes | 187.00 KiB/s, done.
From https://github.com/syedzayyan/shadharon
 * branch            828bb05ba53c7cd3e0a21b2513234c4d7cd4a54c -> FETCH_HEAD
Submodule path 'themes/shadharon': checked out '828bb05ba53c7cd3e0a21b2513234c4d7cd4a54c'
$ nix build .\?submodules=1
warning: Git tree '/home/nmelzer/tmp/blog' is dirty
$ echo $?
0

PS: Seems as if after a recursive clone the submodules=1 isn't even necessary here, due to the way how you copy.

The problem you have is, that in your original scenario git lost track of the folder holding the submodule, as it was empty, the submodule not loaded and therefore wasn't in the index anymore (git doesn't care for folders, only files).

With the recursive clone, you make sure that the folder will never be empty (for git).

Thanks to @NobbZ, @max-privatevoid, and @viperML over on the (unofficial) Nix/NixOs discord, we managed to track down the core of my issue;

Misunderstanding

Yes, as @NobbZ's three comments suggest, the two below suggestions will work:

nix build '.?submodules=1
adding an explicit git submodule add/clone (of the submodule repo) during the buildPhase

I argue that these suggestions did not adhere to the CAVEAT EMPTOR/item #2 in top list of OP.

Root cause

What I didn't get, but @max-privatevoid helped me understand (but @NobbZ also referenced) is this:

nix seems to invoke git in 2 slightly different ways, one of which results in the directory being copied into the source, and in the other case it gets stripped if the working tree is clean, nix will use git archive to copy the source into the store. if not, it will use git ls-files to get a list of files tracked by git, and copy the working directory to the store by itself with a filter based on that command's output

Whether this is a bug or a feature, I leave for someone else to decide. I do believe though, that this is worthy of having a "oops, buyer beware" issue/note of somewhere (like this issue can turn into now).

Final solution for today

Finally, @viperML helped finding a perfectly acceptable fix for the flake.nix file, until proper support for submodules land:

- cp -r ${zolaTheme} themes/shadharon
+ cp -rT ${zolaTheme} themes/shadharon

(For some reason, adding mkdir -p themes/shadharon did not suffice for a fix...)

I leave this issue for wiser people than me to close/leave open.

I just bumped into this when building a package with nix build and src = ./.. When source is dirty for any reason, and you are not including git submodules (with f.x. .?submodules=1), it will create a folder structure as follows:

default> configuring
default> .
default> |-- extern
default> |-- flake.lock
default> |-- flake.nix

However, once this is committed, and therefore your repo is longer dirty, it will now change its behaviour to include an empty folder for the not included git submodule.

default> configuring
default> .
default> |-- extern
default> |   `-- mtest
default> |-- flake.lock
default> |-- flake.nix

For now im doing a preConfigure step which includes a line like [ -d "extern/mtest" ] && rm -r extern/mtest to deal with this strangeness between dirty vs non-dirty repos.

NixOS / nix