nix flakes: add support for git submodules

[DanilaFe]

I recently learned about Nix Flakes, and I tried out making my blog a perfectly reproducible build. The trouble is, my blog uses a Git submodule to reference some code that I then present in my blog posts. This is fine and all, but I run into an issue - although I can fetch my blog source using Flakes' input attribute, it doesn't fetch submodules! This means that the code that I reference in my build process is simply not there.

This seems to be almost possible; from a brief glance around Nix's source code, I discovered that a flake input gets turned into a "regular" input, which can point to a git repo and have a submodules attribute that, I think, would do what I want. It feels as though it was intended that this would work, but it doesn't.

It doesn't work because, for a reason unknown to me, a flake input's attributes must all be strings (except for flake). Here's the code that rejects non-string arguments:

https://github.com/NixOS/nix/blob/724b7f4fb660212a97ba6482208c299158720c5b/src/libexpr/flake/flake.cc#L123-L127

If non-string arguments were not rejected, a submodules field could be forwarded to the Git input scheme, which I suspect would make this work. Is there a reason that Flakes rejects all non-string "third party" attributes? It puts them into an attribute set anyway. Right now, I'm falling into a rather amusing loop:

If submodules is a string, it's rejected by the Git input scheme. But if it's a bool, it's rejected by the Flake code.

[knedlsepp]

I think this is a duplicate of https://github.com/NixOS/nix/issues/4357

[blaggacao]

Apparently closed by #4435

[knedlsepp]

I think there is still one piece missing: the flake itself is not being fetched using submodules.

[DanilaFe]

Yes, perhaps it's worth closing this issue, and opening the original one (since this is a duplicate)?

Also, I am not sure how to go about requiring a flake itself to pull its own submodules. If a flake only has inputs and outputs, and an external definition determines the method for downloading it (a flake can be a local folder, too, or a tarball, can't it?), then where's the compromise?

A flake is assumed to be a git repo, so it would make sense for a universal "fetch submodules" option, perhaps as an output attribute. This would be orthogonal to the type = "git"; submodules = "true" attributes inside an input, since it wouldn't work for non-flake inputs.

[blaggacao]

What would be a good & concise way to present the design decision abstract primitives still in question to @edolstra / the maintainers? (possibly on another issue for proper framing)

[L-as]

@knedlsepp

I think there is still one piece missing: the flake itself is not being fetched using submodules.

I made an ugly patch where submodules are always fetched: https://github.com/L-as/nix/commit/ce0a54782039ca21d2afe585b87922de957f0995

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?). My C++ skills are subpar though, and my patch may just make it fetch submodules for all git flake inputs (haven't tested), but it works for my case.

[ruler501]

@L-as yeah that's just turning it on for everything. If you look at the constructor for one of the classes for the nix develop command it has a default url of . making that instead be .?submodules=1 or whatever the correct format for that url is should cause it to only affect the flake in the repo the command is running in.

[L-as]

It has to work with all nix commands though, and I can't see the use in being forced to manually write ?submodules=1 every time.

[ncfavier]

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?).

Consider a public NixOS config repo with a private submodule. One may want to be able to build the config even without access to the private parts.

[L-as]

You could make it so that failure to clone the submodules is a non-fatal error.

[bqv]

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?).

Consider a public NixOS config repo with a private submodule. One may want to be able to build the config even without access to the private parts.

I'm not sure that's desirable behaviour. It's nondeterministic, then

[edolstra]

We should enable submodules by default and make sure they work for top-level flakes.

[ony]

Are there any work-arounds for top flake?

nix build  '.?submodules=1#nixosConfigurations.myhost.config.system.build.toplevel'

Doesn't work. Inside of /nix/store/*-source/ I see .gitmodules, but no corresponding folders.

[FlorianFranzen]

I currently just use fetchGit to re-clone the flake input with submodules like this:

flake_input_fixed = builtins.fetchGit {
      url = "<your-url>";
      inherit (flake_input) rev;
      ref = "<branch>"; # optional if not default
      submodules = true;
};

@ony: You might be able to do the same with self and then import any nix expressions that required the submodule from there.

EDIT: It should be noted that self.rev is only available when the repo is clean, so this approach is probably quite limiting when used with self. For now I just have a separate flake repo that I manually keep in sync, which works but is rather annoying.

[DieracDelta]

I'm having trouble fitting flakes to the use case of developing a git repo (e.g. self) that has multiple submodules.

My use case is as follows: I have a top level flake (and git repo) with a bunch of git submodules. I want to change these submodules and have those changes show up when I build outputs of the top level flake. Maybe I'm going about this wrong? Is there an easy way to get this sort of workflow?

I see the listing all submodule directories as url = path:$SUBMODULE_PATH as a possible work around, but this is impure. Perhaps something akin to url = path+git:$SUBMODULE_PATH that uses git to maintain purity.

Another solution could be a flag that specifies whether or not to fetch self submodules from git or to use the locally checked out versions. Though, this would require gathering the local submodule path information from somewhere...

[matu3ba]

The only usable way I see is to get all the submodule hashes and print them in nix-usable format with a nix-specific forward like nix git submodule init and nix git submodule status --recursive (printing non-recursively is a footgun).

Ideally one can then extract already existing pixpkgs or flakes to use instead of the very brittle to use hashes.

However I am not sure, what guarantees git makes about the output format as the status has no --porcelain. git submodules without guarantees how to machine-readable the package status sounds more like a hackish thing, so I am not entirely sure if nixOS should endorse that. I would favor a solution that descibes how to get away from git submodules until git gives better guarantees.

[ztlevi]

Is top-level flake supported?

I'm using a submodule inside my flake repo, in nixos, I can use nixos-install/rebuild to build it without any issue. But for the github workflow, it's using ubuntu and only has nix binary.

I'm build the system with this [command])https://github.com/ztlevi/dotty-nix/blob/main/.github/workflows/build.yml#L45). But it pops submodule error. The submodule here is call config.

See my build failure here

Since the issue is closed, @edolstra to add some visibility.

[DieracDelta]

Strawman question: have you tried a more recent version of nix @ztlevi ? It looks like the https://github.com/NixOS/nix/pull/4922 was merged 3 days ago. But the nix compiler version is from 2021-08-23 (https://github.com/NixOS/nix/commit/af94b54db3a2be100731a215cb5e95f306471731) which is several weeks ago. You might try bumping it to a more recent version of master.

Just to add in my 2 cents: I've built flake attributes that use git submodules on yesterday's nix master on ubuntu machines with nix installed and have had no issues. However, this has not been for nixos configurations.

[ztlevi]

@DieracDelta How am I able to test the latest nix build in the github workflow? ttps://github.com/ztlevi/dotty-nix/blob/main/.github/workflows/build.yml#L45

The nix release is too old

• https://github.com/NixOS/nix/releases
• https://github.com/numtide/nix-unstable-installer/releases/

[DieracDelta]

Don't have an "elegant" answer. My solution was to use old nix to build new nix (it is a flake after all and the cli is great for this). Then I would just call new nix.

EDIT ncfaviers solution is better.

[ncfavier]

RTFM

[ztlevi]

@ncfavier I'm using it here, https://github.com/ztlevi/dotty-nix/blob/main/.github/workflows/build.yml#L31

But how can I find the latest binary link. It's not yet released...

[DieracDelta]

Oh can you not just swap the date and hash? I thought that's what @ncfavier meant (but I was also half asleep)

Anyway as a workaround in the workflow just do: nix build github:NixOS/nix -o nixMaster && ./nixMaster/bin/nix $whatever_toplevel_target_you_wanted @ztlevi

[ncfavier]

The section I linked answers your question. Please read it.

[DieracDelta]

Ah yeah To install Nix from any commit, go to the corresponding installer_test action and click on "Run cachix/install-nix-action@XX" step and expand the first line.

[ztlevi]

All recent builds fail... I couldn't find such installer_test available within 3 days. https://github.com/NixOS/nix/actions

I'll just wait until they fix that and retry. Thanks for the help!

[DieracDelta]

https://github.com/NixOS/nix/issues/4423#issuecomment-917063439 this will work if you want a quick fix in the meantime. You can even specify a rev there if you want.

[ztlevi]

I tested and it works. Thanks!

[nrdxp]

I guess since https://github.com/NixOS/nix/pull/5284, this should be reopened until an alternative solution emerges.

[knedlsepp]

:clap: For @Kha working on https://github.com/NixOS/nix/pull/5399 which might imply that we could revert https://github.com/NixOS/nix/pull/5284.

[L-as]

I don't think we can revert #5284 . This change breaks backward compatibility. There are likely other solutions to this problem. It apparently also did not work well with git-crypt according to users of it.

[maydayv7]

Am affected with this issue, needs to be reopened until an alternative like #5399 is merged and #5284 is reverted, or #5312 is solved

[poelzi]

/reopen

This is a flake showstopper for many projects

[kaii-zen]

... siphoning submodules stakeholders to https://github.com/NixOS/nix/pull/5497

[lovesegfault]

Hit this today, it'd be great if this issue was reopened since I spent a while thinking this wasn't my actual problem and looking elsewhere.

cc. @edolstra

[tmplt]

I'm not sure if I'm hitting the same issue. I replaced a secrets/ directory with a submodule, and building with imports = [ ./secrets/secret-module.nix ]; yields

error: getting status of '/nix/store/f99g58wwx2j5akbnai3bdpb3vvwykj4z-source/secrets/secret-module.nix': No such file or directory

[DieracDelta]

@tmplt the current nix behavior is: "if you have a submodule, all the files within that submodule will not be copied to the store when building." So anything within the secrets/ directory will not be available at build time. I'm not aware of a good solution to this right now, sadly. An easy way to test this would be to move secret-module.nix out of the secrets folder, change your import path, and see if it builds.

Unrelated, if you're setting up secret management, sops-nix and agenix both do a good job of preserving secrets.

[tmplt]

Unrelated, if you're setting up secret management, sops-nix and agenix both do a good job of preserving secrets.

Neither seem to support home-manager at the moment, unfortunately.

[maydayv7]

Unrelated, but you can try homeage

[Atry]

It seems that nix profile upgrade does not support submodules as reported at https://github.com/NixOS/nix/issues/6788

[knedlsepp]

As #5284 happened this issue should be reopened.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-build-flake-cant-read-a-file-in-a-submodule/21932/3

[MangoIV]

What's the situation on this issue? For me, neither

inputs.<name>.submodules = true

nor

nix build .?submodules=true#bla

work on nix 2.11. There's no documentation on this, anywhere, either.

Edit: we apparently have to use an explicit type = git; in our inputs to allow for the submodules = bool flag.

[jakubgs]

How can I apply submodules=true to self without having to specify it in CLI every time?

[cxandru]

@MangoIV How exactly did you get it to work? How can one use it for the current project (self?)

[cxandru]

@jakubgs I don't even know how to get it to work in CLI. How do you do that?

[jakubgs]

Exactly as shown in https://github.com/NixOS/nix/issues/4423#issuecomment-1262247071.

[cxandru]

@jakubgs but how do we set type="git" to self? inputs.self.type = "git" didn't work for me. Also, does the .gitmodules file need to be committed or is it fine if it's only stashed?

[hraban]

In case anyone is interested in using submodules on flake inputs , not just the flake directory itself, you have to specifically use =1, not =true, and you have to use a "git" type input, notably not github:user/repo syntax because that's just a HTTP download of an archive from github's "download as .tar.gz" endpoint.

Example:

  inputs = {
    nyxt = {
      flake = false;
      url = "git+https://github.com/atlas-engineer/nyxt?submodules=1";
    };

C.f. https://github.com/NixOS/nix/blob/31ffd0c1fe1d5112746f3c8d608cbfb4d6290d1b/src/libfetchers/git.cc#L270-L271

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/get-nix-flake-to-include-git-submodule/30324/1