How can we pass `.?submodules=1` argument?

[bglgwyng]

My project includes git submodule so I need to set .?submodules=1 to let nix recognize the files in submodules.

use flake . .?submodules=1

I tried this and didn't work. Do we have any way to achieve it?

[bbenne10]

have you tried use flake .?submodules=1?

[bglgwyng]

Yes. But it didn't work.

[bbenne10]

What "didnt work"? Do you have a minimal reproduction case that I can look at? Looking over the code, I see no reason this should not work, so I will need to put some more time into finding the root cause.

[bbenne10]

This is a REALLY dumb U/DX problem...

submodules=1 should really be an argument to nix rather than a modifier of the flake specifier. Right now, we only proxy $@ to the nix print-dev-env invocation, which means that we lose the submodules=1 "modifier", since we discard the flake specifier as no longer relevant after we've generated the cache. I have to give some more though as to how to fix this...

(the thing I recommend in the short term is not using submodules if you need nix-direnv. Maybe additional flake inputs representing the submodules can overcome the problem immediately? This is inconvenient as it requires reworking your git repository and doesn't properly handle "non-nix-native" builds, but I don't have a better solution yet. I'm working on it)

[bglgwyng]

Thanks for your kind explanation!

[bbenne10]

So I just looked at this again and I was wrong in my last update. nix-direnv gets this right - the "$@" we pass to the underlying nix print-dev-env call does not have the flake reference shifted out of it, as I had feared. The bug is a bit deeper and not actually in nix-direnv, I think. A reproduction case is below and I'll speculate about what is happening below that.

The flake itself is nothing special. The submodule is just anything that can be copied to $out. Note that there are two packages and the one using the submodule is the default package. This is important later.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
  };

  outputs = { self, nixpkgs }: 
    let 
      pkgs = nixpkgs.legacyPackages.x86_64-darwin;
    in {
      packages.x86_64-darwin.hello = pkgs.hello;
      packages.x86_64-darwin.default = pkgs.stdenvNoCC.mkDerivation {
        src = ./html-website-templates;
        name = "test";
        version = "0.0";
        doBuild = false;
        installPhase = ''
          mkdir -p $out/bin
          cp -r . $out
          touch $out/bin/in_dev_shell
        '';
      };
      devShells.x86_64-darwin.default = pkgs.mkShellNoCC {
        packages = [ self.packages.x86_64-darwin.default ];
      };
  };
}

When .envrc contains use flake '.?submodules=1' (or use flake .?submodules=1 - the quotes changed nothing, but were a road I walked down a bit to see if the string was getting expanded somehow), it is broken and the submodules don't end up copied into the store. However, when using use flake .?submodules=1#default, everything works fine. I only tried this with lix 2.90, but I suspect this behavior is inherited from the upstream nix implementation.

I suspect that what is happening is that there's some sanitization happening that causes the "query parameter" (the submodules=1 part) to be lost if there's not an explicit package reference. Can you reproduce this? I think all you'd have to do would be to add #default to your use flake call.

The problem seems the same with devShells. You need an explicit default devShell for your platform (things like flake-parts, devEnv, or flake-utils make this easy enough to do, I just didn't pull them in for such a small example) and you need to explicitly specify both the ?submodules=1 and the devShell name.

I don't like this, but I don't think that there's much we can IMMEDIATELY do. If my understanding of the bug is correct, this needs to be fixed in nix itself before we can expose better experiences around it.

(Note that there is a small bug in nix-direnv that this exposes: We don't properly sanitize the ?submodules=1 out of the flake reference passed to us and so if you're doing .?submodules=1, we look for a flake.nix and flake.lock inside /.?submodules=1/ to add them to direnv's watches. This is an inconvenience, but you can just watch_file ./flake.nix ./flake.lock to work around it for now.)

[bglgwyng]

@bbenne10 Sorry for the late reply. https://github.com/bglgwyng/flake-submodule-sandbox I made a repo to reproduce the issue. Did I follow your suggestion correctly? I found that nix build ?.submodules=1 works well while nix-direnv failed to initialize the shell.

[bbenne10]

I thought the resolution was to add #default to your flake reference in .envrc, requiring you to change use flake .?submodules=1 to use flake .?submodules=1#default. I forked your sandbox to try and make a working example out of it by adding the #default, but had some hard times initializing the repo because of the invalid submodule reference you have pushed. (Not sure how it got that way - git complains pretty loudly when it's broken like that, as I found when I tried it.)

I then created my own reproduction case at https://github.com/bbenne10/nd_submodule_test_parent. Oddly, I cannot reproduce the behavior you're seeing - even without the #default! Testing again, the behavior I am seeing is noted below:

• .?submodules=1: Broken, no submodule copied into store
• .?submodules=1#default: Broken, no submodule copied into store
• '.?submodules=1': works (Note single quotes around the flake reference)
• '.?submodules=1#default': works (Note single quotes around the flake reference)

I think what this means is that it is the quotes that matter here. The inconsistency probably has to do with bash's field splitting happening in a way we don't expect. I cannot guarantee that, but I would be interested in hearing if my example repo above works for you.

[bglgwyng]

@bbenne10 I just tested it and observed the exact same result you listed. Quotes makes the difference.

[bbenne10]

Calling this done.