Better support for creating flakes with derivations

[tejing1]

Is your feature request related to a problem? Please describe.

I'd like to be able to use the nix registry to expose various aspects of my nixos system, such as the internal pkgs attribute, for easy use in the nix CLI. For example, I'd like to have nix shell pkgs#hello-unfree -c hello-unfree just work (as opposed to complaining about unfree packages) if my system configuration contains nixpkgs.config.allowUnfree = true;. Same for overlays and other such modifications.

I currently have the following: (in a nixos module that's declared where self from the inputs is still in scope)

nix.registry.pkgs.flake = pkgs.writeTextFile { name = "source"; destination = "/flake.nix"; text = ''
  {
    inputs.config.url = "path:${self.outPath}?narHash=${self.narHash}";
    inputs.nixpkgs.follows = "config/nixpkgs"; # Shouldn't be necessary, but is due to the current implementation of '.follows' in input flakes.
    outputs = { self, config, nixpkgs }: {
      legacyPackages."${config.nixpkgs.system}" = config.nixosConfigurations."${config.system.name}".pkgs;
    };
  }
'';};

Which almost works... but the CLI errors out because it can't create the flake.lock file, so I have to put --no-write-lock-file everywhere. I've been trying to come up with a good way to generate a flake.lock file that will shut it up, but all of them seem ugly and brittle, requiring too much intimate knowledge of flake.lock file format.

Describe the solution you'd like I'd like to see the nix CLI silently continue as if --no-write-lock-file had been specified on the command line (without the warning that option prints, or the 'Added' messages) if the following 2 conditions are both met: 1) The flake directory is in the nix store, precluding the creation of a flake.lock file. 2) The flake inputs are already fully specific, making the flake.lock file not actually convey any information.

Expansion of the above criteria may be desired, such as replacing 'is in the nix store' with 'is read-only', or still moving forward, but with a warning, if 1 is met but not 2.

Describe alternatives you've considered The only other plausible approach I see to support this would be if functionality were added to allow derivations to create correct flake.lock files in their output, but this would be difficult, particularly in a context where the inputs, though fully specific, were non-local, since that would require network access. Ultimately it would probably have to be built into the nix language itself, which I don't think anyone wants to expand unnecessarily.

[tejing1]

I finally decided to bite the bullet and push through this. I've come up with something that works, though it seems likely to break easily with future changes to nix. For reference on what the workaround looks like in comparison to the straightforward code, and in case it's useful to anyone:

{ config, lib, inputs, my, ... }:
with inputs;
with lib.strings;
with my.lib;
{
# build and register a flake to capture this config's pkgs attribute
nix.registry.pkgs.flake = mkFlake {config = self;}
  "{config,...}: {legacyPackages.${escapeNixIdentifier config.nixpkgs.system}=config.nixosConfigurations.${escapeNixIdentifier config.networking.hostName}.pkgs;}";
}

where mkFlake comes from my flake-local library my.lib, and is defined as

with builtins;
with pkgs;
with lib;
with lib.strings;

flakeInputs: outputsCode:
let
  inputsCode = "{${concatStrings (
    mapAttrsToList (n: v: "${escapeNixIdentifier n}.url=${escapeNixString "path:${v.sourceInfo.outPath}?narHash=${v.sourceInfo.narHash}"};") flakeInputs
  )}}";
  cleanNode = flake:
    let spec = {type="path";path=flake.sourceInfo.outPath;inherit (flake.sourceInfo) narHash;};
    in {inputs = mapAttrs (n: v: cleanNode v) flake.inputs;locked = spec;original = spec;};
  rootNode = {inputs = mapAttrs (n: cleanNode) flakeInputs;};
  flattenNode = prefix: node:
    let
      ids = mapAttrs (n: v: (flattenNode (prefix + "-" + n) v).name) node.inputs;
      nod = concatMap (x: x) (attrValues (mapAttrs (n: v: (flattenNode (prefix + "-" + n) v).value) node.inputs));
    in nameValuePair prefix ([ (nameValuePair prefix (node // { inputs = ids; })) ] ++ nod);
  lockJSON = toJSON {
    version = 7;
    root = "self";
    nodes = listToAttrs (flattenNode "self" rootNode).value;
  };
in
runCommand "source" {} ''
mkdir -p $out
cat <<"EOF" >$out/flake.nix
{inputs=${inputsCode};outputs=${outputsCode};}
EOF
cat <<"EOF" >$out/flake.lock
${lockJSON}
EOF
''

The flake.lock I'm producing here is still kinda odd, but it's good enough for nix to accept, at least in 2.4pre20210601_5985b8b.

A more correct implementation would need to import the flake.nixs along the way and parse their input sections to set the original section properly for each node and augment the locked section as well, rather than forcibly converting all the inputs to path types and setting original equal to locked, as I'm doing now.

Going to this much coding effort to shut up a tool about something that wasn't actually a problem in the first place is fairly frustrating, so my feature request stands, but I have what I wanted, in the immediate sense.

[thufschmitt]

Since all your flake inputs are local paths, you can actually run nix flake update inside a derivation to recreate a lockfile. The only gotcha is that since Nix won’t be able to write to /nix/store, you need to redirect the store to another location. But something like the below works (and produces a properly locked flake):

{
  outputs = { self, nixpkgs }: {

    defaultPackage.x86_64-linux =
      let pkgs = nixpkgs.legacyPackages.x86_64-linux; in
      pkgs.runCommandNoCC "myFlake" {
        buildInputs = [ pkgs.nix ];
      } ''
        mkdir -p $out
        cat <<EOF > $out/flake.nix
        {
          inputs.nixpkgs.url = "path:${pkgs.path}";
          outputs = { self, nixpkgs }: {
            packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;
          };
        }
        EOF

        export HOME=$PWD
        nix --experimental-features 'nix-command flakes' flake update $out --store $PWD
      '';
  };
}

[tejing1]

I didn't realize you could call nix in a derivation at all without recursive nix, but it makes sense now that you mention it.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[tejing1]

Still important to me.