niv status, sources discovery and niv init templates

Got a lot of feedback over the past few months, and a lot more experience with niv. This in an outline of the sources handling changes that will happen soon; it's aimed at making basic nix projects even simpler, while giving people more flexibility re. the location of niv's files. Speak now or forever hold your peace!

discovery

sources.json: niv will go up in the directories (til filesystem boundary) until if finds either a sources.json or a nix/sources.json. When it does, it will try to parse it (should be an object and all values should be objects). If it doesn't parse, keep searching.

sources.nix: same as above, but with sources.nix or nix/sources.nix, and will hash it for a known sources.nix hash (or niv: no_update)

inlined sources.nix: niv will go up in the directories (til filesystem boundary) and for every file with a .nix extension in . and ./nix/ and will scan them (up to n characters) for the upload URL of sources.nix (i.e. https://raw.githubusercontent.com/nmattia/niv/<version>/nix/sources.nix) and the sha256 of a known sources.nix.

Generally the discovery should be done lazily (not performed if not needed), should be short-circuited with global flags (like --sources-file does already) and should not take more than n milliseconds.

niv status

This is a new command that displays:

the path to sources.json
the number of packages handled by niv
the niv version
the path to sources.nix, or the path of the nix file where it's inlined (i.e. fetchurl with remote source)
the version of sources.nix

niv init templates

niv init tries to discover sources.json and sources.nix. If it finds them (NOTE: both or either?) it considers niv was already initiated and only updates sources.nix (if necessary). If it does not find them, then it initializes directory.

There will be two templates to start with, light and heavy (not sure yet on the name). light is the default.

$ niv init (--light|--heavy)

light:

# default.nix
{ system ? builtins.currentSystem
, crossSystem ? null
, config ? {}
}:
let
  sources = import sourcesnix { sourcesFile = ./sources.json; inherit pkgs; }; # path to sources.json customizable

  sourcesnix = builtins.fetchurl {
    url = https://raw.githubusercontent.com/nmattia/niv/<version>/nix/sources.nix; # the release commit will be tagged with version
    sha256 = "<sha256>";
  };

  pkgs = import sources.nixpkgs {
    inherit system crossSystem config; # or whatever is best practices nowadays
    overlays = [ (self: super: { inherit sources; } ) ];
  };
in
pkgs.hello

# sources.json
{ "nixpkgs" : { ... } }

heavy:

nix/default.nix similar to default.nix in light, except that it returns pkgs (not pkgs.hello) and that it imports a local sources.nix that's written in nix/sources.nix. Might add a flag (--remote-sources) to disable this.

# default.nix
{ system ? builtins.currentSystem
, crossSystem ? null
, config ? {}
}:
let pkgs = import ./nix/default.nix { inherit system crossSystem config; }; in pkgs.hello

the sources.json go to nix/sources.json (instead of ./sources.json) and include niv as well (instead of just nixpkgs).

Love it, more opinionation is always good in this kind of tooling.

Some personal feelings:

discovery
- I don't care much about this. Having to use niv in the top-level project directory is not much of a cost to me, since it's a tool I use comparatively infrequently. The level of annoyance is comparable to "oh, I ran grep in a subdir so it won't get everything, better move up". Plus, it adds behavioural complexity (next points).
- Continuing when you find a correctly-named but malformed file seems more likely to mask errors than anything. What usecase does this help? Other sources.json files which aren't objects? So it'll break if the user turns it into an object later?
- Does sources.nix have to be by sources.json?
inlined sources.nix
- This sounds complex and fragile to me. If I used this, I would not look forward to debugging it when I accidentally broke it.
niv status seems good
niv init
- I would find it somewhat counterintuitive that niv init does anything other than initialize a project. I would prefer something like niv regenerate which regenerates sources.nix.
- "light" and "heavy" will work fine until you come up with a third way :p

I think a bunch of these issues center around the fact that niv has no configuration and so relies a lot on the (implicit) locations of its files. So here's a sketch of an alternate design:

Add a configuration file, .niv.yaml (or whatever format). This allows stating:
- The location of sources.json, otherwise it's $(dirname $NIV_CONFIG_FILE)/nix/sources.json.
- The location of sources.nix, including "don't expect one" (i.e. I inlined it), perhaps with an explicit version stated, otherwise it's $(dirname $NIV_CONFIG_FILE)/nix/sources.nix.
- You could put the "don't update" setting in here instead of in a comment in sources.nix too.
Discovery
- You could discover niv.yaml by upwards traversal: it's a single, unambiguously named file, so not too bad.
- No need to discover the other files: they're either in a fixed location relative to the config file, or explicitly stated.
Init
- The project is initialized iff we find .niv.yaml.

To me, the original proposal sounds very reasonable, but I do have some theoretical concerns. I realize that these might not be actual flaws in the design, but I'd still like to see them addressed.

builtins.fetchurl The fetchurl built-in still seems to be forbidden in restricted evaluation mode. This is not really an issue for me, but I think it would be a good idea to mention if/how this potentially could cause problems for other users. I am not very familiar with Nix Flakes yet, but investigating how builtins.fetchurl interacts with those might also be of interest
Generating a default.nix in the root directory I suspect many projects using Nix already have a default.nix file in their root directory. If this is the case, I think Niv should emit a small guide on how to manually edit a default.nix to use Niv sources.
Niv binary and sources.nix getting out of sync If I understand this correctly, Niv updates the sources.nix file/import based on the Git hash of some branch in the upstream Niv repository. This means the behavior of Niv essentially relies on two update streams, which might be confusing for the user. Maybe it would be a good idea to, at build time, include a hash of the current sources.nix file in the Niv binary and check whether this hash matches before updating the file/import.
When does the sources.nix file/import get updated? From your original proposal, it seems unclear to me when the sources.nix file/import is updated. Will niv init continue to serve as the facility to update sources.nix? In that case I agree with @michaelpj in preferring another command that only updates sources.nix.

Unlike @michaelpj, I do not think Niv would work better with a configuration file. The original proposal is very opinionated, but should already be flexible enough to support most workflows.

Thanks for giving the community an opportunity to respond to your plans!

I just want to chime in and say that @michaelpj concerns about discovery seems quite fair to me.

I am probably missing the point but I don't see why niv makes it so complex. Why not doing what every other tools would do; go and search a default place. For users with special requirements, make it possible to specify another default.

In any case, I would challenge any added complexity, would go with sane defaults and have good documentation for users that want to follow more advanced avenues.

I am actually using niv in some global overlays without any default.nix. I only do sources = import ./nix/sources.nix; so the whole business with niv init (--light|--heavy) goes a bit off the top of my head.

This would be nice, because sometimes I put files in nonstandard directories, and passing -s source.json to niv still fails, because 1) it's doesn't seem to be the right thing anyway 2) it can't find sources.nix even though it's right next to it.

nmattia / niv