Remove the flake registry or diminish its role

[roberth]

Describe the bug

The flake registry is global mutable state that's outside the view of beginners. NixOS is supposed to be the distro without such mutable magic.

The global flake registry is mostly append-only, so it usually won't cause problems. This is mostly about the system flake registry.

Steps To Reproduce

(edited:)

• See for example https://github.com/NixOS/nixpkgs/issues/204292

• Different users may get different results even if they run nix flake update at the same time on such a flake, due to differing nixpkgs entries in system or user registries; (@balsoft)

• If you have an unusual flake in your system/user registry, you might accidentally use it in the arguments of outputs, and now others can't nix flake update at all; (@balsoft)

• [Assuming we keep some registries, the system and user ones] breaks a nice setup with settings nixConfig.flake-registry in the flake, since system/user registry entries override the global registry entries set by flake-registry. (@balsoft)

Expected behavior

nix flake update always behaves the same.

(edited:)

Possible solutions

There's a matrix of things we can do here. On the first axis, to what degree do we remove it?

• Remove
• Deprecate
• Feature flag
  • allows flakes to stabilize sooner
• Put behind a disabled nix.conf option
  • discourages use
• Recommend not to use unless absolutely necessary, in docs
  • discourages use slightly
• Do not set any "default" registry entries in anywhere in NixOS
  • does not discourage use

On the second axis, which registries are bad?

• All registries
• Per-process registry, --override-flake
• Per-flake registry (has this been properly implemented?)
• Per-user registry
• Per-system registry
• Per-internet registry, aka NixOS/flake-registry. Airgapped networks may have their own.

Let's start with "all registries". This is perhaps too radical, but it could be argued that any mechanism is too complicated; that we should keep things simple and composable, by emphasizing local files and small commands with very few responsibilities. For example, niv does not implement an impurely overridable update mechanism, and I couldn't find such a feature request either.

The per-process case seems harmless because the caller is explicit about the intent. There's no "action at a distance". The per-internet one also seems harmless because it is managed with care and the outcome will be the same for everyone. Airgapped users have their own expectations. We should document how they should manage their registry.

A per-flake registry does involve "action at a distance", but will lead to the same result for everyone.

The per-user and per-system registries are bad, because they are both "action at a distance" and lead to different results for different people.

My current preference is to:

• Remove the system and user registries
• Put the registry feature behind a feature flag
• Make nix run /etc/nixpkgs#hello possible

nix-env --version output

Additional context

I'll use this section to summarize what I believe are the most relevant comments.

We must keep some variant of it otherwise nix shell nixpkgs#hello would not work.

I propose /etc/nixpkgs#hello as an alternative for those who want to pin the nixpkgs version they use on the command line.

Unprivileged users may use ~/nixpkgs if they want to manage their own command line pin.

Unpinned scripts may use e.g.

#!/usr/bin/env nix
#!nix run github:NixOS/nixpkgs/stable#python

# python code here

a common registry for many flakes within the same context (e.g. organization)

"nice setup" allows us to have a common registry for many flakes within the same context (e.g. organization). This is nice for two reasons:

• No need to type in inputs.bar.url = "github:foo/bar" for dozens of flakes = a lot less boilerplate
• The ability to withhold updates of certain inputs (think haskell.nix or nixpkgs) company-wide to prevent mass rebuilds, while keeping the rest up-to-date with daily nix flake update, and also the ability to update those inputs simultaneously over many flakes, resulting in potentially significant savings in build time and nix store space usage.

Valid reasons to allow and perhaps improve the nixConfig.flake-registry feature. It is still action at a distance though, and if this organization publishes their registry and projects as open source, unsuspecting users may be confused why their fork doesn't update.

But is it really mutable state?

Technically yes, because the registry information comes from mutable places. In practice, also yes, because the registry usually isn't pinned to e.g. a project. I'll stop here and not discuss the definition of this term any further. If you prefer another definition of mutable, you may read and discuss the issue equally well without that introductory sentence.

Priorities

Add :+1: to issues you find important.

[SuperSandro2000]

But is it really mutable state? I did a quick browse in the history and it seems that changes to individually entries are rare and mostly things get added.

• Remove the registry altogether

We must keep some variant of it otherwise nix shell nixpkgs#hello would not work. So disabling it by default is also not a good option.

Also the issue you linked is about that the flake template defaults to nixos-unstable which would not be solved by removing the flake registry but would make it more obvious in the flake.nix.

[roberth]

But is it really mutable state? I did a quick browse in the history and it seems that changes to individually entries are rare and mostly things get added.

We must keep some variant of it otherwise nix shell nixpkgs#hello would not work.

The truly global, mostly-append-only registry isn't nearly as bad as the system flake registry. I'll reflect that in the issue description.

To make a comparison, the global registry is like well-managed dns. google.com should just work, but search-engine shouldn't resolve to google.com today and bing.com tomorrow. (Where google.com is like nixos-unstable, for example)

We must keep some variant of it otherwise nix shell nixpkgs#hello would not work

Is it better than nix shell /etc/nixpkgs#hello? We could make that work.

but would make it more obvious in the flake.nix.

Exactly. Changing the ref is necessary. We can't make that go away, but we can make it obvious.

[edolstra]

There is something to be said for ignoring the local/system registry when creating lock files, but from a UX perspective we really cannot expect people to type nix run github:NixOS/nixpkgs#hello instead of nix run nixpkgs#hello.

[nrdxp]

There is also the issue of simple convenience. Chances are if you are using NixOS you are more than okay with using the same checkout of nixpkgs as your system was built with for other things. There are many advantages to locking this revision in the registry. After a nix search of this pin, for example, the nix eval cache will be popullated and future searches will be faster.

One can load a repl with this registry pin to quickly explore a nixpkgs checkout in a repl (I have an alias set for this even). To do a quick nix shell for ad hoc usage is much faster and cheaper if you use the same nixpkgs, since you won't have to redownload a different version of the toolchain (share libraries, etc) just to use a single package.

If the registry isn't the solution to this then okay, but we should provide this convenience somehow.

[roberth]

we really cannot expect people to type nix run github:NixOS/nixpkgs#hello instead of nix run nixpkgs#hello.

using the same checkout of nixpkgs as your system was built with for other things.

Something like nix run /etc/nixpkgs#hello seems like a good middle ground. It even has tab completion.

[nrdxp]

It is an old issue and still a bit fuzzy, and it is a massive wall of text, but this may tie it, at least tangentially, to what is discussed here: #4602

Basically what I proposed there is to have an at least partially complete checkout of nixpkgs locally. Many nix hackers already have this on there system anyway, so it'd be nice if Nix could just pull revisions from this local checkout if they are available. We could even potentially use worktrees to decrease the disk space used, although I guess the hard link optimisation would make that a bit redudant.

I feel that any such local checkout as you seem to suggest @roberth would be better living under /nix though, e.g. /nix/nixpkgs. Or if we want to make this concept generic across flakes: /nix/git/nixpkgs.

[roberth]

Many nix hackers already have this on there system anyway, so it'd be nice if Nix could just pull revisions from this local checkout if they are available.

I wouldn't be comfortable pulling NixOS from my work nixpkgs checkout. I feel like this idea mixes technical requirements with (security) policy and user interface too much.

local checkout [...] would be better living under /nix though, e.g. /nix/nixpkgs. Or if we want to make this concept generic across flakes: /nix/git/nixpkgs.

This is a little tricky with the policy free idea. If /nix/git is owned by the system, who decides where the refs should point? Or is it just a bare mirror? I think the policy of "what's the default nixpkgs" should not be decided by Nix itself, but either helped by NixOS (/etc/nixpkgs) or decided by the users themselves: ~/nixpkgs.

I feel that any such local checkout as you seem to suggest

Referring to /etc/nixpkgs, I think this should be a locked flake reference that matches the system inputs.nixpkgs. We could implement this as a flake { inputs.nixpkgs = ...; outputs = inputs.nixpkgs; } + lock, or Nix could implement a feature to make this simple transparent wrt sourceInfo, by reducing it to mostly a lock file + input name. Possible name: flake-link.lock. Nix could load that just like it would load a flake.nix. (But that's for later; the generated flake solution will do)

This would satisfy the performance requirements as well, because this would also allow a git input to be used, and lazy trees improve storage and download overhead by default for github zipball inputs. That's a latency vs bandwidth tradeoff you can make.

[balsoft]

My main grind with the registries is that they are used, by default, to resolve inputs to flakes when doing nix flake update or after altering the required inputs. E.g.

{
  outputs = { self, nixpkgs }: {
    # ...
  };
}

Will take nixpkgs from "the registry" whenever the lock file is updated. This might save a couple of keystrokes on inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable", but comes with multiple bad consequences:

• Different users may get different results even if they run nix flake update at the same time on such a flake, due to differing nixpkgs entries in system or user registries;
• If you have an unusual flake in your system/user registry, you might accidentally use it in the arguments of outputs, and now others can't nix flake update at all;
• It breaks a nice setup with settings nixConfig.flake-registry in the flake, since system/user registry entries override the global registry entries set by flake-registry.

The latter "nice setup" allows us to have a common registry for many flakes within the same context (e.g. organization). This is nice for two reasons:

• No need to type in inputs.bar.url = "github:foo/bar" for dozens of flakes = a lot less boilerplate
• The ability to withhold updates of certain inputs (think haskell.nix or nixpkgs) company-wide to prevent mass rebuilds, while keeping the rest up-to-date with daily nix flake update, and also the ability to update those inputs simultaneously over many flakes, resulting in potentially significant savings in build time and nix store space usage.

A possible solution to this, while keeping the niceties of registries for CLI usage, is to only use the global registry for flake input resolution in such a manner.

[balsoft]

Make nix run /etc/nixpkgs#hello possible

BTW, this is already possible-ish:

❯ nix run $(realpath /etc/nixpkgs)#hello
Hello, world!

[nrdxp]

I just wanted to steel-man a usecase for registries that isn't very common and so possibly easily overlooked, however, there isn't really a good alternative for it via the flake.nix input API so I believe its worth mentioning.

Say you have a library flake that many project specific flakes consume. You are a large organization and you want to manager the dependencies and versions of the inputs to that flake in a way that is simpler than manually pulling each flake and manually modifying the flake.nix with follows (which doesn't even always works since it is limited to two-levels deep).

You can use a registry to solve this problem in the following way:

• Lock the inputs you want to manage to a specific revision, branch, or tag in your own org wide flake registry.
• For the library flake in question, do not commit a lock file, this defers the locking of its inputs to the consuming flake.
• In the project specific flake, set nixConfig.flake-registry to point to your organizations registry.
• The inputs for that library flake will automatically be updated from your registry whenever they are bumped, without any manual follows. This means that updates can be trivially updated, say in CI.

Of course, we could resolve this by simply ammending the flake API in some way that would accomodate a similar use case, such as #6550. However that particular implementation hasn't garnered a whole lot of favour.

Of course this isn't a huge problem with just a single library flake, but with multiple, possibly interrelated lib flakes, it is basically impossible to automate without this.

[balsoft]

Also, FTR, my opinion is that registries (global, system and user) are great for the CLI use-case, but at least system and user ones should not be available while filling in the missing flake.nix inputs. Maybe even the global registry should be disabled by default, and enabled with a config option (e.g. nixConfig.enable-registry-fallback or similar). I would be willing to implement this if people like the idea.

[nixinator]

we really cannot expect people to type nix run github:NixOS/nixpkgs#hello instead of nix run nixpkgs#hello.

well, it just needs tab completion , based on contents of the registry, but you can't use the registry directly .

tab tab tab!

[nrdxp]

tab tab tab!

sure but are we gonna try complete all of GItHub :sweat_smile:

Obviously we can maintain previously completed uris in some kind of cache though :shrug:

As for my previous comment, I'll just respond to myself by saying that I actually agree to get rid of registries in the end, but not until the previous usecase I mentioned has a viable replacement, as that would essentially break my plans to make our large repository ecosystem trivially maintainable with patch release updates, etc.

In general, the locking mechnism of flakes is probably what needs a massive overhaul more than anything for any kind of Flakes 2.0, so I see this as sort of the "cork pop" at the end of that endeavor.

I wouldn't be comfortable pulling NixOS from my work nixpkgs checkout. I feel like this idea mixes technical requirements with (security) policy and user interface too much.

That's not really what I was trying to suggest, you might pull a copy out the mirror nixpkgs is using to hack on though, which is more in line with what I envision there.

[oxalica]

To make a comparison, the global registry is like well-managed dns.

The policy for inclusion of the global registry is undefined and currently contains random packages which not all Nix users (need to) know. There is also PR's like https://github.com/NixOS/flake-registry/pull/39 using the name std, which is misleading.

We should either strictly limit the global registry, or weaken it to be like npmjs or crates.io, which "may contain arbitrary packages and should be used with care". Note that the former option can be implemented by a default value of system flake nix.registry for NixOS,or a default installation of /etc/nix/registry.json otherwise.

See also https://github.com/NixOS/flake-registry/issues/25#issuecomment-1374546965

For the system registry, it's like a local diverse alias set. I believe there are many ones using shorter names like p for nixpkgs. It's good for CLI but should be rejected in flake.nix.

[yangm97]

Get this hidden mutable state layer off my lawn!

There is something to be said for ignoring the local/system registry when creating lock files, but from a UX perspective we really cannot expect people to type nix run github:NixOS/nixpkgs#hello instead of nix run nixpkgs#hello.

I would rather document/add more examples to nix flake init and limit nix run to the current system flake inputs (or this "local registry"). i.e. nix run <nixpkgs>#hello.

If this registry thing continues we will fall into the same hole as podman in regards to name collisions and asking the user "uh I can't really resolve this alias, from which registry do you want me to pull from?".

Later they gave up alternative top level registries and stuck with docker.io like everybody else, which is a centralized repository, something we want to run away from but from empirical observation shows that if we follow their strategy we will end up with the same issues.

deno, from the creator of node, nodejs, npm.io, whatever, doesn't have a centralized registry but instead have users importing stuff from urls, because they are unique, unambiguous and can be hashed in a lockfile as usual. That must be saying something about centralized registries.

Like with a CAS nix store and flake lockfiles we don't even need to restrict ourselves to HTTP downloading stuff from that specific url, we could have some sort of genericFetcher pulling stuff from wherever it can find i.e. from ipfs, torrent2, metalink, whatever, all in parallel then piece everything together and call it a day.

TL;DR:

• Forget about global/organization/project flake registry. In the long run this brings more pain than writing a full URL by hand
• This is more of a UI/UX issue than technical challenge
• We want to have the least effort and the most features but hopefully not compromise the architecture in name of usability

[RuRo]

The system-wide registry is also useful for pinning the "default" nixpkgs to be the same version, that was used to build the system. This use case applies to both the CLI case and the flake inputs resolution.

[lheckemann]

I can see a valid use for not having this state affect evaluation results. However, why stop there? nix run github:NixOS/nixpkgs#hello relies on (truly) global mutable state too! One should instead use nix run github:NixOS/nixpkgs/f5696ecb413df95d780188f956d74db631a66891#hello!

I think where I'm going with this is that removing the registry won't eliminate the effect of global mutable state on evaluation results, and significantly worsen actually working with Nix. What I do see a valid use for (e.g. for CI) is a mode where as much of this state as possible is eliminated -- so registry references are disallowed, but so is anything else that isn't a fully locked input. This should not be the default, because it's horrible to use and Nix should provide a CLI that's reasonable to use in a terminal on your laptop without wrapping and aliases. I would definitely appreciate it being a nix.conf option. This could even be the future of the existing restrict-eval option.

[roberth]

The point is not to avoid all global state, and arguably state that's truly global, like github, is quite predictable, and that's the point of this. Do you think /etc/nixpkgs is that much worse than nixpkgs?

!

I should probably make the title less baity, or expect more agitated replies.

a CLI that's reasonable to use in a terminal on your laptop without wrapping and aliases

I agree.

[RuRo]

Do you think /etc/nixpkgs is that much worse than nixpkgs?

Yes.

/etc/nixpkgs:

• isn't available out of the box (the user needs to set it up)
• there is no CLI to inspect and/or manage it
• there is no out-of-the box equivalent to nix.registry
• there is no support for multiple levels of overrides (global->system->user)
• there is no equivalent for --registry mycompany.org/registry or --registry /path/to/registry.json
• it can't be used inside flake inputs, only on the CLI

[balsoft]

Once again, I think we should first focus the discussion on registries while resolving inputs in flake.nix. I strongly believe that system and user registries are harmful in that respect, fundamentally because two users running the same nix flake update at the same time may get wildly different results, without ever noticing or realizing. I do think that those registries are fine for the CLI use-case, since there the user sees the registry entry in question as part of the command, so there's less possibility for confusion when the results differ.

The system-wide registry is also useful for pinning the "default" nixpkgs to be the same version, that was used to build the system. This use case applies to both the CLI case and the flake inputs resolution.

I think it can be solved in a flake.nix with an explicit nixConfig.flake-registry = "/etc/nix/registry.json"; (or wherever the system registry is stored, I forget).

[RuRo]

I strongly believe that system and user registries are harmful in that respect, fundamentally because two users running the same nix flake update at the same time may get wildly different results, without ever noticing or realizing.

I disagree with the whole premise that nix flake update should be in any way reproducible. There is no such thing as "at the same time" when it comes to distributed systems and this is a distributed system. Unless the input url is already referencing a specific revision, you can't guarantee, that 2 flake updates will have the same result even without registries.

This is even worse on the CLI, because there the default tarball-ttl value is something like 1 hour IIRC. So even if 2 users try to nix run github:nixos/nixpkgs#hello at exactly the same time, it's still quite likely that they will get different commits due to caching.

Flakes enforce reproducibility using the flake.lock file. That's the whole point. I don't understand, why do you need "reproducible" behaviour when creating/changing the lock file.

I think it can be solved in a flake.nix with an explicit nixConfig.flake-registry = "/etc/nix/registry.json"; (or wherever the system registry is stored, I forget).

This doesn't work. You can't put the "opt-in" flag inside flake.nix. The person, who is "opting in" is supposed to be the system owner, not the flake owner. Additionally, this means that this flake would completely fail to evaluate on systems where nix doesn't live in /etc/nix.

If you are dead set on making it an "opt-in" behaviour, then at the very least it should be something like use-registries-in-flakes = true in nix.conf, not in flake.nix. But I am still not convinced that the current behaviour is problematic in the first place.

[balsoft]

That's the whole point. I don't understand, why do you need "reproducible" behaviour when creating/changing the lock file.

I understand that it will never be reproducible between users, but it does have to be consistent between users. Suppose that someone adds their own nixpkgs fork to their user registry (which is a quite reasonable thing to do for the CLI use-case, I would say). Now, every time they nix flake update on any flake with nixpkgs input coming from the registry, they will screw up flake.lock for everyone else, because now nixpkgs points to their personal fork. (This is precisely why I always make my inputs explicit in my own flakes). I have been bitten by this before when using flake: inputs (with nixConfig.flake-registry, but that's beside the point), many times. This is a real pain-point.

I guess the property that I want to be maintained is that any flake: input is always resolved to the same repository, with modifiedDate of the revision monotonically increasing with sufficiently big time steps, for everyone.

Once again, I do agree that the current behavior is fine for CLI, mainly for the reason that you type out the registry entry name yourself.

The person, who is "opting in" is supposed to be the system owner, not the flake owner

I strongly disagree. Flakes for which the person might want to do this are almost universally going to be their flakes, so they as the owner decide to follow a specific registry for their input updates.

I think code style guides are a great analogy here. Of course, everyone would prefer to use their personal code style, but for any particular project its developers have to agree on a style guide and then use that. It does not make sense for different developers to use different style guides, as that will inevitably lead to inconsistencies and "edit wars".

As source files are affected by the auto-formatter, flake.lock (which is a file committed to the repo) is affected by nix flake update. For exactly the same reason as code-style, it only makes sense for the flake/project owners to choose the registry they want to take updates from (with a fallback on the universal, global registry if no choice is made). Any other choice will lead to edit wars of flake.lock.

Additionally, this means that this flake would completely fail to evaluate on systems where nix doesn't live in /etc/nix.

It will not fail to evaluate (I just checked that). The fact that it would fail to nix flake update is OK IMHO, since the flake owner's intention clearly was for the inputs to follow their system's nixpkgs version, and not someone else's, so nobody else should be running nix flake update in the first place.

[RuRo]

Now, every time they nix flake update on any flake with nixpkgs input coming from the registry, they will screw up flake.lock for everyone else, because now nixpkgs points to their personal fork.

Well then don't do that? Doing a blanket nix flake update is probably a questionable idea anyways. IMHO, you should almost always use nix flake lock --update-input someflake or whatever. And even then, flake.lock changes should be reviewed before merging (even without registries).

The way I see it, by using the registry you are opting in to this behaviour. So if you pinned "nixpkgs" to something that is not actually "the" nixpkgs, then it is your responsibility to keep that in mind. You wouldn't submit a bug report after nix run nixpkgs#hello doesn't work without checking against the "real" nixpkgs and you shouldn't update an input named "nixpkgs" without --registry "" or --override-inputs nixpkgs upstream-nixpkgs or whatever.

Of course, you could argue, that you only wanted to opt in to using registries on the CLI, but then we are back to discussing different opt-in/opt-out mechanisms. Perhaps, the registry entries themselves could be marked as CLI-only?

I strongly disagree. Flakes for which the person might want to do this are almost universally going to be their flakes, so they as the owner decide to follow a specific registry for their input updates.

Fine, I'll concede this point. You convinced me. Having a per-flake flag makes sense.

It will not fail to evaluate (I just checked that).

Sorry, I meant "evaluation" during locking, not "nix expression evaluation" during build etc.

The fact that it would fail to nix flake update is OK IMHO, since the flake owner's intention clearly was for the inputs to follow their system's nixpkgs version, and not someone else's, so nobody else should be running nix flake update in the first place.

Specifying a registry path is still not good enough, imho. It doesn't provide a hierarchy of overrides like the current registry system and it's kind of ugly, if I'm being honest. Also, you are assuming, that the owner only uses/develops this flake on a single machine.

There's also a (slightly cursed) workflow where you .gitignore the flake.lock (although it's currently slightly broken in stable versions of nix, IIRC).

I still think, that we should have a flag for enabling/disabling system and user registries in flake.nix:

1) a flag on the CLI (nix flake lock --blah) 2) a per-flake override (nixConfig.blah = true) 3) a global default in nix.conf

Where blah is either use-registries-in-flakes or disable-registries-in-flakes (or a shorter, more descriptive equivalent) depending on if we decide that this behaviour should be opt-in or opt-out.

Can you explain, why are you against a) allowing setting a global default, b) accepting a boolean flag instead of a specific path?

"You can potentially shoot yourself in the leg with this option" isn't a very good argument for completely banning the current behaviour, imho.

pic related


[balsoft]

Well then don't do that? Doing a blanket nix flake update is probably a questionable idea anyways.

Again, I disagree. nix flake update is a common operation in environments where I have observed flakes being used. It is simply a way to say "I want the latest versions of everything", pretty much the same as updating nixpkgs (where you also get a lot of updates at the same time).

And even then, flake.lock changes should be reviewed before merging (even without registries).

Sure, here I absolutely agree. However, the aforementioned issue creates a lot of unnecessary mistakes which have to be caught. I don't think it's a good idea to ship features which empirically lead to mistakes.

Of course, you could argue, that you only wanted to opt in to using registries on the CLI

Yep, that's precisely my argument.

Specifying a registry path is still not good enough, imho. It doesn't provide a hierarchy of overrides like the current registry system

That's fair, but I think the current hierarchy is rather arbitrary (only 3 registries, where permissions, ways to override them, and their priority are independent). It would be nice to be able to specify multiple registries, with an order of priority. However, I think it's an issue which could be handled separately.

and it's kind of ugly, if I'm being honest.

To my taste, relying on two layers of implicit mutable state is a lot uglier. It's the disappointing "... but I don't know why or how" that often comes with "Wow, so this just works".

I still think, that we should have a flag for enabling/disabling system and user registries in flake.nix

I'd be fine with that, as long as it is opt-in (as in disabled by default). However,

Can you explain, why are you against a) allowing setting a global default

I'm still a bit iffy on allowing to set it in the global config, for exactly the same reasons as a global option to override project-specific code style with my "personal default" code style: I just don't see many places where it would actually be useful and not annoying to other contributors for flakes you're working on. I do however understand that such situations probably exist and so would be fine with a deliberate opt-in, with the option description outlining the possible implications.

b) accepting a boolean flag instead of a specific path?

Those are two entirely different things. nixConfig.flake-registry overrides the path to the global registry. The opt-in setting under discussion would enable system and user registries (which are currently enabled by default, impossible to disable without also disabling the global one, and impossible to override the path for). The former already exists, and it is a way to set a specific registry for a flake rather than using the default global one. The latter should exist as a compliment to the former, as a way to opt-in to local, stateful overrides of that registry, whether it is a flake-specific one or the default one.

"You can potentially shoot yourself in the leg with this option" isn't a very good argument for completely banning the current behaviour, imho.

I agree that there should be a way to get the current behavior back (at least on a per-flake basis). I'm just advocating that it's the wrong behavior to have by default, just like I think it's wrong to equip every infant with a rifle at birth.

[RuRo]

I don't think it's a good idea to ship features which empirically lead to mistakes.

If you'll allow me a little bit of snark, any system that is used by humans empirically leads to mistakes. No system == no mistakes.

On a more serious note, I would argue, that these "problems" with the registry come from a misunderstanding/ambiguity about how registries are "supposed" to be used. To be fair, the documentation for registries only seems to describe how registries work and not the intended semantics for their use.

I was under the impression, that the "primary" use case for overriding stuff with registries is 1) pinning stuff and 2) mirroring stuff. In these cases, the flake would still point to the same "project" (but maybe to a different revision), so an flake update would still resolve to a valid version in the project.

You certainly can use the user registry to point nixpkgs to my-cool-pkgs, but I always thought that it's then your responsibility to ensure that the resulting semantics make sense. Kind of like --override-inputs, but you don't need to manually add it to each command.

So, if you have a fork of nixpkgs that is meaningfully distinct from the upstream, then you should either add it to the registry as mypkgs (or something) OR add it as nixpkgs and be prepared to handle all the problems that arise from breaking the contract.

That's fair, but I think the current hierarchy is rather arbitrary (only 3 registries, where permissions, ways to override them, and their priority are independent). It would be nice to be able to specify multiple registries, with an order of priority. However, I think it's an issue which could be handled separately.

Three levels are sufficient for me, but I have nothing against extending this system. As long as it doesn't require listing all the registries in one place (so something like flake-registries = [...] wouldn't work for me). Of course, this kind of change is probably out of scope for this issue.

b) accepting a boolean flag instead of a specific path?

Those are two entirely different things. nixConfig.flake-registry overrides the path to the global registry. The opt-in setting under discussion would enable system and user registries.

Ah, sorry, I thought that this comment from you

I think it can be solved in a flake.nix with an explicit nixConfig.flake-registry = "/etc/nix/registry.json"; (or wherever the system registry is stored, I forget).

implied that the current 3 tier registry system should be removed without adding any new flags, because specifying a flake-registry is sufficient. We are in agreement that there needs to be an option to keep the current behaviour, if we end up changing the default?

So the only open questions (for me) are

1) whether the default should be changed at all 2) whether there should be a global flag or only a per-flake one

I'm just advocating that it's the wrong behavior to have by default, just like I think it's wrong to equip every infant with a rifle at birth.

Well, in this analogy, the rifle is unloaded by default, because NixOS ships with empty system and user registries.

But this is once again a question of whether we frame this discussion as either

1) "the current behaviour of registries is broken" => "the default should be changed"

2) "the behaviour appears broken, because registries aren't supposed to be used like that" => "this is a documentation/advertisement issue" => "a CLI-only mode for registry entries may be added to support this different use case"

[balsoft]

So, if you have a fork of nixpkgs that is meaningfully distinct from the upstream, then you should either add it to the registry as mypkgs (or something) OR add it as nixpkgs and be prepared to handle all the problems that arise from breaking the contract.

Right, but even if you pin nixpkgs to a specific nixpkgs revision, you can get "rollbacks" if you do nix flake update. I would say this is quite bad still, especially for projects with multiple collaborators.

As long as it doesn't require listing all the registries in one place (so something like flake-registries = [...] wouldn't work for me).

Why? I was playing (so far, only theoretically) with an idea of implementing a similar system. There would be two such lists, one for CLI and one for input locking. As for values, there would be special cases of "user" and "system", and the defaults would be [ "user" "system" "https://raw.githubusercontent.com/nixos/flake-registry/master/flake-registry.json" ] for CLI and [ "https://raw.githubusercontent.com/nixos/flake-registry/master/flake-registry.json" ] for input locking. I think it actually resolves the majority of problems here, while still allowing user and system registries when desired.

[RuRo]

Right, but even if you pin nixpkgs to a specific nixpkgs revision, you can get "rollbacks" if you do nix flake update. I would say this is quite bad still, especially for projects with multiple collaborators.

Well, yeah, you can get rollbacks. But this is also a case of "you got exactly what you asked for".

Why? I was playing (so far, only theoretically) with an idea of implementing a similar system. There would be two such lists, one for CLI and one for input locking. As for values, there would be special cases of "user" and "system", and the defaults would be [ "user" "system" "https://raw.githubusercontent.com/nixos/flake-registry/master/flake-registry.json" ] for CLI and [ "https://raw.githubusercontent.com/nixos/flake-registry/master/flake-registry.json" ] for input locking. I think it actually resolves the majority of problems here, while still allowing user and system registries when desired.

Ah, yeah, what you are describing would work. Although having to explicitly spell out the full nixos/flake-registry url sounds like a pain, so you should probably rename the old flake-registry (singular) option to global-flake-registry and have an extra "global" special case that refers to whatever is specified in global-flake-registry.

You might want to also allow extra-flake-registries so that it's easier to extend the current registry set rather than overwriting it. Also, isn't the order backwards? extra- config variables normally get appended, so it would make more sense to have [ "global" "system" "user" ] be the default rather than [ "user" "system" "global" ].

[balsoft]

Also, isn't the order backwards?

It's the same as for PATH or NIX_PATH priority-wise, but I guess there's nothing stopping us from reversing it :)

[yangm97]

I see some straw men and tangential issues flying around since my last comment.

Yes, registries should be abolished. No, that won't impact CLI UX. The one and only true "registry" should be the flake inputs used by the current system.

That means:

• nix run nixpkgs#foo provides the same output for two users on the same system
• Faster evaluation of nix run since there's no time spent pinging random APIs (+1 privacy?)
• No ambiguity/override hell
• flake input rules still apply for fetching an updated nixpkgs i.e. nix run github:nixos/nixpkgs
• downstream nixpkgs could always overlay on top of nixpkgs, that hasn't changed with flakes and is tangential to this issue anyways

In short, I don't see any benefits in flake registries, they don't seem to be worth the maintenance burden and there's no single thing it does that nix init and equivalents wouldn't.

If you guys want unpredictable short-name resolution so much then we should shift the discussion from "removing registries" to "using namecoin/ipns/dns/WoT/whatever for registries", because that would at least eliminate the idea of a blessed centralized source of truth and make "registries" truly decentralized, through the power of outsourcing 🤣

[RuRo]

I see some straw men and tangential issues flying around since my last comment.

Yes, registries should be abolished. No, that won't impact CLI UX.

In short, I don't see any benefits in flake registries, ...

🤣

"I don't understand feature, so feature - bad."

"My use case is not affected, so it's not worth the maintenance costs."

"Your use case won't be affected. Because I said so, trust me."

"Everyone, who disagrees is attacking straw men."

"🤣"

I mean, I kind of understand where you are coming from, but just reiterating your opinion doesn't seem like a productive way to convince people who disagrees with you, no? Especially, when you take the absolutist position (abolish registries, full stop, no compromises, no backwards compatibility).

[LiGoldragon]

Related: The registry really should be a named map instead of a list. Overriding the registry is currently harder than it needs to be.

[roberth]

Overriding the registry

Seems like you're trying to compose your a registry file. With NixOS? You could prototype your idea by adding an option whose value gets translated to nix.registry definitions.

[LiGoldragon]

Seems like you're trying to compose your a registry file. With NixOS? You could prototype your idea by adding an option whose value gets translated to nix.registry definitions.

Ahh yes! To bootstrap KriomOS with NixOS. Good idea. I'll make some PRs with some of the concepts I'm fleshing out. I just rebased your nixpkgs module, and was dealing with hard pinning nixpkgs in the registry. I'll be coming back to coding now. Thanks again for your insights and interest.

Pluto about to square Rahu's entry into Aries- big changes at the end of this month. Enjoy!

[tejing1]

I would simply draw the line at the registry only affecting things typed into the CLI and getFlake in impure code. In particular, this means indirect flake references in flake inputs should be abolished.

Also, the registry should only be able to affect flake urls that would not resolve without a registry. No redirecting what should be a well-defined absolute reference to something else. If you want to pin something, use an alias. This ensures that an end user can easily predict whether the command he typed does a registry lookup or not.

Having a set of aliases used locally for CLI interaction is useful and not a source of problems on its own, and having them available in impure code via getFlake is also pretty much fine. (It's basically NIX_PATH all over again. If you're in impure code, that kind of thing is something you've got to live with. And it seems to be something of a necessity in practice.)

As for the "which registries" question, I tend to think the global registry isn't of any benefit, and of slight detriment, overall. (Which is why I point it at an empty local registry on my own machines.) But system and user registries make sense to have, and should probably be seeded with a nixpkgs entry on a new install, much as is done with NIX_PATH.

[teto]

disclaimer: this message adds nothing new from past comments (and I already knew about the issue https://github.com/NixOS/nix/issues/7005) but I got bitten by this today. I removed the input A from my flake (since the package A is now avalable from the nixpkgs input), nix flake update, nix develop and my scripts failed . all of a sudden my script stopped working. I ran A --version and got an older version. I was really surprised then looked at the lockfile referencing old urls and remembered about the nix registry.

This can't be the definitive UX, what's the point of making flakes pure by default to then trick them with the registry.

And I dont want to set use-registries = false else I lose the convenient

nix run nixpkgs#A            
error: 'flake:nixpkgs' is an indirect flake reference, but registry lookups are not allowed

I dont have the solution but the registry looks to generate many issues (looking at the nix tracker) for little benefits. Could we have a reserved keyword registry that points at a local flake (or registry, if the implementations differ) and reference that instead to make it explicit ? aka in the flake

inputs = {
   nixpkgs = registry.nixpkgs;
}

or even

inputs = registry: {
   nixpkgs = registry.nixpkgs;
}

[teto]

oh and I just saw on https://github.com/NixOS/flake-registry?tab=readme-ov-file#inclusion-policy

Common names should be avoided to prevent confusion if a user mistypes an input in their flake.nix and it resolves to another input via the registry.

which doesn't inspire confidence.

[RuRo]

Could we have a reserved keyword registry that points at a local flake (or registry, if the implementations differ) and reference that instead to make it explicit ? aka in the flake

inputs = registry: {
   nixpkgs = registry.nixpkgs;
}

Actually, I think that this is a great idea (if we could change the flakes spec, of course, which might not be possible given the whole fiasco with Determinate Systems promising flakes to be stable/backwards compatible forever).

Flakes were meant to be pure after you lock the inputs and the registry can be a useful tool for personal flakes, so completely "killing" the registry is probably overkill, but making the usage of the registry explicit sounds like a reasonably sound middle ground.

It's essentially an opt-in mechanism, but with better UX/DX. We might even be able to slightly alter the semantics of the inputs that explicitly reference the registry to make them more convenient to use as a dependency injection mechanism.

[bew]

FWIW my workaround to keep CLI convenience but (kinda) avoid flakes purity issues is to replace the registry with only pkgs for my current nixpkgs and unstable for unstable.

So flakes can use nixpkgs without fear of using the one from the registry because there isn't one, and the CLI can do nix run pkgs#foo.