Proposal: move tooling that's used within nixos and nixpkgs in-tree

[eclairevoyant]

Issue description

Currently we have some important tooling developed out-of-tree:

• nix-update, used as an update script and even recommended in the nixpkgs manual itself
• nixpkgs-update, used by the update bot
• calamares-nixos-extensions, used in the graphical ISOs
• nix-init, used as a way to generate new package expressions, with explicitly designed support for nixpkgs etc. (There may be more projects that I've not recalled offhand.)

However, if we want to fix issues, this depends on an external project, which in almost all cases has a bus factor of 1. Therefore we are hindered by the pace of external development rather than being able to solve our own problems in-tree. Checking the list of open, unresponded-to issues and PRs in the above repos should give a good idea of the concern.

It also prevents community consensus from being able to improve said tooling, instead depending on a single maintainer to respond and agree.

If there's no defensible objection, I propose moving such tooling in-tree.

CC @Mic92 @ryantm @vlinkz @figsoda for the projects mentioned above

[Mic92]

What I am worried about is that one needs to compromise on CI. For nix-update we need to impurely fetch stuff from the internet and I would like to keep doing this as mocking is very hard to do reliable. Also all the usual other CI checks such as treefmt/mypy and so on. Also issues related to these tools would get very easily drowned by all the other issues with have in nixpkgs already, so they are not visible to both the maintainers and the user. Project wise I try to add at least one other person with git commit access to all my important tooling. For nix-update, it's currently @figsoda and @SuperSandro2000. If you have more maintainer suggestions, let me know. There is one open pull request waiting for feedback of the developer and one pull request marked as draft. Releases happen often after important features are added.

[SuperSandro2000]

Absolutely not!

The nixpkgs tree is already filled with multiple projects which shouldn't be in here like yarn2nix. This not only makes PRing them hard and requires to clone nixpkgs for every contributor but also makes issue tracking, release management and probably more unnecessarily hard. We can always track patches for critical fixes. The correct move would be to move them out but keep them in the NixOS org or at least in nix-community. For nix-update I would suggest to move that to nix-community.

[eclairevoyant]

What I am worried about is that one needs to compromise on CI.

Thanks for raising the point about CI, it would definitely change how we have to test the relevant programs - if any of the tooling were to move in-tree, that must be a primary consideration. If it helps, I can draft how the testing would look like to make sure all the primary use cases are covered.

issues related to these tools would get very easily drowned by all the other issues with have in nixpkgs

I agree this risk is present. nix-update specifically is in a better shape, especially since as you mentioned there are others with commit access, so I am OK with considering it a special case of "let's not break what seems to be working".

I'm still interested in addressing the other projects mentioned. If the filed issues aren't addressed in the respective upstream projects, then they get created here anyway, and we just have to slap a "waiting for upstream fix" label or such.

The correct move would be to move them out but keep them in the NixOS org or at least in nix-community.

nix-init is part of nix-community already, and the calamares scripts are part of the nixos org itself, but the bus factor in each of those cases is still 1. I myself have PRs open in both of these repos for months that basically got ignored, and it's not like we can request someone else to review because there is only one person to review. How does changing the repo owner solve the issues raised?

And for r-ryantm, the bot is somehow allowed to spam the repo, but we cannot make the code changes needed to stop it from doing so, because it's yet another a one-person external project; how is that reasonable either?

The main goal of moving this in-tree would be to raise the number of people that can merge from 1 to 200-something. And in-tree PRs get more opportunity for feedback from nixpkgs contributors (since these projects have an impact on nixpkgs).

[SuperSandro2000]

What I am worried about is that one needs to compromise on CI.

That's not even the biggest problem with CI. If it is out of tree, you can just add new CI checks without much consideration and if it breaks things, it's just an out of tree repo, not that painful. If it would be in tree, every CI change would need to be carefully reviewed to not impact any other files. Scalability is already a big concern and would then be a major factor in choosing CI actions.

Also some tools can only be configured by placing config files in the root of the repository. This would easily clutter the repository.

Also we will easily find projects which want conflicting settings which are then no longer possible.

Triggering ofborg for every push of every project is very expensive and just wasteful.

Configuring CI actions could also become more challenging if we need to carefully configure everything to not work tree wide.

Installing rust crates from git would also no longer be a viable option of cloning the repo can take multiple minutes.

I myself have PRs open in both of these repos for months that basically got ignored, and it's not like we can request someone else to review because there is only one person to review.

Then the root problem is that there are to little maintainers and we need to introduce more together with the original maintainers.

How does changing the repo owner solve the issues raised?

That we have more power in nix-community to add new maintainers in case the original are totally absent. It basically prevents a hard fork.

Also new maintainers can be added with just a few clicks without them having to accept invites to new orgs.

The main goal of moving this in-tree would be to raise the number of people that can merge from 1 to 200-something.

Will that solve the problem? I doubt it. It would at the same time diffuse responsibility between many people and it wouldn't increase the amount of people with good knowledge about the code and that feel responsible to keep the tool in good working condition.

And in-tree PRs get more opportunity for feedback from nixpkgs contributors (since these projects have an impact on nixpkgs).

For very deeply coupled projects this is correct but most tools you listed are just loosely coupled and work as well with our of tree projects.

[eclairevoyant]

Then the root problem is that there are to little maintainers and we need to introduce more together with the original maintainers.

Okay, what would be the process to do so?

[SuperSandro2000]

Probably the best way is to open an issue, offer your help and see where that goes. If the repo is in nix-community and there is no reaction for a month or so, you can probably ask in the matrix, if you could take over the project.

[eclairevoyant]

I see, thanks for explaining.