Bootstrappability and independence from binary sources

[zeuner]

Question

What's your stance towards reducing dependencies from binary sources and improving the bootstrappability of more packages with minimal binary downloads?

Candidates I'd like to get an answer from

No response

Reminder of the Q&A rules

Please adhere to the Q&A guidelines and rules

[mschwaig]

Relying less on binary sources is great for supply chain security, so I think moving into that direction is a good thing.

On the other hand I think these kinds of changes need to be made in a way where they do not introduce meaningful regressions. I see having both a source- and binary-derived version of the same package as a good option for packages that often see on of those options getting broken.

I would also appreciate work on the small set of 'bootstrapping' packages we use to bootstrap the rest of nixpkgs. This is one area where members of the Guix community have done a great job. What @t184256 presented about this topic at NixCon 2023 (https://www.youtube.com/watch?v=Gm8yrvbgY-Y) looked very interesting as well.

If we would manage to move to content addressed derivations at some point, that would make working on the bootstrapping process of nixpkgs much less disruptive. Then, as long as you end up with the same content hash at the end of your bootstrapping process, you can take advantage of the early-cutoff optimization and stop rebuilding at that point instead of having to rebuild the world.

[getchoo]

What's your stance towards reducing dependencies from binary sources

I think this is something very important not just for security, but also improving Nixpkgs' platform support. By avoiding binaries from upstream sources (that usually only build for x86_64-linux, and maybe aarch64-linux if you're lucky), we can open the door for distributing packages for many more common and exotic architectures. There is tangible action here we can take here too, as many packages that are FOSS but currently extract and patch an AppImage/.deb could most likely be built from source -- their build process is just very complicated. This can be eased heavily by introducing new tools and hooks for package maintainers, such as pnpm.configHook and cargo-tauri.hook

improving the bootstrappability of more packages with minimal binary downloads?

As said above by mschwaig, taking inspiration from the Guix bootstrap and t184256's talk is definitely something to explore. It would require a good amount of work, but they would be a great improvement over our current boostrap process. We should also target more native bootstraps IMO

[roberth]

From what I've heard the work to do this in Nix is basically done, but yet to be incorporated into Nixpkgs.

This seems like an "obviously good" improvement to me, and I don't think it would be controversial at all.

[numinit]

Full send, ship it. The bootstrap seeds are too large. I like some of the work that's being done with http://pnut.sh/, where we can compile a C compiler (TCC) as long as we have a POSIX shell. https://nlnet.nl/project/Gash/ can be that shell, since it's implemented in Scheme. And Scheme will probably be able to be bootstrapped in the future from a metacircular evaluator that can be implemented using bin0...

Anyway, what I'm saying is that we should help support some of the GNU full-Scheme bootstrap work, because it's really cool, and a great usecase for Nix.

[proofconstruction]

Once upon a time, I was a Gentoo user and built everything from source. It was pretty painful as a casual desktop Linux user, especially when I'd update things like the KDE suite or Firefox and leave my poor Pentium to burn for days at a time. These days, I have a much better appreciation for why we'd want to strive towards source bootstrapping, particularly as I work to package things from sectors of the tech industry with less software engineering sophistication. Nix, NixOS, and most of Nixpkgs already have a good story here, and where things fall short is mostly out of our control anyway. While I believe this is an important goal in general, I don't currently believe this issue should take precedence over some of the other things we need to address.

[Infinidoge]

Nix (and Guix by extension) are uniquely suited for a source bootstrap, as we can get both the benefits of a source bootstrap and the benefits of downloading compiled binaries. It's very cool work and I think we should support those who are working on it. However, I don't think it should take precedence over many of the existing problems with the Nix ecosystem, which could be made more complicated by trying to instill a complex full-bootstrap process.