Open delroth opened 2 months ago
https://hydra.nixos.org/jobset/equinix-metal-builders/aarch64-community-box now builds the system for netboot.nixos.org.
There's a pile of horrible hacks on https://github.com/lheckemann/aarch64-build-box which is currently running fine but is almost certainly not what we want long-term.
nix-netboot-serve
does some Unusual things with the boot environment, preserving the initrd as the final root filesystem, but we don't want that here. Thus, we apply some deep hacks to have the classic NixOS stage-1 used as the initrd init (even though our initrd contains the whole system), copy the system closure into the real filesystem, and then hand over to something resembling the regular stage-2 (further hacks applied to get the system config path correctly).
We'll probably want to change nix-netboot-serve up a bit to support our use case better -- most importantly getting closer to the regular NixOS boot process, running stage-1 and stage-2 as usual. We should think about this some more at more reasonable "awake times". :)
Note: long term we probably want to converge the semi-persistence setup used on the aarch64-build-box with the one used on hydra builders (which has issues), so taking care of this with the aarch64-build-box seems like a very valuable time investment.
We run a c3.medium.x86 instance on packet that literally only serves static files on domain name netboot.gsc.io for the aarch64 community build box to be able to netboot.
It would be better to:
@lheckemann suggested he might be help to pair up with me on this.
cc @cole-h