NixOS / infra

NixOS configurations for nixos.org and its servers
MIT License
208 stars 91 forks source link

Deprecate netboot.gsc.io ("netboot-foundation" EQM host) #416

Open delroth opened 2 months ago

delroth commented 2 months ago

We run a c3.medium.x86 instance on packet that literally only serves static files on domain name netboot.gsc.io for the aarch64 community build box to be able to netboot.

It would be better to:

  1. Build the community build box image on Hydra, similar to the Hydra builder netboot image.
  2. Switch to using netboot.nixos.org (nix-netboot-serve).

@lheckemann suggested he might be help to pair up with me on this.

cc @cole-h

delroth commented 2 months ago

https://hydra.nixos.org/jobset/equinix-metal-builders/aarch64-community-box now builds the system for netboot.nixos.org.

lheckemann commented 2 months ago

There's a pile of horrible hacks on https://github.com/lheckemann/aarch64-build-box which is currently running fine but is almost certainly not what we want long-term.

nix-netboot-serve does some Unusual things with the boot environment, preserving the initrd as the final root filesystem, but we don't want that here. Thus, we apply some deep hacks to have the classic NixOS stage-1 used as the initrd init (even though our initrd contains the whole system), copy the system closure into the real filesystem, and then hand over to something resembling the regular stage-2 (further hacks applied to get the system config path correctly).

We'll probably want to change nix-netboot-serve up a bit to support our use case better -- most importantly getting closer to the regular NixOS boot process, running stage-1 and stage-2 as usual. We should think about this some more at more reasonable "awake times". :)

delroth commented 2 months ago

Note: long term we probably want to converge the semi-persistence setup used on the aarch64-build-box with the one used on hydra builders (which has issues), so taking care of this with the aarch64-build-box seems like a very valuable time investment.

delroth commented 2 months ago

Filed https://github.com/DeterminateSystems/nix-netboot-serve/issues/26 too