NixOS / infra

NixOS configurations for nixos.org and its servers
MIT License
208 stars 91 forks source link

Fix Darwin passthru VM tests with remote builders #439

Open roberth opened 3 weeks ago

roberth commented 3 weeks ago

Is your feature request related to a problem? Please describe.

Since NixOS tests can run on Darwin since a few months, OfBorg is trying to run them, but these tests rely on native Linux builds, which should be performed by a remote (or local VM) builder.

Describe the solution you'd like

Ideally, all OfBorg nodes can delegate a few builds to each other node, but the remote build implementation does not currently support cyclic remote builder setups.

As an alternative, we could configure the OfBorg cluster in two layers, groups of machines, where the nodes in the top layer run the OfBorg evaluations, and the lower layer only performs builds. The top layer would not receive remote build requests from other nodes in the top layer, and the lower layer only receives builds for the intended derivation system, making this setup acyclic.

If the top layer also performs builds for their native system, that also works, but is perhaps less effective at distributing work, as they can only be allowed to build locally evaluated derivations.

Describe alternatives you've considered

Make the passthru tests use cross compilation. I don't think this is desirable; see https://github.com/NixOS/nixpkgs/pull/317651#pullrequestreview-2102364647

Additional context

The docker-tools-cross VM test also has a multi-system derivation graph that requires remote builders to work. (Cross compile an image on one system, run on another)

vcunat commented 3 weeks ago

This is @cole-h's territory I think. (but I'm not sure this will get high enough relative priority)