Run NixOS tests on Darwin

[nmattia]

Issue description

It would be great to be able to run NixOS tests on Darwin. Right now building/running NixOS tests fails on Darwin when building the Linux VMs.

If one has a Linux builder available, it should be possible to:

1. Build the Linux VMs on the Linux builder.
2. Run the tests on Darwin as well as running the start scripts (for interacting with the VMs locally) using hvf instead of KVM.

My interpretation is that build-vms.nix should be tweaked. buildVM should take a Linux package set (pkgs with system = "...-linux"), and the rest of the build (driver, buildVirtualNetwork) should take a Darwin package set.

https://github.com/NixOS/nixpkgs/blob/93db55bde5ef7d405c45e76f7304eb3559e80fc4/nixos/lib/build-vms.nix#L30-L43

Then the qemu commands should be changed to use accel=hvf (instead of =kvm) on Darwin.

CC @basvandijk

[matthewbauer]

You probably only need to do something like this:

         { key = "qemu"; system.build.qemu = (import pkgs.path { system = builtins.currentSystem; }).qemu; }

Everything else should be part of the actual image.

[edolstra]

If you already have a Linux builder, then can't you just run the NixOS tests on the Linux builder?

[basvandijk]

For debugging purposes it would be nice if NixOS tests can be run interactively on darwin, i.e.:

$ nix-build my-nixos-test.nix -A driver
$ ./result/bin/nixos-test-driver

When you do this on NixOS you get a QEMU window per VM with which you interact with the VM. It would be great to have the same feature on OS X.

[roberth]

This would be great for testing arion on darwin. The arion command should run on darwin but requires a linux builder and a docker host for a complete test. A NixOS test with a darwin test driver is perfect for this.

[stale[bot]]

Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on irc.freenode.net.

[roberth]

Still relevant. MacOS users should be able to interactively develop and debug NixOS tests.

[matthewbauer]

One feature of Qemu that doesn't work on macOS is "virtfs" which the tests use to mount /nix/store onto the VM. We will need https://patchwork.kernel.org/patch/10468143/ to be merged to do this.

[infinisil]

@matthewbauer I tried to apply that patch (see https://github.com/Infinisil/nixpkgs/commit/12da31367061012ba3af9e05433799386b4927eb), but the darwin build fails with that due to missing header files sys/fsuid.h, sys/vfs.h, linux/fs.h and cap-ng.h, which are probably linux specific. And of course I get compilation errors if I just try to remove those (see https://github.com/Infinisil/nixpkgs/commit/61aa1e1c4b1c0d71aefe302237e5d71b48a34d04):

fsdev/virtfs-proxy-helper.c:82:9: error: use of undeclared identifier 'CAP_CHOWN'; did you mean 'T_CHOWN'?
        CAP_CHOWN,
        ^~~~~~~~~
        T_CHOWN
./hw/9pfs/9p-proxy.h:53:5: note: 'T_CHOWN' declared here
    T_CHOWN,
    ^
fsdev/virtfs-proxy-helper.c:83:9: error: use of undeclared identifier 'CAP_DAC_OVERRIDE'
        CAP_DAC_OVERRIDE,
        ^
fsdev/virtfs-proxy-helper.c:84:9: error: use of undeclared identifier 'CAP_FOWNER'
        CAP_FOWNER,
        ^

[domenkozar]

There's an initial prototype in #108984, so I'm closing this as a duplicate not to shut off the more recent discussion there.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/run-nixos-interactive-tests-on-aarch64-darwin/34534/1