Allowing NixOS VM's to be run on macOS

infinisil commented 3 years ago

People using macOS currently can't interactively run NixOS VM's as described e.g. here, even with a remote Linux builder. In this issue I'm describing some ways in how that could be made to work. Note that speed is important as well, so kvm/hvf and co. should be used if possible.

Relevant issue is https://github.com/NixOS/nixpkgs/issues/64578. Ping @zupo @matthewbauer @nmattia @roberth. Any pointers/help with this is appreciated.

This issue is sponsored by Niteo :)

Using qemu directly (doesn't work)

With this change it's possible to build a NixOS VM run script that is executable on macOS (note that this requires a remote Linux builder)

$ nix-build '<nixpkgs/nixos>' \
  --argstr system x86_64-linux \
  --arg configuration '{ virtualisation.qemu.pkgs = import <nixpkgs> { system = "x86_64-darwin"; }; }' \
  -A vm \
  -I nixpkgs=https://github.com/infinisil/nixpkgs/archive/4d244410ee0f3e3ece5494533217bbafbd95d9b3.tar.gz
/nix/store/2a7cbyp9xp12ddc4lxb4h93dxa5yfndy-nixos-vm

However running the VM doesn't actually work:

$ result/bin/run-nixos-vm                                                          
qemu-system-x86_64: -virtfs local,path=/nix/store,security_model=none,mount_tag=store: There is no option group 'virtfs'
qemu-system-x86_64: -virtfs local,path=/nix/store,security_model=none,mount_tag=store: virtfs support is disabled

This is because qemu doesn't support virtfs on macOS.

It was suggested that this patch could be used to make it support virtfs. This was attempted here, however the build doesn't succeed:

$ nix-build https://github.com/infinisil/nixpkgs/archive/57562282ab34fc44a492f32a13939d77e29d0d9b.tar.gz -A qemu
[...]
fsdev/virtfs-proxy-helper.c:16:10: fatal error: 'sys/fsuid.h' file not found
#include <sys/fsuid.h>
         ^~~~~~~~~~~~~
1 error generated.
make: *** [/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/rules.mak:69: fsdev/virtfs-proxy-helper.o] Error 1
make: *** Waiting for unfinished jobs....
builder for '/nix/store/z8mkhjvaz3r7dixv94c1jprwx8gvl0gh-qemu-5.1.0.drv' failed with exit code 2
error: build of '/nix/store/z8mkhjvaz3r7dixv94c1jprwx8gvl0gh-qemu-5.1.0.drv' failed

Attempting to remove all the missing header files from virtfs-proxy-helper.c, including <sys/fsuid.h>, <sys/vfs.h>, <linux/fs.h> and <cap-ng.h> just leads to compilation errors, seemingly indicating that this is a Linux-only functionality.

This page however mentions that:

FSDRIVER: Either "local", "proxy" or "synth". This option specifies the filesystem driver backend to use. In short: you want to use "local". In detail:

local: Simply lets QEMU call the individual VFS functions (more or less) directly on host.

proxy: this driver was supposed to dispatch the VFS functions to be called from a separate process (by virtfs-proxy-helper), however the "proxy" driver is currently not considered to be production grade.

And indeed, the run-nixos-vm script only uses local, quoting the script:

-virtfs local,path=/nix/store,security_model=none,mount_tag=store \
-virtfs local,path=$TMPDIR/xchg,security_model=none,mount_tag=xchg \
-virtfs local,path=${SHARED_DIR:-$TMPDIR/xchg},security_model=none,mount_tag=shared \

Attempting to not compile that tool with this commit however also doesn't work, failing again on seemingly Linux-specific headers:

$ nix-build https://github.com/infinisil/nixpkgs/archive/5ed493ed6ae957aa510d5292eb373b8b1f4a3db1.tar.gz -A qemu
[...]
/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/fsdev/file-op-9p.h:19:10: fatal error: 'sys/vfs.h' file not found
#include <sys/vfs.h>
         ^~~~~~~~~~~
1 error generated.
make: *** [/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/rules.mak:69: fsdev/qemu-fsdev.o] Error 1
builder for '/nix/store/zfnmaj3jh16x2qlryzyf3m2gy3zcww53-qemu-5.1.0.drv' failed with exit code 2
error: build of '/nix/store/zfnmaj3jh16x2qlryzyf3m2gy3zcww53-qemu-5.1.0.drv' failed

So it seems that qemu just doesn't support virtfs on macOS.

Using libvirt (might work)

Above qemu documentation also has this section, which describes how the same can be achieved with libvirt. While I believe that underneath it just uses qemu as well, there might be some additional libvirt magic happening. And it seems that macOS Mojave supports virtio-9p, in which post the author also uses libvirt successfully.

Currently the NixOS VM runner just passes arguments to qemu. In order to use libvirt we'll have to transform all these arguments to the libvirt equivalent in its XML configuration. There even is a libvirt page describing the equivalent of qemu arguments, which will be very useful. There is also the virsh domxml-from-native qemu-argv command which can supposedly do this transformation automatically, though I didn't have any success with it yet.

In above-linked document describing qemu argument equivalents, the -virtfs (or the -fsdev and -device options it's a shorthand for) option is notably missing. This should be replaced with a <filesystem> section as described in the qemu documentation link above.

It would be a good idea to first manually create a libvirt configuration and verifying that it works on a NixOS VM. libvirt has a graphical interface which could also be used.

Removing the need for filesystem mapping (might work)

The main reason this -virtfs argument is used at all is so that the guest machine in the VM can access the host machines /nix/store, which is where the whole system that the guest runs resides in.

An alternate approach however could be to create a /nix/store image that can be used by qemu as the /nix/store directly, therefore not depending on this filesystem mapping to the host.

See the various make-* files in https://github.com/NixOS/nixpkgs/tree/master/nixos/lib, which could be useful for this

YorikSar commented 2 years ago

Updates:

I've rebased the branch on the latest nixpkgs master, applied fixes for #178954
Rebuilt the example from https://github.com/NixOS/nixpkgs/issues/108984#issuecomment-1150149700 and pushed all rebuilt packages to Cachix
Created pull request for qemu 9p patch: #180221
Created pull request for NixOS modules changes: #180222

Next steps:

Get those pull requests merged
Add a test to Hydra that would run this basic configuration on Darwin platforms for unstable.

ElvishJerricco commented 2 years ago

@YorikSar I have to disable the Nix sandbox, otherwise I get errors trying to build python3.10-requests-2.28.0.drv on darwin. Something about sockets in the test suite. Other than that, it appears to work!

YorikSar commented 2 years ago

@ElvishJerricco That's great! Do you get the same errors if you rebuild python310Packages.requests from master with sandbox? If so, maybe file an issue (or even a PR) about that?

peterwaller-arm commented 2 years ago

@willcohen Oh, yeah, to build this you need to have a Linux builder configured for your Nix daemon.

Ah, that's unfortunate -- please could you edit your comment "So far on my M1 MacBook Pro this flake:" to indicate this requirement, since I've hit this and did not understand the subsequent comments until I had the problem.

Requiring a Linux builder to be configured to allow building the Linux VM from Darwin scuppers my plans to use this. What are new users meant to do if you don't have a Linux builder available? I was hoping to have a configuration I could share with anyone, instruct someone to install ~NixOS~ (edit: nix), and all they'd have to do was build a flake and they'd get a VM. But it seems this isn't possible yet, is that right?

esselius commented 2 years ago

@YorikSar I have to disable the Nix sandbox, otherwise I get errors trying to build python3.10-requests-2.28.0.drv on darwin. Something about sockets in the test suite. Other than that, it appears to work!

Works with sandbox turned off for me too!

I'm using a docker container as a aarch64-linux builder

YorikSar commented 2 years ago

I've created a repository with instructions on how to use results of merged PRs: https://github.com/YorikSar/nixos-vm-on-macos - feel free to check that out and see if it works for you.

Gabriella439 commented 2 years ago

@peterwaller-arm: Correct me if I'm wrong, but I believe if cache.nixos.org can serve enough Linux build products to deploy a blank NixOS VM then you can bootstrap things on a Darwin system without a remote Linux builder by:

Deploying a blank NixOS VM for use as a local Linux builder
Use that to then build other NixOS systems

Gabriella439 commented 2 years ago

I had to add highmem=off to the QEMU run script to get this to work on my M1 MacBook Pro due to https://github.com/utmapp/UTM/issues/3946. Without that fix this causes a kernel protection failure.

Specifically, I made the following change:

@@ -29,7 +29,7 @@

 # Start QEMU.
-exec /nix/store/h5yrqjziml4vwqq7azph6y9r2r267aq6-qemu-host-cpu-only-7.0.0/bin/qemu-system-aarch64 -machine virt,gic-version=2,accel=hvf:tcg -cpu max \
+exec /nix/store/h5yrqjziml4vwqq7azph6y9r2r267aq6-qemu-host-cpu-only-7.0.0/bin/qemu-system-aarch64 -machine virt,gic-version=2,accel=hvf:tcg,highmem=off -cpu max \
     -name nixos \
     -m 1024 \
     -smp 1 \

I'll try to put up a PR to fix this.

Gabriella439 commented 2 years ago

Actually, after investigating this further it looks like the real issue was due to me being on an old version of macOS (12.0.1) and setting highmem=off is an undesirable solution (because it limits you to a smaller address space). Upgrading macOS to 12.4 or newer fixes the kernel protection failure.

Gabriella439 commented 2 years ago

It looks like there is another issue for qemu VMs on macOS: the default DNS server (10.0.2.3) created by qemu as part of user-mode networking (SLiRP) does not work. See:

https://github.com/utmapp/UTM/issues/2353

You have to override it with the IP address of a working DNS server (e.g. networking.nameServers = [ "8.8.8.8" ];) in order for DNS to work on the guest VM.

Gabriella439 commented 2 years ago

Alright, I managed to get the idea I proposed in https://github.com/NixOS/nixpkgs/issues/108984#issuecomment-1250066968 working. Specifically, I created a repository that lets you bootstrap a Linux guest of the same architecture on a macOS host, relying on a pre-seeded cache instead of a separate Linux builder (similar to https://github.com/NixOS/nixpkgs/issues/108984#issuecomment-1234408582):

https://github.com/Gabriella439/macos-builder

You can then use that Linux guest to build and run additional NixOS VMs on macOS.

As far as I can tell, the only remaining improvements I can think of are:

cache.nixos.org should probably host those cached build products instead of using my own cache so that there's a better chain of trust
Maybe we should wait until https://github.com/utmapp/UTM/issues/2353 is closed before marking this done?

I'm not sure if this should be a blocker, though, because the work-around (hard-coding a public DNS server) is pretty easy

Other than that, I don't see any work remaining here.

roberth commented 2 years ago

cache.nixos.org should probably host those cached build products instead of using my own cache so that there's a better chain of trust

We should add the default config toplevel to the nixpkgs-darwin jobsets, so that we can guarantee its availability in the cache.

Perhaps the keys could be loaded through a 9p mount instead of baking them into the config?

Maybe enable automatic gc? That will save some annoyance and confusion. Relevant paths will be gc-rooted on the host anyway, and copying between the host and vm should be fairly quick.

Having this in a nix-darwin module would be awesome.

virusdave commented 2 years ago

Having this in a nix-darwin module would be awesome.

+100 to this idea!

anka-213 commented 2 years ago

How does this compare to https://github.com/nix-community/linuxkit-nix, which also seems to provide a virtual machine that can be used as a remote linux builder for a macOS machine?

auscompgeek commented 2 years ago

@anka-213 that is orthogonal to this issue which aims to allow building interactive VMs (per the first sentence of the description). Looks like it could be used to bootstrap a NixOS VM build though.

SCOTT-HAMILTON commented 2 years ago

So is it now possible to run github actions nixos vm tests on macos runners to speed up the CIs ?

YorikSar commented 2 years ago

Unfortunately, not yet. Current state is that you can build a VM from NixOS configuration that will run on macOS. This PR by @roberth provides a way to run NixOS tests on non-Linux: https://github.com/NixOS/nixpkgs/pull/193336, and it worked for me on macOS, but it is still a draft.

Gabriella439 commented 1 year ago

@roberth: Alright, I added automatic GC and also removed the hardcoded keys like you suggested. The next step is that I'll work on upstreaming this into Nixpkgs so that it can be built and cached as a nixpkgs-darwin jobset

Gabriella439 commented 1 year ago

I have a PR up for this now: https://github.com/NixOS/nixpkgs/pull/206951

schickling commented 1 year ago

@Gabriella439 @domenkozar very excited to see this being pushed over the finish line!

Wondering whether there is any tutorial, blog post or similar that explains this further?

infinisil commented 1 year ago

@schickling The PR added a Nixpkgs manual section: https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder

Gabriella439 commented 1 year ago

@schickling: I'm also going to announce this via a blog post, but that will focus more on the background work and motivation for this

misuzu commented 1 year ago

% nix run nixpkgs#darwin.builder
error: a 'aarch64-linux' with features {} is required to build '/nix/store/6i2m765ifgzihprpzix5f5ghghpi2sk3-install-credentials.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

Am I doing something wrong?

Gabriella439 commented 1 year ago

I think the issue is that it's not being built by hydra.nixos.org because it's missing a platforms attribute. I'm looking into this

Gabriella439 commented 1 year ago

This should hopefully fix it: https://github.com/NixOS/nixpkgs/pull/207459

misuzu commented 1 year ago

It's still not working 😔

% nix run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder
error: a 'aarch64-linux' with features {} is required to build '/nix/store/6i2m765ifgzihprpzix5f5ghghpi2sk3-install-credentials.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}
% nix build --dry-run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder
these 2 derivations will be built:
  /nix/store/6i2m765ifgzihprpzix5f5ghghpi2sk3-install-credentials.drv
  /nix/store/yl0r0qgdrx50ymbrwi66wpxf4krwkrxy-create-builder.drv
these 223 paths will be fetched (220.65 MiB download, 983.89 MiB unpacked):
<..................>

Interesting. https://hydra.nixos.org/build/202864391#tabs-buildinputs nix build /nix/store/920lc5kg8sshjdf54rbdpi9yn2c884mc-create-builder && ./result/bin/create-builder is working, but nix run github:NixOS/nixpkgs/705e2261b4c624ea282186a7f48f4101de7a563f#darwin.builder isn't.

Gabriella439 commented 1 year ago

Yeah, this should hopefully be the last fix: https://github.com/NixOS/nixpkgs/pull/207577

bergkvist commented 1 year ago

This was working for me, but started failing just now:

% nix run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder
cannot build on 'ssh-ng://builder@localhost': error: cannot open connection to remote store 'ssh-ng://builder@localhost': error: unexpected end-of-file
Failed to find a machine for remote build!
derivation: g43xrrmfr80sza9fshg99l1yc80ny1il-nixos-disk-image.drv
required (system, features): (aarch64-linux, )
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
(aarch64-linux, 10, , )
error: a 'aarch64-linux' with features {} is required to build '/nix/store/g43xrrmfr80sza9fshg99l1yc80ny1il-nixos-disk-image.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

% nix build --dry-run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder
these 4 derivations will be built:
  /nix/store/g43xrrmfr80sza9fshg99l1yc80ny1il-nixos-disk-image.drv
  /nix/store/1lhnpvb9n70gz7sf3yapwybl476cd9rd-run-nixos-vm.drv
  /nix/store/hzy3zqwb795fai8zla07hf8c04gb1cvj-nixos-vm.drv
  /nix/store/vcps33v4bra88pv8h00v41mqs3vka3sd-create-builder.drv

% nix flake metadata github:NixOS/nixpkgs/nixpkgs-unstable
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/2d38b664b4400335086a713a0036aafaa002c003
Description:   A collection of packages for the Nix package manager
Path:          /nix/store/ld350gbq2w5654yw4q95kxhxp4569zi6-source
Revision:      2d38b664b4400335086a713a0036aafaa002c003
Last modified: 2023-01-17 10:21:52

Trying a slightly older commit, and it works again:

% nix run github:NixOS/nixpkgs/6fecd5a257327db6116b28acdfb45561e65e3dfa#darwin.builder
Formatting '/Users/tobias/nixos.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=21474836480 lazy_refcounts=off refcount_bits=16
Formatting '/tmp/nix-vm.qudT2qcgrQ/store.img', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1198522368 backing_file=/nix/store/9rk814dr9dr1cxz698iq22g3c5713hfy-nixos-disk-image/nixos.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
...
<<< Welcome to NixOS 23.05pre-git (aarch64) - ttyAMA0 >>>

Run 'nixos-help' for the NixOS manual.

nixos login: builder (automatic login)

[builder@nixos:~]$

From doing a manual bisection, it seems like this commit is what broke the darwin.builder: https://github.com/NixOS/nixpkgs/commit/594b94b4c3038f5c2cfb2f5d9c10ef30c7070a4c

% nix build --dry-run github:NixOS/nixpkgs/594b94b4c3038f5c2cfb2f5d9c10ef30c7070a4c#darwin.builder
these 7 derivations will be built:
  /nix/store/rvcavzsa1y3yk7c3v38rsvbz1lvkm0nn-stage-2-init.sh.drv
  /nix/store/w24wrk1w0cizpj8dc0plyjr04cn45abp-etc.drv
  /nix/store/52ib3iq1hr1c0ynxs9279n5qq8kx4asy-nixos-system-nixos-23.05pre-git.drv
  /nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv
  /nix/store/wr12pkszc3gj021fy08cch9syhqvwyj2-run-nixos-vm.drv
  /nix/store/p5pa1imyb5yc6l3wy1cda5bz38dc430f-nixos-vm.drv
  /nix/store/ld4rlr7sf599qi53bmd4d2xrjsm53n11-create-builder.drv
these 4 paths will be fetched (0.09 MiB download, 0.49 MiB unpacked):
  /nix/store/786ah3bs0n5lcvjq2qhp4laiqkg60wr0-etc
  /nix/store/dvz0v04kb7sw5fa54734i0hynnkfvxa9-closure-info
  /nix/store/mj26i1zj0bjv1b732bafrg2ffly34adj-nixos-system-nixos-23.05pre-git
  /nix/store/wrz223m16k41if9jyyrjpbl6668ny333-closure-info

% nix build --dry-run github:NixOS/nixpkgs/af89d3a2be6f70edb187dd817377d6c4360134fa#darwin.builder

peterwaller-arm commented 1 year ago

@bergkvist (I'm an interested user, non-authoritative comment) One tricky thing to consider is whether the packages are in the cache yet for the commit you're building. If not, your machine would try to build it and fail.

I think the nixos/nixpkgs/nixpkgs-unstable branch might run ahead of the cache being populated. Try nixos/nixpkgs/nixos-unstable instead, which I think is updated as the cache is updated if I understand correctly.

bergkvist commented 1 year ago

When I try to build github:NixOS/nixpkgs/594b94b4c3038f5c2cfb2f5d9c10ef30c7070a4c#darwin.builder inside of my builder (nix run github:NixOS/nixpkgs/af89d3a2be6f70edb187dd817377d6c4360134fa#darwin.builder)

% nix build github:NixOS/nixpkgs/594b94b4c3038f5c2cfb2f5d9c10ef30c7070a4c#darwin.builder
error: build of '/nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv' on 'ssh-ng://builder@localhost' failed: builder for '/nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv' failed with exit code 32;
       last 10 log lines:
       > copying path '/nix/store/65cvxfd36l9cawzj1gkx3zpzj0bdfg0b-unit-serial-getty-.service' to 'local'...
       > copying path '/nix/store/8izgxj2nzpwkd0hpm3r38zkiklm1jkzr-unit-nixos-activation.service' to 'local'...
       > copying path '/nix/store/82nc5jrzri2w4hw6rhffis9m6n25xnbv-unit-systemd-fsck-.service' to 'local'...
       > copying path '/nix/store/r9h4h2j299iz32gnnr2qksz2hsi6i8cw-unit-systemd-udevd.service' to 'local'...
       > copying path '/nix/store/klkbaapdbcs35bxyp0l97ic3lpp0njnb-user-units' to 'local'...
       > copying path '/nix/store/c04f008fdyqmarjl6nf35fdrzfm9h6lk-system-units' to 'local'...
       > copying path '/nix/store/786ah3bs0n5lcvjq2qhp4laiqkg60wr0-etc' to 'local'...
       > copying path '/nix/store/mj26i1zj0bjv1b732bafrg2ffly34adj-nixos-system-nixos-23.05pre-git' to 'local'...
       > mount: /build/root/build/root: must be superuser to use mount.
       >        dmesg(1) may have more information after failed mount system call.
       For full logs, run 'nix log /nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv'.
error: builder for '/nix/store/rnwbjdkq9wz43nlsmggr7ysn7k9h9z2w-nixos-disk-image.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/wr12pkszc3gj021fy08cch9syhqvwyj2-run-nixos-vm.drv' failed to build
error: 1 dependencies of derivation '/nix/store/p5pa1imyb5yc6l3wy1cda5bz38dc430f-nixos-vm.drv' failed to build
error: 1 dependencies of derivation '/nix/store/ld4rlr7sf599qi53bmd4d2xrjsm53n11-create-builder.drv' failed to build

https://github.com/NixOS/nixpkgs/pull/210812 also seems to have broken other builds - which triggered this emergency fix: https://github.com/NixOS/nixpkgs/pull/211218. Doesn't seem to have made it to the unstable branch yet though. https://nixpk.gs/pr-tracker.html?pr=211218

dhess commented 1 year ago

Is it possible to add NixOS modules to the darwin.builder's config? If so, it's not clear how.

I ask because I'd like to run this on our aarch64-darwin CI machines, but enable the NixOS Tailscale module to make each aarch64-darwin's darwin.builder available to our Hydra as a remote aarch64-linux builder, as well. (I believe this would also get around any host port 22 and/or macOS firewall issues.)

Gabriella439 commented 1 year ago

@dhess: Yes. If you copy the code from here:

https://github.com/NixOS/nixpkgs/blob/0cf78f562607f75256703af568814d3a62ded6de/pkgs/top-level/darwin-packages.nix#L215-L233

… then you can add in additional modules of your own

bergkvist commented 1 year ago

In case you need more than 3GB of memory for your builder

# Starts darwin.builder VM with 8GB RAM
QEMU_OPTS="-m 8192" nix run nixpkgs#darwin.builder

misuzu commented 1 year ago

Does qemu support passing rosetta binary to guest system? Would be awesome to enable https://github.com/NixOS/nixpkgs/pull/202847 to support building for x86_64-linux too.

YorikSar commented 1 year ago

@misuzu Rosetta "drive" is mounted via higher-level Virtualisation.framework while QEMU uses lower-level Hypervisor.framework. The later is more like kvm while the former is more like QEMU itself, so I doubt its support will happen in QEMU. I think, #5241 is relevant to bringing virtualisation backends other than QEMU for running NixOS VMs.

roberth commented 1 year ago

UTM does support it, is supposedly based on QEMU, is packaged, and we have a NixOS module to support it on its guests https://github.com/NixOS/nixpkgs/pull/202847. If it has a qemu compatible command line it might be close to a drop-in replacement.

5241

If it's like qemu, you can set qemu.package at the test level if you use the documented entrypoint.

YorikSar commented 1 year ago

@roberth UTM docs clearly state https://docs.getutm.app/advanced/rosetta/:

Rosetta allows you to run Intel Linux executables in an Apple Silicon Linux virtual machine (using Apple Virtualization backend).

QEMU VMs don't have this setting in UTM.

ElvishJerricco commented 1 year ago

@roberth IIUC, UTM only supports rosetta if you tell it to use Apple Virtualisation instead of QEMU.

willcohen commented 1 year ago

I love QEMU, but this is awesome. I still don't quiiite have copy/paste working this way (not sure if spice + wayland is a thing), but using Apple Virtualization solves the other major pain point of having normal-ish resolutions relative to my laptop and external monitor working with wayland/sway on a graphical NixOS VM. I feel like there should be a flashing exclamation point somewhere that this works so well.

roberth commented 1 year ago

supposedly based on

This was outdated information then. Thanks for correcting me.

I was thinking about the VM tests in my last comment. It'd be far easier to add Apple Virtualisation support if it's just for the darwin.builder, as it that's a simple entrypoint. I guess you'd have to figure out

how to invoke Apple Virtualization from the command line. Some apple command? UTM? Custom program?
create a new module with a new option that holds the Apple Virtualization invocation
- figure out which NixOS options hold the image and kernel
change the darwin.builder package to get that option instead of the qemu one

misuzu commented 1 year ago

how to invoke Apple Virtualization from the command line. Some apple command? UTM? Custom program?

Maybe this could be adapted to our use-case: https://github.com/Code-Hex/vz/tree/main/example/linux https://github.com/lima-vm/lima/pull/1155

Another approach might be to mount /Library/Apple/usr/libexec/oah/RosettaLinux into QEMU VM, but we have to figure out how to enable TCO mode.

ElvishJerricco commented 1 year ago

I do wish the kernel had support for enabling TSO mode per process. I don't even know if that's possible in a VM, but it'd be very nice for asahi linux to be able to use the rosetta binary via hacks and with TSO enabled.

YorikSar commented 1 year ago

There's a vmcli project: https://github.com/gyf304/vmcli#vmcli-1 - it allows to run VMs with Virtualisation.framework from CLI. We could use it to start VM and add Rosetta support to it.

fkorotkov commented 1 year ago

There is also Tart that has Rosetta support. One just need to:

brew install cirruslabs/cli/tart
tart clone ghcr.io/cirruslabs/ubuntu:20.04 ubuntu
tart run --rosetta="rosetta" ubuntu

And make sure Rosetta is configured inside the VM according to these docs.

dhess commented 1 year ago

I've gotten pretty far with Tart. Thanks to @fkorotkov for making me aware of it in the above comment!

Create a new, empty Virtualization.Framework VM named nixos with tart create nixos --linux
Build an aarch64-linux NixOS configuration as a raw-efi image using https://github.com/nix-community/nixos-generators. For bonus points, set virtualisation.rosetta.enable = true and nix.settings.extra-platforms = [ "x86_64-linux" ] in the VM's config, so that the VM will be able to run x86_64 Linux binaries, as well.
Overwrite the Tart VM's disk.img with the nixos.img file from step 2.
Run the VM with tart run nixos --rosetta rosetta

I've spent most of my time on this so far just learning how Virtualization.Framework works, and not much time experimenting with the running VM, but everything I've tried so far works great, including building and running x86_64-linux derivations. Additionally, I think it would be straightforward to automate steps 1-3 in a Nix derivation.

Even better would be to use the Mac's /nix/store in the VM. To experiment with this, I did the following:

Run the VM with tart run nixos --rosetta rosetta --dir="store:/nix/store"
In the running VM, run mount -t virtiofs com.apple.virtio-fs.automount /mnt. This mounts the Mac's /nix/store on /mnt/store in the running VM. I can see all the files there, though performance is a bit slow. Maybe mounting it with some cache= options would help.
Bind mount the virtiofs-mounted Mac Nix store to the VM's /nix/store via mount --bind /mnt/store /nix/store. (I think this extra step is necessary due to the way that Virtualization.Framework makes all shared filesystems available under a single mount point, but I might be missing something.) Confirm that everything still works.

So far, so good. However, my particular use case for this VM is to use it as an aarch64-linux remote builder, so I want to be able to build derivations in it, while re-using the Mac's own /nix/store for disk and network savings. Now I try to build something that's not in the store: nix run nixpkgs#hello. This fails with error: cannot open connection to remote store 'daemon': error: reading from file: Connection reset by peer

Looking at the logs, the problem is apparent: unexpected Nix daemon error: error: changing ownership of path '/nix/store': Operation not permitted

This makes sense: Nix is running in multi-user mode on my Mac, so my Mac's /nix/store is owned by root and managed by the local nix-daemon running as root. I'm running tart as my local macOS user, not root, so tart doesn't have permissions to change ownership on the Mac's /nix/store.

Anyway, besides that small catch, this route looks very promising. I think the main issue for our use case is that Virtualization.Framework doesn't support nested virtualization, so we won't be able to use this virtualized remote aarch64-linux builder to run NixOS tests.

Gabriella439 commented 1 year ago

@dhess: You probably don't want to share the host's /nix/store with the builder anyway, for the reason outlined in this comment:

https://github.com/NixOS/nixpkgs/blob/99563190d585896d7ddb43ea448a95574dfa2373/nixos/modules/profiles/macos-builder.nix#L159-L168

dhess commented 1 year ago

@Gabriella439 Thanks, I did see that comment linked from elsewhere, either in this discussion, or in a related one.

However, if I understand the comment correctly, I don't think it applies in our use case. We have several dedicated macOS remote builders (let's call them mac1, mac2, etc.), which appear in our x86_64-linux NixOS dedicated builder's /etc/nix/machines file; let's call that machine nixos1. mac1 etc. are not for interactive use: they only build derivations that are delegated by nixos1.

nixos1 cannot build aarch64-linux derivations, but virtualized aarch64-linux NixOS VMs running on mac1 etc. could. So what we'd like to do is have mac1 etc. run these aarch64-linux VMs (let's call them arm1 etc.), list those VMs in nixos1's /etc/nix/machines file, and then have nixos1 delegate aarch64-linux jobs to those VMs. Therefore, unless I'm missing something, the comment does not apply in our case, because the host is not delegating any jobs to the guest, only nixos1 is.

Perhaps it's possible that nixos1 could ask arm1 to build platform-independent derivation foo at the same time as it's asking mac1 to build it? But even in this case, I assume that whichever remote builder wins the race will go first while the other waits for the lock, then proceeds (and presumably sees that the derivation has magically appeared in the store in the meantime). In any case, I wouldn't expect a deadlock to occur in this scenario.

ghost commented 11 months ago

Over in this podman issue, I see that virtiofs in MacOS doesn't bypass open file limits, which leads to unexpected behavoir in the guest OS: https://github.com/containers/podman/issues/16106

I'm addressing the issue directly in podman, but I was hoping someone from this issue might have an opinion on if qemu should be bumping it's own ulimits up when it is acting as the virtiofs daemon?

Gabriella439 commented 8 months ago

You can now run NixOS tests on macOS, too. See: https://github.com/NixOS/nixpkgs/pull/282401/

Note that you still need a Linux builder to build the test VM, though

NixOS / nixpkgs