Allowing NixOS VM's to be run on macOS

infinisil commented 3 years ago

People using macOS currently can't interactively run NixOS VM's as described e.g. here, even with a remote Linux builder. In this issue I'm describing some ways in how that could be made to work. Note that speed is important as well, so kvm/hvf and co. should be used if possible.

Relevant issue is https://github.com/NixOS/nixpkgs/issues/64578. Ping @zupo @matthewbauer @nmattia @roberth. Any pointers/help with this is appreciated.

This issue is sponsored by Niteo :)

Using qemu directly (doesn't work)

With this change it's possible to build a NixOS VM run script that is executable on macOS (note that this requires a remote Linux builder)

$ nix-build '<nixpkgs/nixos>' \
  --argstr system x86_64-linux \
  --arg configuration '{ virtualisation.qemu.pkgs = import <nixpkgs> { system = "x86_64-darwin"; }; }' \
  -A vm \
  -I nixpkgs=https://github.com/infinisil/nixpkgs/archive/4d244410ee0f3e3ece5494533217bbafbd95d9b3.tar.gz
/nix/store/2a7cbyp9xp12ddc4lxb4h93dxa5yfndy-nixos-vm

However running the VM doesn't actually work:

$ result/bin/run-nixos-vm                                                          
qemu-system-x86_64: -virtfs local,path=/nix/store,security_model=none,mount_tag=store: There is no option group 'virtfs'
qemu-system-x86_64: -virtfs local,path=/nix/store,security_model=none,mount_tag=store: virtfs support is disabled

This is because qemu doesn't support virtfs on macOS.

It was suggested that this patch could be used to make it support virtfs. This was attempted here, however the build doesn't succeed:

$ nix-build https://github.com/infinisil/nixpkgs/archive/57562282ab34fc44a492f32a13939d77e29d0d9b.tar.gz -A qemu
[...]
fsdev/virtfs-proxy-helper.c:16:10: fatal error: 'sys/fsuid.h' file not found
#include <sys/fsuid.h>
         ^~~~~~~~~~~~~
1 error generated.
make: *** [/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/rules.mak:69: fsdev/virtfs-proxy-helper.o] Error 1
make: *** Waiting for unfinished jobs....
builder for '/nix/store/z8mkhjvaz3r7dixv94c1jprwx8gvl0gh-qemu-5.1.0.drv' failed with exit code 2
error: build of '/nix/store/z8mkhjvaz3r7dixv94c1jprwx8gvl0gh-qemu-5.1.0.drv' failed

Attempting to remove all the missing header files from virtfs-proxy-helper.c, including <sys/fsuid.h>, <sys/vfs.h>, <linux/fs.h> and <cap-ng.h> just leads to compilation errors, seemingly indicating that this is a Linux-only functionality.

This page however mentions that:

FSDRIVER: Either "local", "proxy" or "synth". This option specifies the filesystem driver backend to use. In short: you want to use "local". In detail:

local: Simply lets QEMU call the individual VFS functions (more or less) directly on host.

proxy: this driver was supposed to dispatch the VFS functions to be called from a separate process (by virtfs-proxy-helper), however the "proxy" driver is currently not considered to be production grade.

And indeed, the run-nixos-vm script only uses local, quoting the script:

-virtfs local,path=/nix/store,security_model=none,mount_tag=store \
-virtfs local,path=$TMPDIR/xchg,security_model=none,mount_tag=xchg \
-virtfs local,path=${SHARED_DIR:-$TMPDIR/xchg},security_model=none,mount_tag=shared \

Attempting to not compile that tool with this commit however also doesn't work, failing again on seemingly Linux-specific headers:

$ nix-build https://github.com/infinisil/nixpkgs/archive/5ed493ed6ae957aa510d5292eb373b8b1f4a3db1.tar.gz -A qemu
[...]
/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/fsdev/file-op-9p.h:19:10: fatal error: 'sys/vfs.h' file not found
#include <sys/vfs.h>
         ^~~~~~~~~~~
1 error generated.
make: *** [/private/var/folders/l6/7v9ppmg12q90382m246y3g5c0000gn/T/nix-build-qemu-5.1.0.drv-0/qemu-5.1.0/rules.mak:69: fsdev/qemu-fsdev.o] Error 1
builder for '/nix/store/zfnmaj3jh16x2qlryzyf3m2gy3zcww53-qemu-5.1.0.drv' failed with exit code 2
error: build of '/nix/store/zfnmaj3jh16x2qlryzyf3m2gy3zcww53-qemu-5.1.0.drv' failed

So it seems that qemu just doesn't support virtfs on macOS.

Using libvirt (might work)

Above qemu documentation also has this section, which describes how the same can be achieved with libvirt. While I believe that underneath it just uses qemu as well, there might be some additional libvirt magic happening. And it seems that macOS Mojave supports virtio-9p, in which post the author also uses libvirt successfully.

Currently the NixOS VM runner just passes arguments to qemu. In order to use libvirt we'll have to transform all these arguments to the libvirt equivalent in its XML configuration. There even is a libvirt page describing the equivalent of qemu arguments, which will be very useful. There is also the virsh domxml-from-native qemu-argv command which can supposedly do this transformation automatically, though I didn't have any success with it yet.

In above-linked document describing qemu argument equivalents, the -virtfs (or the -fsdev and -device options it's a shorthand for) option is notably missing. This should be replaced with a <filesystem> section as described in the qemu documentation link above.

It would be a good idea to first manually create a libvirt configuration and verifying that it works on a NixOS VM. libvirt has a graphical interface which could also be used.

Removing the need for filesystem mapping (might work)

The main reason this -virtfs argument is used at all is so that the guest machine in the VM can access the host machines /nix/store, which is where the whole system that the guest runs resides in.

An alternate approach however could be to create a /nix/store image that can be used by qemu as the /nix/store directly, therefore not depending on this filesystem mapping to the host.

See the various make-* files in https://github.com/NixOS/nixpkgs/tree/master/nixos/lib, which could be useful for this

zupo commented 3 years ago

Another benefit if we get MacOS + QEMU testing story working: ATM, running NixOS tests on various CI providers such as CircleCI and GitHub Actions is very slow since these services do not provide /dev/kvm/ on their Linux runners. But they do provide MacOS runners, that come with Mac's KVM alternative, so if MacOS + QEMU story is improved, even people that don't really care about MacOS, but do run CI in the cloud (looking at @domenkozar), stand to gain lots of speed improvements.

infinisil commented 3 years ago

I don't have a lot of time right now, but here's a basic skeleton that could be used to create a libvirt xml config for seeing if that approach is viable:

let
  pkgs = import <nixpkgs> {};
  config = (import <nixpkgs/nixos> {
    system = "x86_64-linux";
    configuration = <nixpkgs/nixos/modules/virtualisation/qemu-vm.nix>;
  }).config;
in pkgs.writeText "nixos.xml" ''
  <domain type='qemu' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    ...
    <os>
      ...
      <kernel>${config.system.build.toplevel}/kernel</kernel>
      ...
    </os>
    ...
    <filesystem type='mount' accessmode='$security_model'>
      <source dir='/nix/store'/>
      <target dir='store'/>
    </filesystem>
    ...
  </domain>
''

domenkozar commented 3 years ago

Refs https://github.com/NixOS/nixpkgs/issues/5241

domenkozar commented 3 years ago

virtio-9p can be slow. There is a faster replacement virtio-fs. However as far as I know, macOS doesn't have a driver for that.

Did you see https://passthroughpo.st/mac-os-adds-early-support-for-virtio-qemu/

infinisil commented 3 years ago

@domenkozar Yeah, click the "uncovered by Qemu developer" link there, it leads to a blogpost I linked above, which only mentions virtio-9p, not virtio-fs. I'm now just seeing that that news article mistakenly used virtio-fs in its picture..

domenkozar commented 3 years ago

Ah! That was confused me.

domenkozar commented 3 years ago

Benchmarks show that should also speed up the NixOS tests https://matrix.org/_matrix/media/r0/download/johnguant.com/qgAVstnDkCydsABgKmOXRPGW

cc @JJJollyjim do you have a branch for this?

Gaelan commented 3 years ago

The patch you linked seems to be only one of a series - presumably you need all of them for it to work. The series is visible here: https://lore.kernel.org/qemu-devel/cover.1529196703.git.keno@juliacomputing.com/. Not sure if there's a way to get the whole series from Patchwork.

JJJollyjim commented 3 years ago

Oh hey, just catching up here, I'm a little confused - the stuff where apple has added support for something is all about osx guests, not osx hosts which is what we care about here right?

I don't believe the current virtiofsd (host component) is able to run on osx (e.g. it uses namespaces to sandbox itself), but I don't think there is any technical limitation stopping it being ported?

I recommend patchew if you want a nice way to search for qemu patches and their status: https://patchew.org/QEMU/cover.1529196703.git.keno@juliacomputing.com/ (wish the kernel had something like this... -_-).

Anyway, let me know if you find that switching to virtio-fs will make OSX support work:

I have a branch, but haven't worked on cleaning it up and posting it because for some reason the DAX patches still haven't been submitted to qemu (last I checked), and without DAX the tiny performance improvement isn't worth the complexity. That calculus changes if it will fix OSX though :)

JJJollyjim commented 3 years ago

On second thoughts, I patch out the namespace stuff (so it works in the nix sandbox), so it's possible it will run on osx without any further changes. I don't have an osx machine to test on :(

Gaelan commented 3 years ago

I have a macOS machine I'm happy to test stuff on if it'd be helpful for you.

Oh hey, just catching up here, I'm a little confused - the stuff where apple has added support for something is all about osx guests, not osx hosts which is what we care about here right?

Not quite sure what you're referring to, but here's a quick run-down of Apple's recent-ish features for macOS hosts:

Hypervisor.framework: 3-5 years old. Provides low-level userspace access to the CPU's virtualization features. I think QEMU has a backend for this, called HVF.
Virtualization.framework: New with Big Sur, this year's release. Much higher-level - you give it a kernel, initrd, and disk, and it goes from there. Pretty limited - only emulates a few devices (the virtio devices for disk, network, serial, RNG and memory balloons), and only supports raw images (though APFS's sparse files negate the need for qcow2 a bit). vftool provides a thing command-line wrapper over this.

(As a side note, I've been experimenting with using Virtualization.framework to implement a simple Linux VM, using a tiny linux kernel with u-root as a simple "bootloader" to find and exec the real kernel image off a Linux filesystem; I've got it working with Ubuntu, but not NixOS; I suspect there's some issue with virtio-console support on the install ISO.)

JJJollyjim commented 3 years ago

@Gaelan I was referring to Domen's Passthru Post link about virtio :)

Gaelan commented 3 years ago

Ah, my bad.

mroi commented 3 years ago

I ported the original 9p patchset (mentioned earlier in this thread) to QEMU 5.2.0. In initial tests, 9p support on Darwin appears to be working.

But this is a large change that should probably go to upstream and not to Nixpkgs. If you want to try anyways: https://github.com/mroi/nixpkgs/commit/839559b486d4c4dbd911a0f13a3b2d319b942cd5

anthr76 commented 3 years ago

@mroi Would be help to at least submit the patch to Homebrew. Thanks for the work! I'll look on giving it a try.

SCOTT-HAMILTON commented 3 years ago

I would really love the test driver to run on MacOS with HAXM enabled. (-enable-hax)

I tried to merge both patches from @mroi and @Infinisil and ran the same command as above :

$ nix-build '<nixpkgs/nixos>' \
            --argstr system x86_64-linux \
            --arg configuration '{ virtualisation.qemu.pkgs = import <nixpkgs> { system = "x86_64-darwin"; };
}' \
            -A vm \
            -I nixpkgs=http://github.com/SCOTT-HAMILTON/NixPkgs/archive/230f823cd985e499ff2fd450b419bc5b44c6db
f8.tar.gz

But I get this error :

error: a 'x86_64-linux' with features {} is required to build '/nix/store/qbr4wjfz9lzrqha4gl6mjp2l0imwqdgh-append-initrd-secrets.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}

domenkozar commented 3 years ago

For testing patches you'll need a remote builder to build the Linux bits. The quickest way is to follow https://nix.dev/tutorials/continuous-integration-github-actions.html and get them from your binary cache.

domenkozar commented 3 years ago

Steps left:

make a PR for https://github.com/Infinisil/nixpkgs/commit/4d244410ee0f3e3ece5494533217bbafbd95d9b3
make a PR for https://github.com/mroi/nixpkgs/commit/839559b486d4c4dbd911a0f13a3b2d319b942cd5
build on linux & test on macOS

mroi commented 3 years ago

make a PR for mroi@839559b

You mean as a PR toward Nixpkgs or toward upstream QEMU? I can easily do the former. For the latter I would need to familiarise myself with their e-mail-based workflow. (Anyone here with experience in that?)

domenkozar commented 3 years ago

Former, although upstream patch would be really nice.

anthr76 commented 3 years ago

make a PR for mroi/nixpkgs@839559b

You mean as a PR toward Nixpkgs or toward upstream QEMU? I can easily do the former. For the latter I would need to familiarise myself with their e-mail-based workflow. (Anyone here with experience in that?)

The linux kernel has a bot explaining their email workflow which you might find similar with QEMU

https://github.com/torvalds/linux/pull/803#issuecomment-592644193

mroi commented 3 years ago

I updated the 9p patches for current QEMU 6.0: #122420

I’ll look into proposing this to upstream. (Thanks @anthr76 for the link.)

domenkozar commented 3 years ago

@r2r-dev did you manage to get it working?

r2r-dev commented 3 years ago

Yup. I've prepared 2 test branches:

Based mostly on stuff mentioned in this issue, as well as my fix for virtfs.
- https://github.com/r2r-dev/nixpkgs/commits/qemu-darwin-fixes can be tested via:
- https://github.com/r2r-dev/nixos-shell/commit/95b586bd5aef938a64ce2a9e9b61c99c9da936a2
With additional patches for M1, and vmnet device adapter:
- https://github.com/r2r-dev/nixpkgs/commits/wip-qemu-darwin-fixes can be tested via:
- https://github.com/r2r-dev/nixos-shell/commit/1f53a820f1570f9e75263e6a69103465e0cf2c46

Obviously, both of these branches aren't pretty and surely will need some cleanup.

Some remarks:

Qemu's v9fs requires additional patching (see 1st branch). Without it, any getdents or getdents64 syscalls on shared filesystem will result in a deadlock.
overlayfs won't work on top of v9fs mounts. It seems that v9fs cannot handle xattrs querying. This can be fixed by mounting lower-dir with -o version=9p2000.u. However, even though I managed to boot into VM, it was quite unstable. In order to mitigate that I used squashfs-based /nix/store instead of one mounted from host.

Refs: https://github.com/Mic92/nixos-shell/issues/16

pl-misuw commented 3 years ago

@r2r-dev This is far beyond cool, works well for my 2018 15' MBP (11.2.3 (20D91))! Kudos.

domenkozar commented 3 years ago

This is superb.

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-macos-monthly/12330/4

hexagonal-sun commented 2 years ago

@r2r-dev Hi, I'm giving this a try on a M1 machine and I'm getting the error:

# ./result/bin/nixos-shell examples/darwin.nix -- -- --argstr system aarch64-linux -I nixpkgs=http://github.com/r2r-dev/nixpkgs/archive/f68911645209742013dd6abe570a23548d9a5780.tar.gz --option builders-use-substitutes true

[...]

error: a 'x86_64-darwin' with features {} is required to build '/nix/store/zq54yx4gydijq5hryk60qs660pw84s5h-MacOSX-SDK-11.0.0.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test, recursive-nix}

If I change virtualisation.qemu.pkgs.system = "aarch64-darwin" in examples/darwin.nix I now get:

error: anonymous function at /nix/store/5s210gnpc6f7nn2jhnaakq8zm47p2dlz-source/pkgs/build-support/bintools-wrapper/default.nix:8:1 called without required argument 'lib'

       at /nix/store/5s210gnpc6f7nn2jhnaakq8zm47p2dlz-source/pkgs/stdenv/native/default.nix:133:18:

          132|       inherit lib nativePrefix;
          133|       bintools = import ../../build-support/bintools-wrapper {
             |                  ^
          134|         name = "bintools";
(use '--show-trace' to show detailed location information)

Any ideas on how to get this working on aarch64-darwin? Thanks!

willcohen commented 2 years ago

See https://github.com/NixOS/nixpkgs/pull/162243 for an updated PR with the upcoming QEMU functionality backported to v6.2.0.

willcohen commented 2 years ago

Looking back over this thread, what remains for nixpkgs? At this point, is it only the qemu-vm.nix stuff?

alyssais commented 2 years ago

I've just merged #162243. Is there anything else to be done here, or can this be closed now?

Artturin commented 2 years ago

Perhaps a test that can, for example, run on ofborg be created so that there are no regressions?

YorikSar commented 2 years ago

I'm trying to run NixOS VM on my macOS machine with M1. What's missing so far:

either QEMU 6.2 or virtfs doesn't work (see below)
we need to backport commit by @Infinisil that allows to build VM script for macOS (it applies quite cleanly)
still missing ability to detect whether to use kvm or hvf in nixos/lib/qemu-common.nix

Here's what I'm trying to build: flake.nix:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };

  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      qemu = nixpkgs-aarch.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          (import ./qemu.nix { inherit nixpkgs; })
        ];
      };
    };
  };
}

qemu.nix:

{ nixpkgs }:
{ config, pkgs, lib, modulesPath, ... }:

{
  imports = [
    "${modulesPath}/virtualisation/qemu-vm.nix"
  ];

  system.stateVersion = "21.11";

  networking.useDHCP = false;
  networking.interfaces.eth0.useDHCP = true;

  services.getty.autologinUser = "test";
  users.users.test = {
    isNormalUser = true;
    extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
    hashedPassword = "$5$ZjxV1oXdKi61yoVw$iBoQwq3URyEeN56FZWEouw4G1UJM0eGQMLm54hmvd/8"; # "test"
  };
  security.sudo.wheelNeedsPassword = false;

  virtualisation = {
    qemu.pkgs = nixpkgs.legacyPackages.aarch64-darwin;
    # These two disable 9p/virtfs entirely
    useNixStoreImage = true;
    sharedDirectories = lib.mkForce { };
    graphics = false;
  };
}

I build it like this:

point nixpkgs input at a repo with cherry-picked qemu.pkgs commit
nix build .#nixosConfigurations.qemu.config.system.build.vm
copy result/bin/run-nixos-vm and replace -enable-kvm -machine virt,gic-version=host -cpu host in qemu arguments with -machine virt,accel=hvf,highmem=off -cpu cortex-a72

I was trying this with previous version of nixpkgs with the qemu.pkgs patch, and it worked fine, I got autologin on console. After I've rebased that commit on top of latest nixpkgs master, I don't get any output on both text and graphical consoles, both with useNixStoreImage and sharedDirectories lines and without them.

I'll post both rebased qemu.pkgs commit and my example somewhere a bit later, and will keep trying.

ElvishJerricco commented 2 years ago

@YorikSar Looks like qemu 7.0 enables 9pfs for macOS: https://wiki.qemu.org/ChangeLog/7.0#9pfs

That should fix the virtfs stuff, right?

willcohen commented 2 years ago

@ElvishJerricco the feature you’re noting for 7.0 is what was backported to 6.2 via #162243!

YorikSar commented 2 years ago

With everything needed for HVF and 9p landed in QEMU master branch, I've posted a nixpkgs branch with the fixes I've accumulated so far. I'll be creating separate PRs for pieces of it. So far on my M1 MacBook Pro this flake:

{
  inputs = {
    nixpkgs.url = "github:YorikSar/nixpkgs/vm-on-darwin";
  };

  outputs = { self, nixpkgs }: {
    nixosModules = {
      vm =
        { config, pkgs, lib, modulesPath, ... }:

        {
          imports = [
            "${modulesPath}/virtualisation/qemu-vm.nix"
          ];

          system.stateVersion = "22.05";

          networking.useDHCP = false;
          networking.interfaces.eth0.useDHCP = true;

          services.getty.autologinUser = "test";
          users.users.test = {
            isNormalUser = true;
            extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
            hashedPassword = "$5$ZjxV1oXdKi61yoVw$iBoQwq3URyEeN56FZWEouw4G1UJM0eGQMLm54hmvd/8"; # "test"
          };
          security.sudo.wheelNeedsPassword = false;

          virtualisation.graphics = false;
        };
    };
    nixosConfigurations = {
      vm-x86_64 = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          self.nixosModules.vm
          {
            virtualisation.qemu.pkgs = nixpkgs.legacyPackages.x86_64-darwin;
          }
        ];
      };
      vm-aarch64 = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          self.nixosModules.vm
          {
            virtualisation.qemu.pkgs = nixpkgs.legacyPackages.aarch64-darwin;
          }
        ];
      };
    };
    packages.x86_64-darwin.default = self.nixosConfigurations.vm-x86_64.config.system.build.vm;
    packages.aarch64-darwin.default = self.nixosConfigurations.vm-aarch64.config.system.build.vm;
  };
}

On aarch64-darwin nix run runs VM with console output, HVF acceleration and /nix/store mounted from the host. It's crazy fast (after QEMU gets compiled, of course) and nice. I'll find an Intel Mac somewhere to test it there as well.

As a next step I want to make nixos-rebuild build-vm build VM start script with the current system in mind, so that you can always run it, whether you are on macOS or Linux.

zupo commented 2 years ago

@YorikSar: I have an Intel-based MacBook 13" that is just sitting on my desk. I'd love to help testing this! I'm a bit of a noob so need some handholding though. Alternatively, I can give you SSH access?

willcohen commented 2 years ago

My intel mac is having a slow day so it's going to take a while for it all to build, but I'm trying nix run on this flake now

willcohen commented 2 years ago

nix run on an intel mac with this flake fails with:

error: a 'x86_64-linux' with features {} is required to build '/nix/store/89zr2gg8dd8mw8x5nq33fyy65imsc2qv-append-initrd-secrets.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}.

It earlier also failed with the same error but for stdenv-linux, but rerunning nix run a few times settled on initrd-secrets.

YorikSar commented 2 years ago

@willcohen Oh, yeah, to build this you need to have a Linux builder configured for your Nix daemon. I actually have built everything for this already, now copying everything to an Intel MacBook with very old Nix setup, hoping that it'll be able to run this...

@zupo I'll try this out first on a laptop that I have nearby. I'll reach out to you if it won't be enough.

willcohen commented 2 years ago

Of course, duh. I unfortunately don't have a linux builder set up! If no one else is able to find one I can get one set up but it probably won't be until next week...

willcohen commented 2 years ago

Once this shakes out for an actual remote builder, it does seem like -- if it's possible -- being able to create an auto-generated little QEMU VM that can be sshed into as a local-ish remote builder might be an important additional step. My little hand-rolled QEMU NixOS VM involved a fair bit of guess-and-checking BIOS settings in configuration.nix to be compatible with QEMU and partitioning just so, but it works okay. Is there a fundamental reason that the builders themselves can't be virtual machines running locally?

YorikSar commented 2 years ago

I've tested it on Intel, it seems like I've messed up QEMU arguments and also there's some problem with HVF there. I wonder if old macOS SDK on Intel is to blame. I will keep figuring this out.

By the way, one of the outcomes of this project that I see would be some way to have a local Linux builder on the Mac without any hassle. Something like a VM that is automatically spun up when the builder is needed.

willcohen commented 2 years ago

This is how I invoke it on my x86 mac (using nix-built QEMU, so hopefully SDK isn't the issue!): qemu-system-x86_64 -machine type=q35,accel=hvf -smp 2 -hda nixos.qcow2 -m 4G -vga virtio -usb -device usb-tablet -display default,show-cursor=on,gl=off -virtfs local,path=/path/to/host/dir,mount_tag=foo,security_model=mapped-xattr

YorikSar commented 2 years ago

type=q35 instead of virt did the trick! Now it booted into the system, just like on M1 :) I'll check out what does q35 actually mean later. For now I've updated the commit and uploading (x86 virtualised build is painfully slow) my build artefacts to Cachix. You should be able to run nix run nixpkgs#cachix -- use yoriksar-gh to pull them during nix build.

ElvishJerricco commented 2 years ago

@YorikSar Tried to give it a go. Got a hash mismatch on the 9p patchfile:

error: hash mismatch in fixed-output derivation '/nix/store/5l7yvn3kgsinggwpn6bb40wxxbvb7d0g-f5643914a9e8f79c606a76e6a9d7ea82a3fc3e65.patch.drv':
         specified: sha256-VBcvy+t6UAQeUe4hqytBJIORKoe9auAmiVur9ZLUIhs=
            got:    sha256-8i13wU135h+YxoXFtkXweBN3hMslpWoNoeQ7Ydmn3V4=

YorikSar commented 2 years ago

@ElvishJerricco I've fixed the hash and rebased the branch on the current master. Please check again, don't forget to update flake inputs.

khaled commented 2 years ago

By the way, one of the outcomes of this project that I see would be some way to have a local Linux builder on the Mac without any hassle. Something like a VM that is automatically spun up when the builder is needed.

@YorikSar Do you have an outline of how this could work? Today it seems like you need a remote Linux builder to build the builder image itself, unless it has already been built and cached somewhere...

roberth commented 2 years ago

unless it has already been built and cached somewhere...

That would be hydra.nixos.org. If we add a system configuration to the relevant channels, as channel blockers, we can pretty much guarantee an "image" to be available. Scare quotes because we could use 9p for the system store, just like we do in the tests. The work of configuring the VM is best done through a nix-darwin module. Or perhaps two modules. One for bootstrapping a lightweight builder, and a second one for configuring a VM with arbitrary configuration, that can also serve as a linux builder.

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-32/19865/1

NixOS / nixpkgs