qemu: vmnet networking needs elevated permissions on darwin

[layday]

Describe the bug Qemu 6.0.0 builds fail on darwin because the codesign executable is not available in the build environment. You can observe this on Hydra: https://hydra.nixos.org/build/142456539/nixlog/1

To Reproduce Steps to reproduce the behavior:

1. nix-env -i qemu

Notify maintainers @eelco @alyssais

Metadata

• system: "x86_64-darwin"
• host os: Darwin 20.4.0, macOS 10.16
• multi-user?: no
• sandbox: no
• version: nix-env (Nix) 2.3.10
• channels(layday): "nixpkgs-21.05pre287094.c5173fe2085"
• nixpkgs: /Users/layday/.nix-defexpr/channels/nixpkgs

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[mroi]

I have worked around this issue locally by adding this substitution to preConfigure:

    substituteInPlace meson.build \
      --replace 'if exe_sign' 'if false'

You can test this workaround with an overlay:

nix-shell -p '(import <nixpkgs> { overlays = [ (self: super: { qemu = super.qemu.overrideAttrs (attrs: { preConfigure = attrs.preConfigure + "substituteInPlace meson.build --replace \'if exe_sign\' \'if false\'"; }); }) ]; }).qemu'

I do not know whether there is an official strategy for dealing with code signing on Darwin, which should inform a proper fix. Especially on Darwin/ARM signing seems to be an increasing necessity.

[domenkozar]

cc @thefloweringash

[stephank]

The upstream commit is: https://gitlab.com/qemu-project/qemu/-/commit/8a74ce618b647e71ac703b5fe5dba3a2329a4d76

I believe this is an aarch64-darwin thing? I'm not sure exactly how things work there, but it looks like we sign everything with sigtool and that doesn't (yet) understand --entitlements. (I'm not sure if entitlements even work for ad-hoc signatures? They do. The upstream codesign comand is actually adding an ad-hoc signature as well.)

Would it be a good idea, for now, to move ahead with @mroi's fix, with a TODO comment? I verified it works with -accel hvf on x86_64-darwin.

(I think marking qemu entirely broken on aarch64-darwin might be a bit too much. HVF is a very important mode of operation, but not the only one.)

[stephank]

I went ahead and created a PR for this: https://github.com/NixOS/nixpkgs/pull/123872

[layday]

Do I understand correctly that #123872 disables the hypervisor? The performance impact of that would be severe. Can this issue be reopened if that's the case?

[mroi]

It seems to work with hypervisor acceleration on Darwin/x86. I cannot test Darwin/ARM, but since code signing is handled more strictly there, I can imagine that this breaks HVF.

If that is the case, I would agree to reopen this issue until code signing with entitlements is available in Nixpkgs.

[layday]

Okay, I've tried it on my x86 machine and it does indeed work without the entitlements. Thank you for providing a workaround. If it doesn't work on M1, then it might still be a good idea to keep this issue open.

[lloeki]

This is what happens on M1:

$ qemu-system-aarch64 -accel hvf -m 2048 -cpu cortex-a57 -M virt,highmem=off -drive file=/opt/qemu-5.2.0/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=ovmf_vars.fd,if=pflash,format=raw -serial telnet::4444,server,nowait -drive if=none,file=disk.qcow2,format=qcow2,id=hd0 -device virtio-blk-device,drive=hd0,serial="dummyserial" -device virtio-net-device,netdev=net0 -netdev user,id=net0 -vga none -device ramfb -cdrom ubuntu-20.04.2-live-server-arm64.iso -device usb-ehci -device usb-kbd -device usb-mouse -usb -monitor stdio                    

qemu-system-aarch64: -accel hvf: invalid accelerator hvf
$ qemu-system-aarch64 -accel help 
Accelerators supported in QEMU binary:
tcg

But of course while tcg works, it's ungodly slow when compared to near-native speed of using hvf.

[lloeki]

Of note: I did my own non-nix build of qemu 5.2 to test lima because I could not get HVF to operate on 6.0. UTM has a properly signed and HVF enabled qemu 6.0 though, so it's possible but I just could not find what makes it fail on my 6.0 as I build it the exact same way as 5.2:

$ /opt/qemu-5.2.0/bin/qemu-system-aarch64 -accel help
Accelerators supported in QEMU binary:
hvf
tcg
$ /opt/qemu-6.0.0/bin/qemu-system-aarch64 -accel help
Accelerators supported in QEMU binary:
tcg

In both cases as mentioned here I had to re-sign with the entitlements for some reason:

For some reason, the qemu binary is modified during make install. You need to resign it with the correct entitlements, otherwise you will get an Unknown Error:

sudo codesign --entitlements /path/to/qemu/accel/hvf/entitlements.plist --force -s - `which qemu-system-aarch64`

[mstone]

Folks,

In case it helps you find a way forward here, following these and other instructions that I found online, I managed to piece together how to use @thefloweringash's unreleased sigtool entitlements branch to add the required hypervisor entitlement as part of (in my case, a nix flake-based) packaging of qemu 6.1.0.

Patches are at https://github.com/mstone/qemu-m1 if you want to take a look, though they can certainly be cleaned up further -- apologies in advance.

Like @mroi, I also disable qemu's meson-based codesign scripting since nixpkgs has gone with sigtool-based code signing instead, although here since I've already made a separate repo for applying Alex Graf's patches, I just chose to tack one more patch on to this end instead of handling things at the nix level.

[lloeki]

While we're talking entitlements, I'd like to encourage to make sure the networking entitlements are in good order as well, especially, and if at all possible, the bridged networking one.

[mstone]

@lloeki -- I am unfortunately not yet familiar with the networking entitlements you're referring to. Can you please share some more details of what you're thinking of? Thanks!

[lloeki]

@mstone (caveat: most of this is from the top of my head, I looked into this a couple of months ago, hit the entitlement issue for networking, and now my memory of it is a bit hazy)

IIRC there's com.apple.security.hypervisor and com.apple.vm.networking, although I seem to recall another entitlement but I can't find it.

QEMU by default uses slirp, which is entirely userland and virtual at the TCP/UDP level, where the qemu process is the one opening sockets. If you want real networking (ICMP, multicast, WOL, and other non-TCP ingress to the VM from the host or network...) you need to use something like tuntaposx.

But since kexts don't work in most cases now (at least not without jumping through hoops that include disabling SIP through a couple of reboots to recovery) tun/tap is now largely unavailable (and due to slowly die out) so one has to go through the macos facilities to create similar interfaces for the VMs with the new first-party API. If one wants to achieve that some entitlements are needed.

Without the first one I think hvf won't be available on arm even when compiled, and it also provides the ability to create a (virtio?) NAT or host-only interface (that's where my memory fails me about the unfindable entitlement I seem to recall having seen). The second one is to be able to create a similar bridged interface to the LAN, and apparently to obtain it it needs approval from Apple and an updated dev cert with the good bits flipped to be issued.

At some point during 5.x or 6.0 there was a set of patches proposed to add support for these macos facilities. Without the proper entitlements that would not work. I'm not sure if that was merged for 6.1 though but even then that will come at some point.

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_hypervisor

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_vm_networking

[lloeki]

Here's the patchset on qemu-devel to use vmnet.framework I mentioned.

There are also more FAQ-style details over there, at a project whose intent is to handle the vmnet.framework part out of the qemu process (as the patchiest aims to include), so that it doesn't need the entitlements nor being root (when the binary is not signed nor has com.apple.vm.networking).

[stale[bot]]

I marked this as stale due to inactivity. → More info

[n8henrie]

@layday @lloeki Is this still an issue? If so is there a different way for me to test? I'm tinkering with MacOS codesigning and entitlements issues in other projects and would be happy to try to help.

$ /run/current-system/sw/bin/qemu-system-aarch64 -accel help
Accelerators supported in QEMU binary:
hvf
tcg

$ nix run nixpkgs#nix-info -- -m
 - system: `"aarch64-darwin"`
 - host os: `Darwin 22.4.0, macOS 13.3.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.13.3`
 - channels(n8henrie): `"darwin, nixpkgs-22.11"`
 - channels(root): `""`
 - nixpkgs: `/Users/n8henrie/.nix-defexpr/channels/nixpkgs`

[mstone]

@n8henrie - my impression is that qemu from nixpkgs now has working HVF support on aarch64-darwin due to the sigtool fixes that @thefloweringash and I collaborated on.

Beyond that I am unaware of anyone pursuing the fancier networking entitlements that @lloeki requested.

If this accurately describes the overall state of things then my only further suggestion is that, documentation-wise, either this issue should be marked fixed and a new one opened for those remaining improvements or that this issue be left open (and possibly retitled) to better reflect the remaining work that could reasonably be done here.

[n8henrie]

It doesn't look like the qemu team has figured out those entitlements: https://gitlab.com/qemu-project/qemu/-/issues/1364

nor the homebrew team:

$ codesign -dvv --entitlements - /opt/homebrew/bin/qemu-system-aarch64
Executable=/opt/homebrew/Cellar/qemu/8.0.0/bin/qemu-system-aarch64
Identifier=qemu-system-aarch64-555549440574b530fe373e57b656e842b60b5db4
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=145333 flags=0x2(adhoc) hashes=4530+7 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12
[Dict]
        [Key] com.apple.security.hypervisor
        [Value]
                [Bool] true

It looks like the processes to gain this entitlement is... not straightforward: https://developer.apple.com/forums/thread/656411

seems to be corroborated by the UTM codebase, whose workaround includes a helper app and some kind of Apple-supplied provisioning profile:

• https://github.com/utmapp/UTM/blob/a0b0ed8fed513a13768b82d3c2fac372fc4c3709/CodeSigning.xcconfig.sample#L41
• https://github.com/utmapp/UTM/issues/3215

So my humble opinion would be to close this as an upstream issue.

[lloeki]

If this accurately describes the overall state of things

Looks like so, HVF does appear to work now.

then my only further suggestion is that, documentation-wise, either this issue should be marked fixed and a new one opened for those remaining improvements

That sounds like a very sensible move to me.

It looks like the processes to gain this entitlement is... not straightforward [...] So my humble opinion would be to close this as an upstream issue.

Agreed, definitely not straightforward. Also OK with that action, either with a separate issue left open (which might linger on for a while) or a documentation note somewhere?

Thanks for having taken a look into this!