Enable Nested Virtualization on Mac OS 15

[aparashk]

Continuation of #3405

Mac's Hypervizor.Framework supports nested virtualization starting with Mac OS 15.0 (Sequoia), if the host CPU is M3 or later. We need this to be also supported in UTM, so that UTM Linux VMs can make use of KVM.

[andmpel]

https://developer.apple.com/documentation/virtualization/vzgenericplatformconfiguration/4360552-isnestedvirtualizationenabled?changes=latest_minor

Relevant apple documentation

[andriysavin]

So what about nested virtualization for Windows guest? Low level hypervisor API seems to support it as well https://developer.apple.com/documentation/hypervisor/4359587-hv_vm_config_set_el2_enabled

[osy]

For QEMU, we typically wait for the feature to be implemented upstream or we pick a patch. I think enabling nested virt requires doing more than just setting that flag.

[neverpanic]

This doesn't seem to work for me. I'm on 15.1 (24B83), M3 Pro, UTM 4.6.0 (102), with a freshly created Fedora 40 VM using Hypervisor.framework, and it doesn't seem to enable KVM support on boot:

root@localhost:~# dmesg | grep 'kvm'
[    0.050577] kvm [1]: HYP mode not available
root@localhost:~# dmesg | grep 'CPU:'
[    0.026095] CPU: All CPU(s) started at EL1
root@localhost:~# dmesg | grep 'CPU features:'
[    0.000000] CPU features: detected: Address authentication (IMP DEF algorithm)
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Spectre-v4
[    0.026107] CPU features: detected: ARMv8.4 Translation Table Level
[    0.026107] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.026108] CPU features: detected: Common not Private translations
[    0.026108] CPU features: detected: CRC32 instructions
[    0.026108] CPU features: detected: Data cache clean to Point of Deep Persistence
[    0.026109] CPU features: detected: Data cache clean to Point of Persistence
[    0.026109] CPU features: detected: Data independent timing control (DIT)
[    0.026109] CPU features: detected: E0PD
[    0.026110] CPU features: detected: Enhanced Privileged Access Never
[    0.026110] CPU features: detected: Generic authentication (IMP DEF algorithm)
[    0.026111] CPU features: detected: RCpc load-acquire (LDAPR)
[    0.026111] CPU features: detected: LSE atomic instructions
[    0.026111] CPU features: detected: Privileged Access Never
[    0.026112] CPU features: detected: RAS Extension Support
[    0.026112] CPU features: detected: Speculation barrier (SB)
[    0.026112] CPU features: detected: TLB range maintenance instructions

Maybe my expectations are wrong, but I would have expected the CPU to be started in EL2 and HYP mode to be available. Am I doing things wrong? Is there a debug log I could check to make sure nested virtualization is actually enabled on this VM?

[neverpanic]

Nevermind, works just fine with Fedora 41, so I'm guessing the issue is on the VM client side. For reference, this is what it looks like when it works as expected:

root@localhost:~# dmesg | grep 'kvm'
[    0.105119] kvm [1]: nv: 529 coarse grained trap handlers
[    0.105154] kvm [1]: IPA Size Limit: 40 bits
[    0.105671] kvm [1]: GICv3: no GICV resource entry
[    0.105674] kvm [1]: disabling GICv2 emulation
[    0.105697] kvm [1]: GIC system register CPU interface enabled
[    0.105705] kvm [1]: vgic interrupt IRQ9
[    0.105753] kvm [1]: Hyp nVHE mode initialized successfully
root@localhost:~# dmesg | grep 'CPU:'
[    0.042219] CPU: All CPU(s) started at EL2

[chodorenko]

At the same not work MacOS 15.1 Sequoia (24B83) MacBook Air M3 UTM 4.6.0 (102) Clean install Fedora Server 41 aarch64 from Fedora-Server-netinst-aarch64-41-1.4.iso Maybe i do not correct understand howe to enable nested virtualization ? config file config.plist.zip

[sidkshatriya]

@neverpanic how did you get this to work ? You simply tried with Fedora 41 instead of Fedora 40 ? Or did you make some other configuration changes in UTM ?

[sidkshatriya]

@neverpanic My guess is this worked because you checked the checkbox "Use Apple Virtualization" when creating a new virtual machine ? (This way we don't need to worry about qemu's support for nested virtualization)

[neverpanic]

Yes, this only works with Apple Virtualization, as documented in the release notes.

However, this also didn't work reliably for me. I had to create multiple VMs, and it worked on some of them. Once it started working on a VM, it generally kept working on that particular VM, though — I just ended up creating multiple, testing whether nested virtualization worked, and then overwriting the disk with the one I wanted to be attached.

[sidkshatriya]

However, this also didn't work reliably for me.

Thanks for your reply -- Very mysterious indeed. Not able to get this to work at all. Not sure what caused your VM to work suddenly. Maybe only P cores support nested virtualisation ? Maybe addition of certain devices / displays causes nested virtualisation to get disabled ? I tried with various configuration variations of Fedora Server 41 without any luck !

[chodorenko]

Yes, this only works with Apple Virtualization, as documented in the release notes.

However, this also didn't work reliably for me. I had to create multiple VMs, and it worked on some of them. Once it started working on a VM, it generally kept working on that particular VM, though — I just ended up creating multiple, testing whether nested virtualization worked, and then overwriting the disk with the one I wanted to be attached.

You can give your config.plist from VM which Nested Virtualization work ? Or maybe can you find diff which my config from post https://github.com/utmapp/UTM/issues/6700#issuecomment-2458508936 ?

[chodorenko]

The difference is in this block

        <key>GenericPlatform</key>
        <dict>
            <key>machineIdentifier</key>
            <data>
            YnBsaXN0MDDRAQJUVVVJRE8QEDeWlFjsnkhKlOlHjO7qQRsICxAA
            AAAAAAABAQAAAAAAAAADAAAAAAAAAAAAAAAAAAAAIw==
            </data>
        </dict>

in my config its block not present. After i add Your block and Nested Virtualization work on vm @osy May be You can answer why its block do not generated on wizard of create wm ? May be need add some option in wizard ?

[sidkshatriya]

Good news ! I was able to get nested virtualization to work in UTM by creating a VM that:

• Uses "Apple Virtualization" in UTM as discussed above
• Choosing "Boot from kernel image"
• Subsequently passing in an uncompressed kernel image, initrd and raw disk

This way I was able boot into EL2 (Hypervisor mode) reliably. After that I was able to run a nested VM successfully using KVM. (Note that you will need to make sure that the linux user has access to the /dev/kvm device etc. i.e. the usual things one needs to run a VM in Linux)

Basically booting into a VM that boots into a bios/UEFI before the Linux kernel (which is the usual scenario) makes booting into EL2 more difficult it seems. By explicitly telling UTM to boot with a kernel image my guess is that hardware initialization happens properly so that you're dropped into EL2.

Where can you get a Linux kernel and initrd ? Try the various cloud images provided by various vendors. Some of them should provide you with a kernel and initrd separately or you should be able to extract it.

I got this working with the Ubuntu Cloud image for 24.10 . (I needed to create an extra cloud init iso file which sets things up for the user like password etc. This iso I attached as an additional raw disk in UTM)

---

$ sudo dmesg | grep kvm
[    0.086810] kvm [1]: nv: 529 coarse grained trap handlers
[    0.086843] kvm [1]: IPA Size Limit: 40 bits
[    0.087435] kvm [1]: GICv3: no GICV resource entry
[    0.087437] kvm [1]: disabling GICv2 emulation
[    0.087532] kvm [1]: GIC system register CPU interface enabled
[    0.087565] kvm [1]: vgic interrupt IRQ9
[    0.087737] kvm [1]: Hyp nVHE mode initialized successfully
$ sudo dmesg | grep EL2
[    0.028073] CPU: All CPU(s) started at EL2