utmapp / UTM

Virtual machines for iOS and macOS
https://getutm.app
Apache License 2.0
27.15k stars 1.34k forks source link

Enable Nested Virtualization on Mac OS 15 #6700

Closed aparashk closed 1 month ago

aparashk commented 1 month ago

Continuation of #3405

Mac's Hypervizor.Framework supports nested virtualization starting with Mac OS 15.0 (Sequoia), if the host CPU is M3 or later. We need this to be also supported in UTM, so that UTM Linux VMs can make use of KVM.

andmpel commented 1 month ago

https://developer.apple.com/documentation/virtualization/vzgenericplatformconfiguration/4360552-isnestedvirtualizationenabled?changes=latest_minor

Relevant apple documentation

andriysavin commented 1 month ago

So what about nested virtualization for Windows guest? Low level hypervisor API seems to support it as well https://developer.apple.com/documentation/hypervisor/4359587-hv_vm_config_set_el2_enabled

osy commented 1 month ago

For QEMU, we typically wait for the feature to be implemented upstream or we pick a patch. I think enabling nested virt requires doing more than just setting that flag.

neverpanic commented 3 weeks ago

This doesn't seem to work for me. I'm on 15.1 (24B83), M3 Pro, UTM 4.6.0 (102), with a freshly created Fedora 40 VM using Hypervisor.framework, and it doesn't seem to enable KVM support on boot:

root@localhost:~# dmesg | grep 'kvm'
[    0.050577] kvm [1]: HYP mode not available
root@localhost:~# dmesg | grep 'CPU:'
[    0.026095] CPU: All CPU(s) started at EL1
root@localhost:~# dmesg | grep 'CPU features:'
[    0.000000] CPU features: detected: Address authentication (IMP DEF algorithm)
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Spectre-v4
[    0.026107] CPU features: detected: ARMv8.4 Translation Table Level
[    0.026107] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.026108] CPU features: detected: Common not Private translations
[    0.026108] CPU features: detected: CRC32 instructions
[    0.026108] CPU features: detected: Data cache clean to Point of Deep Persistence
[    0.026109] CPU features: detected: Data cache clean to Point of Persistence
[    0.026109] CPU features: detected: Data independent timing control (DIT)
[    0.026109] CPU features: detected: E0PD
[    0.026110] CPU features: detected: Enhanced Privileged Access Never
[    0.026110] CPU features: detected: Generic authentication (IMP DEF algorithm)
[    0.026111] CPU features: detected: RCpc load-acquire (LDAPR)
[    0.026111] CPU features: detected: LSE atomic instructions
[    0.026111] CPU features: detected: Privileged Access Never
[    0.026112] CPU features: detected: RAS Extension Support
[    0.026112] CPU features: detected: Speculation barrier (SB)
[    0.026112] CPU features: detected: TLB range maintenance instructions

Maybe my expectations are wrong, but I would have expected the CPU to be started in EL2 and HYP mode to be available. Am I doing things wrong? Is there a debug log I could check to make sure nested virtualization is actually enabled on this VM?

neverpanic commented 3 weeks ago

Nevermind, works just fine with Fedora 41, so I'm guessing the issue is on the VM client side. For reference, this is what it looks like when it works as expected:

root@localhost:~# dmesg | grep 'kvm'
[    0.105119] kvm [1]: nv: 529 coarse grained trap handlers
[    0.105154] kvm [1]: IPA Size Limit: 40 bits
[    0.105671] kvm [1]: GICv3: no GICV resource entry
[    0.105674] kvm [1]: disabling GICv2 emulation
[    0.105697] kvm [1]: GIC system register CPU interface enabled
[    0.105705] kvm [1]: vgic interrupt IRQ9
[    0.105753] kvm [1]: Hyp nVHE mode initialized successfully
root@localhost:~# dmesg | grep 'CPU:'
[    0.042219] CPU: All CPU(s) started at EL2
chodorenko commented 2 weeks ago

At the same not work MacOS 15.1 Sequoia (24B83) MacBook Air M3 UTM 4.6.0 (102) Clean install Fedora Server 41 aarch64 from Fedora-Server-netinst-aarch64-41-1.4.iso Maybe i do not correct understand howe to enable nested virtualization ? config file config.plist.zip

Screen Fedora 41 aarch64
sidkshatriya commented 1 week ago

@neverpanic how did you get this to work ? You simply tried with Fedora 41 instead of Fedora 40 ? Or did you make some other configuration changes in UTM ?

sidkshatriya commented 1 week ago

@neverpanic My guess is this worked because you checked the checkbox "Use Apple Virtualization" when creating a new virtual machine ? (This way we don't need to worry about qemu's support for nested virtualization)

Screenshot 2024-11-11 at 4 16 39 PM
neverpanic commented 1 week ago

Yes, this only works with Apple Virtualization, as documented in the release notes.

However, this also didn't work reliably for me. I had to create multiple VMs, and it worked on some of them. Once it started working on a VM, it generally kept working on that particular VM, though — I just ended up creating multiple, testing whether nested virtualization worked, and then overwriting the disk with the one I wanted to be attached.

sidkshatriya commented 1 week ago

However, this also didn't work reliably for me.

Thanks for your reply -- Very mysterious indeed. Not able to get this to work at all. Not sure what caused your VM to work suddenly. Maybe only P cores support nested virtualisation ? Maybe addition of certain devices / displays causes nested virtualisation to get disabled ? I tried with various configuration variations of Fedora Server 41 without any luck !

chodorenko commented 1 week ago

Yes, this only works with Apple Virtualization, as documented in the release notes.

However, this also didn't work reliably for me. I had to create multiple VMs, and it worked on some of them. Once it started working on a VM, it generally kept working on that particular VM, though — I just ended up creating multiple, testing whether nested virtualization worked, and then overwriting the disk with the one I wanted to be attached.

You can give your config.plist from VM which Nested Virtualization work ? Or maybe can you find diff which my config from post https://github.com/utmapp/UTM/issues/6700#issuecomment-2458508936 ?

chodorenko commented 1 week ago

The difference is in this block

        <key>GenericPlatform</key>
        <dict>
            <key>machineIdentifier</key>
            <data>
            YnBsaXN0MDDRAQJUVVVJRE8QEDeWlFjsnkhKlOlHjO7qQRsICxAA
            AAAAAAABAQAAAAAAAAADAAAAAAAAAAAAAAAAAAAAIw==
            </data>
        </dict>

in my config its block not present. After i add Your block and Nested Virtualization work on vm @osy May be You can answer why its block do not generated on wizard of create wm ? May be need add some option in wizard ?

sidkshatriya commented 1 week ago

Good news ! I was able to get nested virtualization to work in UTM by creating a VM that:

This way I was able boot into EL2 (Hypervisor mode) reliably. After that I was able to run a nested VM successfully using KVM. (Note that you will need to make sure that the linux user has access to the /dev/kvm device etc. i.e. the usual things one needs to run a VM in Linux)

Basically booting into a VM that boots into a bios/UEFI before the Linux kernel (which is the usual scenario) makes booting into EL2 more difficult it seems. By explicitly telling UTM to boot with a kernel image my guess is that hardware initialization happens properly so that you're dropped into EL2.

Where can you get a Linux kernel and initrd ? Try the various cloud images provided by various vendors. Some of them should provide you with a kernel and initrd separately or you should be able to extract it.

I got this working with the Ubuntu Cloud image for 24.10 . (I needed to create an extra cloud init iso file which sets things up for the user like password etc. This iso I attached as an additional raw disk in UTM)


screenshot
$ sudo dmesg | grep kvm
[    0.086810] kvm [1]: nv: 529 coarse grained trap handlers
[    0.086843] kvm [1]: IPA Size Limit: 40 bits
[    0.087435] kvm [1]: GICv3: no GICV resource entry
[    0.087437] kvm [1]: disabling GICv2 emulation
[    0.087532] kvm [1]: GIC system register CPU interface enabled
[    0.087565] kvm [1]: vgic interrupt IRQ9
[    0.087737] kvm [1]: Hyp nVHE mode initialized successfully
$ sudo dmesg | grep EL2
[    0.028073] CPU: All CPU(s) started at EL2
hooji commented 6 days ago

I'm seeing a similar issue with nested virtualization trying to run docker inside a macOS VM.

This is on an M4 Pro new Mac Mini.

Docker reports "Hypervisor check failed" on startup.

Both the host macOS and virtualized macOS are 15.1. Running latest docker.