search

machyve / xhyve

xhyve, a lightweight OS X virtualization solution
Other
6.44k stars 356 forks source link

Error on macOS aarch64 #206

Open chadbrewbaker opened 3 years ago

chadbrewbaker commented 3 years ago

Built from commit b7e589f91e68b86d6f02e85c51f9a5997ebaf148 .

xcodebuild
./build/Release/xhyve -h
zsh: killed   ./build/Release/xhyve -h

Is this expected behavior or a bug?

chadbrewbaker commented 3 years ago

This is upsetting. https://forums.macrumors.com/threads/if-you-disable-sip-all-you-ios-apps-will-stop-working-on-your-m1.2269661/

Turning off SIP just to run dtruss hoses apps permanently. You would have to link with a tracing glibc to get system calls.

chadbrewbaker commented 3 years ago 
ls /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk//System/Library/Frameworks/Kernel.framework/Versions/Current/Headers/arm64 
machine_cpuid.h     machine_machdep.h

These seem to be the arm64 headers to fix src/vmm/x86.c ?

chadbrewbaker commented 3 years ago

https://developer.apple.com/documentation/hypervisor/apple_silicon

chadbrewbaker commented 3 years ago

@jonmasters got Fedora to boot on M1 https://twitter.com/jonmasters/status/1331137023386800128

741g commented 3 years ago

That's with Virtualization.framework, which is the easier-to-use library versus Hypervisor.framework (and may be more appropriate even?)

The tweet references https://github.com/KhaosT/SimpleVM

jonmasters commented 3 years ago

Right. I was trying to get something quick and dirty up and running, so I used the Virt framework. A full solution built using the Hypervisor framework would be nice to have since that would provide an ability to add/remove hardware and change the machine model, among many other things.

fayep commented 3 years ago

To answer @chadbrewbaker original question. The build generates an x86_64 binary and Rosetta won’t work for virtualization which was made clear by Apple. This won’t work for arm64 until it is ported and if you are interested you should look at the architecture specific code in FreeBSD-UPB/freebsd:bhyvearm64-rebase for inspiration. While following this paper: https://papers.freebsd.org/2019/bsdcan/elisei-bhyvearm64_cpu_and_memory_virtualization_on_armv8.0_a/

741g commented 3 years ago

I'm working on a stack of Hypervisor.framework patches here: https://android-review.googlesource.com/c/platform/external/qemu/+/1510277

brandonros commented 3 years ago

I'm working on a stack of Hypervisor.framework patches here: https://android-review.googlesource.com/c/platform/external/qemu/+/1510277

@741g looks like that got merged. what are the next steps? :)

https://gist.github.com/osy86/10d0ac0210f0eb99474ae31cf8d6dd9e#file-windows-10-arm-utm-zip for anybody else that lands on this

741g commented 3 years ago

You can pretty much follow those and incorporate into the xhyve implementation. It was quite straightforward compared to x86 support and as long as you have virtual firmware for setting up memory maps and routing interrupts, it should be doable. The key parts are:

  1. Setting up an arm64 set of source files in xhyve/src for storing register state along with timer and GIC state. You can use the qemu version as a reference. You'll probably need a different set of virtual firmware sources. You can use the virt board (https://android.googlesource.com/platform/external/qemu/+/refs/heads/emu-master-dev/hw/arm/virt.c) as a reference. You'll need to implement something that stores information for memory maps, irqs, etc. and implements the state for GIC and vtimer.

  2. I notice there's a lot of x86-isms still present in the source, such as vmm_ioport.c, vmm_util.c, vmm_lapic.c, etc. vmm.c will need different handlers for ARM64 style exits such as DABT and WFx in addition to Hypevisor.framework's VTIMER. (see the code for reference).

  3. First, try getting something basic running that uses hv_vcpu_run, hv_vm_create, hv_vm_map that follows how it's being used in xhyve, while aborting on all exits, similar to https://android.googlesource.com/platform/external/qemu/+/c17eb6a3ffd50047e9646aff6640b710cb8ff48a

You'll need to ensure that

  1. Implement VM exit handling

There will be these types of Hypervisor.framework exits that you need to handle for ARM64:

  1. Implement kicking the vcpus out of running the guest (https://android.googlesource.com/platform/external/qemu/+/cfaec60e391f549fed43736aa09fa670416b6bdf)