JayFoxRox commented 6 years ago

As a host or as guest?

As a host: Last I checked, FreeBSD had a KVM kernel module. Even if that was removed again, it's probably easier to fix it up, rather than working on HAXM (which uses a very similar interface, but has less features at the time of writing). With competition by WHPX and HVF I assume that HAXM will eventually fade into obscurity unless it can find its niche, or the 2 first-party APIs suddenly stop being worked on. HAXMs major benefit over those APIs is that it's open-source and supports older versions of host OS. However, the same is true for KVM, which also happens to be a more mature project (with very active development, as there's no competition on Linux).

raphaelning commented 6 years ago

@JayFoxRox Thanks for pitching in. @NeoChen1024 Please clarify your feature request, and perhaps explain why you think it's useful. My guess is that you want to run FreeBSD on HAXM as a guest OS. We haven't tried that yet. If you have, what error did you run into?

As a host: Last I checked, FreeBSD had a KVM kernel module.

Thanks for the information. FreeBSD also has its own hypervisor bhyve, which is like KVM + QEMU combined. So before we consider porting HAXM to FreeBSD hosts, we have to ask:

What are the main advantages of using QEMU over bhyve?
Is it possible to implement the QEMU accelerator interface using bhyve API?

NeoChen1024 commented 6 years ago

bhyve on FreeBSD lacks emulation for a normal Graphic card, and sound hardwares like Adlib, GUS. But QEMU have all of above

AlexAltea commented 5 years ago

I assume that HAXM will eventually fade into obscurity unless it can find its niche

I believe there's such a niche: There's no cross-platform permissively-licensed hypervisor. This might sound as quite a weak argument, but reminds me of how Clang began at the time when GCC was a more mature compiler. Sure, it offered minor performance/flexibility perks (similarly to minor compatibility/flexibilty perks offered by HAXM). However, I find one of the main reasons for Clang's success to be its license, which attracted companies like Apple, Sony, and recently even Microsoft (which already maintains a compiler).

Of course, comparing compilers and hypervisors might be a big stretch: there's too little people directly involved with the latter, among other reasons. But it can teach us some lessons:

For instance: Thinking long term, the biggest obstacle to adoption by developers is annoying them with "yet another standard". Honestly, the KVM API is the most convenient/complete one, unsurprising considering how long it's been around. Adopting it would be a huge push to HAXM, since it reduces the overhead for developers. Note how this matches what Clang did with GCC, as they are frequently interchangeable in makefiles with no side effects. (Anyway, this is a discussion for another day).

Back to the original issue: Although I'm not interesting in adding FreeBSD support myself, I hope the changes/refactoring coming in #108 and related patches motivate people at joining and improving HAXM.

raphaelning commented 5 years ago

@AlexAltea Thanks for sharing your insights! Yes, I encourage people who really want to see HAXM working on FreeBSD follow the example set by #108.

tuxillo commented 5 years ago

I don't know if it already exists, but maybe a "Porting" document would be good to have? Even if it's too basic, it can be improved over time :-)

krytarowski commented 5 years ago

I started to work on NetBSD port last year, but I was swamped in memory management API porting. Now since the Linux version is available, I will rebase my patches and try to push it forwards.

For start, I've pushed my generic patches that I kept locally.

AlexAltea commented 5 years ago

I don't know if it already exists, but maybe a "Porting" document would be good to have?

@tuxillo What kind of contents would that document have? If you mean guidelines about porting HAXM to other platforms maybe the best location would be comments on top of the corresponding functions, e.g.:

https://github.com/intel/haxm/blob/master/include/hax_host_mem.h#L39-L51

Right now, there's several functions that are not properly documented, so porting them requires reading the implementations for other platforms which is not really ideal.

@tuxillo @krytarowski Probably you guys are already aware, but you two are working towards the same goal. Just pointing that fact so that you don't "duplicate/overlap work". :-)

krytarowski commented 5 years ago

@AlexAltea tuxillo informed me about his DragonFlyBSD porting off-GH and news about Linux port.

I had an issue that I couldn't express all the needed operations with UVM for HAXM, especially those ones related to physical memory, probably there is need to develop dedicated kernel interfaces. My port was barely insertable into the kernel (and immediately triggering panic) as a loadable module.

I put some notes on HAXM-4-NetBSD at our wiki, but it didn't get much traction.

http://wiki.netbsd.org/users/kamil/qemu/

With a Linux port it will be now much easier to get it to work.

krytarowski commented 5 years ago

https://github.com/krytarowski/haxm/compare/615f5942f17b1569d9e026130002fa354c6ab76a..26d8d685171c6a56779586776a24ed3e72122a9a my original work.. very dirty scratch. I've shared it with @tuxillo .

tuxillo commented 5 years ago

Hi guys,

@AlexAltea thanks for sending the linux port upstream, I think the BSDs can benefit from it in case they decide to port it.

With regards to the porting documentation I was thinking more of a short guide, to see briefly what is expected from the target OS to have. As @krytarowski pointed out the VM API for Linux is quite different from the BSDs and even the BSDs have big differences among them. I may create a draft of such document as I go with my porting efforts for DragonFly BSD but I don't promise anything :-)

Currently I'm mostly doing an exercise of learning while trying to port it, we'll see how far I can get. I'll coordinate with @krytarowski for sure to join efforts if/when needed.

raphaelning commented 5 years ago

Right now, there's several functions that are not properly documented, so porting them requires reading the implementations for other platforms which is not really ideal.

Yes, sorry about that. I think part of the problem comes from the old physical memory allocation/mapping functions, i.e. those implemented by platforms/windows/hax_mem_alloc.c and platforms/windows/hax_mm.c (they don't even have a dedicated header file...). I think it's possible to get rid of them and use only the newer functions in include/hax_host_mem.h; these new functions were introduced with CONFIG_HAX_EPT2, and we tried our best to document them. Most of the old functions are used solely in !CONFIG_HAX_EPT2 code paths, which are now obsolete.

However, the new functions also deal with host physical memory and may not be easy to implement for BSDs. @krytarowski @tuxillo Please feel free to propose a more generic abstraction.

krytarowski commented 5 years ago

I've learnt looking at the Linux code that indeed CONFIG_HAX_EPT2 is reducing the difficulty largely! I've missed it last year and died with my porting as we in general intend to use virtual memory everywhere unless we are accessing PCI/similar buses with dedicated APIs that are designed only for them.

krytarowski commented 5 years ago

I've rebased my old patches, fixed (they were very wrong) and got HAXM to load into the NetBSD kernel.

I'm now trying to run it against qemu&HAXM.

http://netbsd.org/~kamil/haxm/first-load-successful.txt dmesg of inserted the driver into the kernel (with some extra debug)

http://netbsd.org/~kamil/haxm/qemu-first-try.txt something is still wrong with ioctl(2) between qemu and the driver.. investigating.

krytarowski commented 5 years ago

Please share exact ISO images with supported Open-Source OSes that are known to work well (by link). It will make testing the driver easier.

AlexAltea commented 5 years ago

@krytarowski debian-9.5.0-amd64-xfce-CD-1.iso should boot alright (at least the installer).

krytarowski commented 5 years ago

Thanks @AlexAltea !

I've fixed more bugs... and got to the state that HAXM tries to run a guest, but it fails in ept

http://netbsd.org/~kamil/haxm/unsuccessful-run-in-qemu-dmesg-kernel.txt

[   598,082363] haxm_debug: vcpu begin to run....
[   598,082363] haxm_debug: vcpu begin to run....in PE
[   598,082363] haxm_debug: vm entry!
[   598,082363] haxm_debug: ....exit_reason.raw 30, cpu 0 0
[   598,082363] haxm_debug: vmread_cr cr0 30, cr0_mask ffffffffe0000020, state->_cr0 60000010
[   598,082363] haxm_debug: vmread_cr, state->_cr0 60000010
[   598,082363] haxm_debug: ept_tree_get_next_table: Created EPT page table: gfn=0xfffc0, next_level=2, pfn=0x7f14c1, kva=0xffffe528a6cc1000, freq_page_index=1
[   598,082363] haxm_debug: ept_tree_get_next_table: Created EPT page table: gfn=0xfffc0, next_level=1, pfn=0x7f0c7a, kva=0xffffe528a647a000, freq_page_index=5
[   598,082363] haxm_debug: ept_tree_get_next_table: Created EPT page table: gfn=0xfffc0, next_level=0, pfn=0x7ef98b, kva=0xffffe528a518b000, freq_page_index=-1
[   598,082363] haxm_debug: ept_handle_access_violation: Created 64 PTEs for GFN range: gpa=0xfffffff0, start_gfn=0xfffc0, npages=64

I still don't understand what's wrong and I will need to debug it. Pity I keep rebooting my computer a lot of times due to crashes and instability, this slows the porting process down.

I've found more bugs in the Linux version. In hax_pin_user_pages() pinning user pages will never actually happen as we always trigger one of the following conditions:

    if (start_uva & ~PAGE_MASK)
        return -EINVAL;
    if (size & ~PAGE_MASK)
        return -EINVAL;

(at least always on NetBSD).

I had to drop bit-wise inversion operator.

krytarowski commented 5 years ago

@AlexAltea could you please enable all logging messages in HAXM and share a dmsg log for qemu-system-x86_64 -curses --enable-hax? It will help me to spot what's broken on NetBSD, by comparison with a functional implementation.

krytarowski commented 5 years ago

I've fixed few more issues and HAXM seems to work on NetBSD as host. I will try to boot debian now.

I need to fix an issue with attach and get access to more than 1 VM and 1 CPU. Once it will work, I will submit a patch for review.

krytarowski commented 5 years ago

Oops, booting real OS breaks because we trigger a panic 'fpudna from kernel'.

AlexAltea commented 5 years ago

@AlexAltea could you please enable all logging messages in HAXM and share a dmsg log for qemu-system-x86_64 -curses --enable-hax?

@krytarowski Here you go: haxm-master-debian.zip

krytarowski commented 5 years ago

@AlexAltea Thank you! I got Debian to boot... but I had to patch fpudna in the kernel to workaround a behavior that might be a bug of HAXM.

krytarowski commented 5 years ago

137 Experimental NetBSD host support has been submitted to review.

krytarowski commented 5 years ago

HAX in pkgsrc http://mail-index.netbsd.org/netbsd-users/2019/02/13/msg022207.html

intel / haxm

Add Support for FreeBSD? #106

137 Experimental NetBSD host support has been submitted to review.