Nested virtualization support

[originx]

Hello,

we are trying to have docker images with running android emulators (QEMU). In order to allow acceleration in any environment HAXM would need to support nested virtualisation. I guess this is an interesting use case not just for Android Emulators but for other virtualisation environments/tools Here are the details from the support ticket for Android Emulator.

https://issuetracker.google.com/issues/79499949

Would it be possible to create nested virtualization support and what effort would it take, it would be nice if we could kickstart this.

Thanks

[raphaelning]

Thanks for creating this issue and introducing this new use case to the HAXM community. It was me who commented on Android Bug Tracker, and who implied that the key requirement for HAXM to enable this use case was to implement support for nested virtualization. However, there's another important feature I missed: Docker integration.

Here's an illustration of the proposed usage scenario, based on my understanding of how Docker works on non-Linux hosts:

     Android       <-- L2 guest
------------------
   QEMU + KVM      <-- L2 hypervisor
------------------
Ubuntu container /
 Linux by Docker   <-- L1 guest
------------------
  Docker backend   <-- L1 hypervisor
------------------
 Windows or macOS  <-- Host
------------------
     PC or Mac     <-- Hardware

Clearly, the L1 hypervisor must support nested virtualization. Moreover, it must also be integrated with Docker as a hypervisor backend for the desired host platform, because the L1 guest is started and managed by Docker. However, there is no Docker backend based on HAXM:

• Docker for Mac uses HyperKit, which is based on xhyve.
• Docker for Windows also supports only one backend, which integrates with Hyper-V.

So I think you should try these existing solutions first. I know at least Hyper-V (bundled with recent Windows 10 releases) supports nested virtualization, although you may have to turn on that feature manually. Let us know if they already work as expected.

Strictly speaking, HAXM is not a complete hypervisor, because it relies on QEMU for device emulation. If we want to create a Docker backend for HAXM, we may have to either integrate Docker with QEMU (perhaps not feasible), or implement our own device emulation. The latter approach would require a lot of work - similar to what xhyve has built on top of macOS Hypervisor.framework, and what crosvm has built on top of KVM.