Nested Virtualization for WSL2 VM

[ameeno]

Greetings,

I am trying to do Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true

On the wsl2vm itself so I can use KVM inside it.

However, I cannot do that, and I don't know why Get-VM does not see the wsl2vm so I cannot expose these vars.

how do I do it?

[Biswa96]

You can get Hyper-V VM ID with hcsdiag list command as admin. But I don't know if nested virtualization can be done in WSL2. It's like Inception... 🤓

[ameeno]

Hi Biswa96 yes I will try and make it like Inception WIn10 -> wsl(VM) -> KVM -> OSX->Parrallels->Win10 :)

[therealkenc]

I'm on KVM->Win10->WSL2 FWIW.

[kterhorst]

I am interested in this as well. Is there any way to enable ExposeVirtualizationExtensions for the Hyper-V instance running WSL2?

[ameeno]

Well my main desktop rig has been rebuilt around ryzen 3600. Unfortunately, windows does not support nested virtualization in any form for and processors. So it seems a linux Kvm host as the base operating system is required now.

You can enable nested in kvm and have osx and windows 10 SMS with pic box pass through which is almost natively Experiance. However you need two graphics cards. 1 for linux and 1 for your vm.

[scottjoshuac]

This would be very useful to have, that way I don't have to have two separate Linux VMs (WSL2 + my GNS3 VM). Please enable this feature as it's already built into Hyper-V.

[tsujp]

Is there any comment on if and when this will be a thing? It's kind of a requirement for WSL2 to be actually useful, not being able to run VMs with KVM for hardware acceleration in 2019 is a bit.. concerning.

I used WSL1 and it topped out being useful at letting me type Linux commands as it used to have all manner of port issues, restrictions, slowdowns and the like. WSL2 might be faster with I/O but VMs are a CORE REQUIREMENT of any remotely professional setup.

If Microsoft want people to use Windows with WSL2 as a host operating system, WSL2 must support KVM or there needs to be another non-Intel way to get hardware virtualization working.

[steffengy]

Technically there's already everything in place for it to work.

If someone wants to play with this right now with an insider build: 1) Build a kernel with KVM support (missing kvm_intel & msr mostly). 2) Pass ExposeVirtualizationExtensions to the utility VM and boot a different kernel. This is very hacky. Let me know, if you can't wait or are interested in a 1-click workaround.

Et voilà

Point 1) in Microsofts upstream kernel might lead to a bit of discussion, but with both kvm_intel & kvm_amd as kernel modules everybody should be happy? For 2) it's either allowing configuration or just passing true.

[scottjoshuac]

Yep, they just need to enable the "ExposeVirtualizationExtensions" feature as it's only available to Hyper-V machines that are in the Hyper-V manager. Since the WSL2 VM does not have a name, we can't apply it.

[steffengy]

@scottjoshuac Technically you can as the screenshot above shows, it's just not easy at all because it doesn't seem to be an intended feature yet ;) (The WSL2 VM is hosted through HCS, therefore doesn't show up in hyperV, which the powershell CMDLets use. Also a new VM is created each time, so for a workaround you need to intercept its creation and set ExposeVirtualizationExtensions before its started up)

[rick-pri]

What I'd like to do, and so which requires the WSL 2 isolated VM to be started every time with the ExposeVirtualizationExtensions option, is run the likes of Minikube within WSL 2. Having to have another VM running to then run minikube.exe from within WSL is a real horrible cludge.

Basically, I'm looking for the GUI desktop support of Windows, with the *nix support of WSL in Windows Terminal; as an alternative to MacOS. This has been working okay, until I try to get Minikube to run within the Hyper-V VM that is WSL 2; which it can't of course.

Adding the parameter seems to be a no brainer, it's already there in the technology so there's not really any dev time needed, just testing, although I can see how having a VM running in the background when you kill your terminal session might prove to be a bit of a headache to handle though.

The other point about AMD support is valid too, we need to see this support coming through as well, especially given AMD's current acendency in desktop and laptops.

[rick-pri]

So, I got fed up with this situation today and decided to install Ubuntu in a Hyper-V VM with the ExposeVirtualizationExtensions flag passed to it. I installed the gcloud sdk, installed the components, skaffold, minikube and kubectl and then installed virtualbox-6.1. One slow minikube start later and I can see this:

And so my point from above, turning this on works and so it would be much more helpful if it was enabled for WSL 2.0, either by default or as a configurable, because then I wouldn't have to roll my own VM and add the flag manually. It's just the little additional touches which will make this so much more user focused.

[offlinehacker]

@steffengy how do you intercept creation and set ExposeVirtualizationExtensions, do you maybe have some script available?

[steffengy]

@offlinehacker Sure, it's not very user-friendly though: https://gist.github.com/steffengy/62a0b5baa124830a4b0fe4334ccc2606

[Biswa96]

There are some changes in WSL2 Linux kernel repo with v4.19.104. The config files has some KVM features enabled. So, one may compile the kernel now and try it. No hacky step required.

[steffengy]

@Biswa96 seems like it, though I'd argue that the hacky step is not compiling the kernel but fiddling with HCS.

[Biswa96]

HCS (Hyper-V Compute System) APIs isn't fully** documented for WSL2. But compiling Linux kernel is.

** but it can be found in Windows SDK. And also Docker Desktop uses it.

[steffengy]

@Biswa96 Except that using the HCS API doesn't help here. You currently have to interfere with how LxssManager interacts with HCS and due to LxssManager being a protected process you have to do a lot of hackery (as shown in the GIST) to intercept API calls in HCS internals, which is entirely undocumented besides debug symbols. So yeah building the kernel in anycase is the very very easy part. If it's implemented in windows eventually and this newer kernel ships, ofcourse you won't have to do anything anymore.

[offlinehacker]

@steffengy Thank you for sharing.

Ok i tried windbg script, but on my build of windows 19041.81, i do not get {"Owner":"WSL"' magic string. I see {"PropertyTypes":["Memory"]} and {"VirtualNodeCount":1,"VirtualMachineMemory":{"AvailableMemory":62,"AvailableMemoryBuffer":832,"ReservedMemory":1048576,"AssignedMemory":1048576,"SlpActive":false,"BalancingEnabled":false,"DmOperationInProgress":false},"VirtualNodes":[{"VirtualNodeIndex":0,"PhysicalNodeNumber":0,"VirtualProcessorCount":8,"MemoryUsageInPages":1048576}]}, but nothing else, even when starting wsl.exe after shutting it down.

Which build of windows are you running?

This method is really hacked up, but i kind of expected that you need to hook dll methods, what else to expect in windows.

[steffengy]

@offlinehacker Works for me at 19041.84. Make sure to start a CMD with administrative privileges (else LxssManager is not being restarted) in the folder where you placed the files from the GIST and then run "start-wsl.bat" (double clicking it does not work due to CWD).

[offlinehacker]

@steffengy I have modified your script and managed to get it work. Don't need two step process anymore, directly read, parse, modify and write memory. https://gist.github.com/offlinehacker/4d48f46fdcd64c9a24e3952c69bfcd09

[steffengy]

@offlinehacker Great that you enjoy it, I updated the gist above with your changes. :)

[scottjoshuac]

@steffengy and @offlinehacker thanks for sharing this! I may switch back to Windows now... Although I'm getting used to Linux as my primary and have Wine running the only Windows app I "need" (League of Legends) so it's a tough choice!

[offlinehacker]

I am always using Linux for development, windows was just a gateway drug, as I want to have portable dev environment, that also runs in hyper-v 😅 Just look at ridiculous hacks we need to do, to change a json string passing from one service to other, where in Linux it would be 5 minute fix. Microsoft has done a lot for open source in last years, but as long as windows will not be more open, it is too rigid platform for a lot of developers.

[offlinehacker]

@steffengy thanks for updating gist 🙂

[rick-pri]

I'm back on Linux, this time with my nVIDIA GPU enabled and my 4K screens plugged into that. It means that I get <1hr battery life for meetings and I have to plug in 5 cables daily instead of 3, and I have to sort out my screen layout every day but for development work it's just working. Linux desktop is still nowhere near where it needs to be to challenge Windows/MS but development wise it's far ahead.

[offlinehacker]

I have also written notes and scripts i use for rebuilding kernel and starting WSL with nested virtualization here: https://gist.github.com/offlinehacker/b1d96515f87a47bd0b0bea574eab5583

[kwinz]

I have the same issue, except I want to enable ExposeVirtualizationExtensions for the Windows 10 client, not for the WSL2 client, if that makes sense? #5030 I can't just wrap the main Windows 10 OS vm start in a script like @offlinehacker's excellent script for WSL2. Is it feasible to enable ExposeVirtualizationExtensions globally?

[Biswa96]

If anyone have latest Windows 10 insider build, try adding these lines in %UserProfile%\.wslconfig file:

[wsl2]
debugConsole=true
pageReporting=true
nestedVirtualization=true

debugConsole and nestedVirtualization options are undocumented now. I have not any beefy CPU & RAM to test nested VM it.

[onomatopellan]

Thanks @Biswa96! nestedVirtualization works now in build 19619 inside a hyper-v VM. 😀 Before that the script of this thread only did work on host but not inside the VM.

[kode54]

You can enable nested in kvm and have osx and windows 10 SMS with pic box pass through which is almost natively Experiance. However you need two graphics cards. 1 for linux and 1 for your vm.

Actually, if you don't need any interaction with the host Linux OS, you can pass through the only GPU available to the host, as well as any USB interfaces it will allow you to pass through, to the guest. All you need to do is configure the host with video=efifb:off, blacklist the GPU modules, and configure the vfio module to attach to the host GPU on boot. The host machine will display the two status lines for loading the kernel and the initramfs, then the display will remain unresponsive.

You'll need to log into the machine remotely via ssh if you do this, but you should be able to boot your bare metal VMs, one at any given time if they all require the single GPU. With a little luck, you should even be able to shut them off and start others up in their place, again, using an external device to access the ssh session on the host OS. This is what I'm currently doing with Proxmox.

[Marietto2008]

@kode54 :

so what ? it seems that with the latest windows 10 update we can passthrough gpu and usb devices ?

[kode54]

No. WSL2 uses Hyper-V, which has never been able to pass through host devices. I was referring to running a Linux OS on bare metal, which does support VFIO, and passing devices directly to the Windows guest and any other guests of the top level Linux OS.

You can then enable nested virtualization and run WSL2 on that Windows guest. WSL2 still won't support any form of passthrough, either way.

[Marietto2008]

Anyway,reading all around what the people says,I suspect that MS built one only Hyper-V and they use it on Windows 10 consumer and server edition. BUT,on the consumer version the possibility of making the passtrhough is disabled for commercial reasons. It's not the first time that MS do that. As also I'm sure that there are people who knows how to enable it.

[LiuChangFreeman]

If anyone have latest Windows 10 insider build, try adding these lines in %UserProfile%\.wslconfig file:

[wsl2]
debugConsole=true
pageReporting=true
nestedVirtualization=true

debugConsole and nestedVirtualization options are undocumented now. I have not any beefy CPU & RAM to test nested VM it.

However, this prevents me from starting WSL2

[onomatopellan]

@LiuChangFreeman It's your CPU from AMD? It's well known nested Hyper-v virtualization doesn't work on AMD cpus.

[benhillis]

@LiuChangFreeman - anything interesting in the debug console?

[LiuChangFreeman]

@LiuChangFreeman It's your CPU from AMD? It's well known nested Hyper-v virtualization doesn't work on AMD cpus.

Yes, I'm using ryzen 1700.

[onomatopellan]

@LiuChangFreeman Then you will need to use nestedVirtualization=false on your .wslconfig file. Right now is not possible to run an hyper-v VM inside an hyper-v VM on an AMD cpu. https://github.com/MicrosoftDocs/Virtualization-Documentation/issues/1276

[ThaDaVos]

I tried the .wslconfig thingy but my KVM will still not start:

$ sudo kvm
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Is there something extra I need to do after creating the .wslconfig file and inserting:

[wsl2]
debugConsole=true
pageReporting=true
nestedVirtualization=true

I also did wsl --shutdown to be sure

[Marietto2008]

you should recompile the kernel for enabling virtualization

[ThaDaVos]

Oh I thought that was unneeded as of Biswa's comment - I thought it was included in the kernel by now

Any guide on how to recompile the kernel? And is this per distro or for all distro's?

[ThaDaVos]

I followed the instructions to create a new kernel with the KVM enabled but I'm still getting the following error when I try to enable the kvm_intel modprobe:

$ sudo modprobe kvm_intel
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.84-microsoft-kvm+/modules.dep.bin'
modprobe: FATAL: Module kvm_intel not found in directory /lib/modules/4.19.84-microsoft-kvm+

[Marietto2008]

exactly. the same error that I had for a long time. as soon as u have recompiled the new kernel,you should place it in c:\users\your username\ and u should create a .wslconfig file with this content inside

[wsl2] kernel=C:\Users\marietto2020\vmlinux-usb (name of your new kernel) debugConsole=true pageReporting=true nestedVirtualization=true

after this,you can restart wsl and u can try to load the kernel modules.

[ThaDaVos]

The thing is, that's exactly as I did - if you look at the error you can see the different kernel name even (I replace standard with kvm) - and I only set the nestedVirtualization option to true - I left the other two out

As I can see my custom kernel name when I run uname -a I assume it's loaded, but it won't work

[therealkenc]

It's your CPU from AMD? It's well known nested Hyper-v virtualization doesn't work on AMD cpus.

Been meaning to backlink this uservoice for a while. Encourage everyone to upvote. [Discourage everyone from adding a vacuous quip like the sorry state of the comment section over there.]

[onomatopellan]

@ThaDaVos If you are on latest fast ring build 19631 you dont' need to recompile the kernel to at least see kvm acceleration enabled after typing sudo kvm-ok. Just open Notepad and create a .wslconfig file on your Windows user folder with this content:

[wsl2]
nestedVirtualization=true

Make sure the Notepad status bar shows Windows (CRLF) and UTF-8.

Edit: BTW what's your CPU?

[ThaDaVos]

@ThaDaVos If you are on latest fast ring build 19631 you dont' need to recompile the kernel to at least see kvm acceleration enabled after typing sudo kvm-ok. Just open Notepad and create a .wslconfig file on your Windows user folder with this content:

[wsl2]
nestedVirtualization=true

Make sure the Notepad status bar shows Windows (CRLF) and UTF-8.

Edit: BTW what's your CPU?

I'm on the exact build before that.... my pc was just downloading that build... but on 19631 KVM is natively added?

[onomatopellan]

On intel CPU I have been using the nestedVirtualization option since build 19619. It's the official microsoft kernel 4.19.104.

Edit: I must add I only used sudo kvm-ok and it said "KVM acceleration can be used". I didn't try anything more than that.

[ThaDaVos]

Only on Insiders Build 19631 I get the "KVM Acceleration can be used" with sudo kvm-ok - stupid thing is, when I start the virt-manager and want to create a VM it shows this:

So it doesn't fully work, or I am missing something I tried to do the sudo modprobe kvm_intel but got greeted with the same error:

$ sudo modprobe kvm_intel
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.104-microsoft-standard/modules.dep.bin'
modprobe: FATAL: Module kvm_intel not found in directory /lib/modules/4.19.104-microsoft-standard

Edit: running sudo kvm does start something but also shows:

$ sudo kvm
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]