Closed ameeno closed 3 years ago
KVM modules are already integrated in the kernel. Maybe you need to do
sudo service libvirtd start
before creating the VM.
Also take a look at the first lines of sudo dmesg
to see if KVM is loaded.
Since Windows Insider Fast build 19640 AMD users can finally use nested Hyper-V Virtualization. 🎉
https://techcommunity.microsoft.com/t5/virtualization/amd-nested-virtualization-support/ba-p/1434841#
For nesting, does that include VirtualBox using it's own engine as a nested VM? Or is it Hyper-V nested in Hyper-V only? So with Insider Fast 19640 I can now run VBox with it's own engine for VMs while also using WSL2?
@Drizzt321 only for Hyper-v inside Hyper-v. VirtualBox and VMWare still need to use the Hyper-V Platform API.
Thanks @onomatopellan, that's what I was thinking. Too bad, really wish MS would expose the needed stuff for nested VMs for other VM engines to hook into.
.wslconfig
with:
[wsl2]
nestedVirtualization=true
Run the following commands:
$ sudo kvm-ok
sudo: kvm-ok: command not found
$ egrep -c '(vmx|svm)' /proc/cpuinfo
12
I start installing KVM using the directions I always use in Ubuntu with the following:
sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
sudo adduser `id -un` libvirt
sudo adduser `id -un` kvm
I then go on to install `docker`, `kubectl`, and `minikube` and restart the WSL service.
$ minikube start --vm-driver=kvm2 --kubernetes-version=1.15.10 😄 minikube v1.11.0 on Ubuntu 20.04 ✨ Using the kvm2 driver based on user configuration
❗ 'kvm2' driver reported an issue: /usr/bin/virsh domcapabilities --virttype kvm failed: error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory 💡 Suggestion: Follow your Linux distribution instructions for configuring KVM 📘 Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/kvm2/
💣 Failed to validate 'kvm2' driver
Check to see why `kvm2` is no longer working:
$ virsh list --all error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
A little research suggests the service isn't running:
$ sudo service libvirtd start libvirtd: unrecognized service $ sudo systemctl start libvirtd System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is down
Others in this issue seem to be able to run `kvm` in WSL2 without issue. What am I doing wrong here?
@eox-dev
sudo apt-get install libvirt-daemon-system-sysv
sudo service libvirtd start
Or consider something like genie to use systemd instead.
@jstangroome Thanks, that did get me further, but running into:
$ minikube start --vm-driver=kvm2 --kubernetes-version=1.15.10
😄 minikube v1.11.0 on Ubuntu 20.04
✨ Using the kvm2 driver based on user configuration
🆕 Kubernetes 1.18.3 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.18.3
👍 Starting control plane node minikube in cluster minikube
🔥 Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔥 Deleting "minikube" in kvm2 ...
🤦 StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=7, Message='can't connect to virtlogd: Failed to connect socket to '/run/libvirt/virtlogd-sock': No such file or directory')
🔥 Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
😿 Failed to start kvm2 VM. "minikube start" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=7, Message='can't connect to virtlogd: Failed to connect socket to '/run/libvirt/virtlogd-sock': No such file or directory')
💣 error provisioning host: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=7, Message='can't connect to virtlogd: Failed to connect socket to '/run/libvirt/virtlogd-sock': No such file or directory')
😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
👉 https://github.com/kubernetes/minikube/issues/new/choose
Still No such file or directory
. I'll keep digging into it.
$ virsh list --all
Id Name State
---------------------------
- minikube shut off
@eox-dev add your user to the kvm and libvirt groups so you have permission to access the relevant sockets. You typically need to re-login for group changes to take effect.
I followed this article: https://boxofcables.dev/accelerated-kvm-guests-on-wsl-2/ And I got:
sudo modprobe -r kvm_intel
libkmod: ERROR ../libkmod/libkmod-config.c:656 kmod_config_parse: /etc/modprobe.d/kvm.conf line 1: ignoring bad line starting with 'modprobe'
modprobe: FATAL: Module kvm_intel is builtin.
During unloading of the module and also:
cat /sys/module/kvm_intel/parameters/nested
N
@startergo Did you mark kvm_intel as M in make menuconfig
step?
@startergo Did you mark kvm_intel as M in
make menuconfig
step?
Yes indeed. Also I get:
sudo modprobe -r kvm_intel
modprobe: ERROR: ../libkmod/libkmod-module.c:799 kmod_module_remove_module() could not remove 'kvm_intel': No such file or directory
WSL2 loads the kernel pretty quickly so there is no need for modules, just integrate it in the kernel.
@startergo As @onomatopellan said, the official kernel (with statically built-in kvm_intel) works pretty well, except that you should pass the contents in kvm-nested.conf
as kernel command-lines.
How so? Why is it showing that nested virtualization is disabled?
@startergo
How so? Why is it showing that nested virtualization is disabled?
I found that since udev
is not a running service in WSL2, the rules in /lib/udev/rules.d/50-udev-default.rules
did not apply and I had to sudo chown root:kvm /dev/kvm
and sudo chmod 0660 /dev/kvm
to fix the error message qemu-system-x86_64: failed to initialize KVM: Permission denied
while trying to minikube start --driver kvm2
but then it all worked.
kernelCommandLine=intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[wsl2]
nestedVirtualization=true
kernel=C:\\Users\\<username>\\bzImage
debugConsole=true
pageReporting=true
kernelCommandLine=intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
is this ok?
@startergo maybe you don't need the custom kernel if you already have 4.19.104+
@startergo maybe you don't need the custom kernel if you already have 4.19.104+
So i just drop the kernel=
line?
Why do I get:
sudo modprobe kvm_intel
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.104-microsoft-standard/modules.dep.bin'
modprobe: FATAL: Module kvm_intel not found in directory /lib/modules/4.19.104-microsoft-standard
I can't even find /lib/modules
~/WSL2-Linux-Kernel-4.19.104-microsoft-standard$ sudo make modules_install --trace
Makefile:641: target 'include/config/auto.conf' does not exist
test -e include/generated/autoconf.h -a -e include/config/auto.conf || ( \
echo >&2; \
echo >&2 " ERROR: Kernel configuration is invalid."; \
echo >&2 " include/generated/autoconf.h or include/config/auto.conf are missing.";\
echo >&2 " Run 'make oldconfig && make prepare' on kernel src to fix it."; \
echo >&2 ; \
/bin/false)
Makefile:1256: target '_modinst_' does not exist
rm -rf /lib/modules/4.19.104-microsoft-standard/kernel
rm -f /lib/modules/4.19.104-microsoft-standard/source
mkdir -p /lib/modules/4.19.104-microsoft-standard/kernel
ln -s /home/<username>/WSL2-Linux-Kernel-4.19.104-microsoft-standard /lib/modules/4.19.104-microsoft-standard/source
if [ ! . -ef /lib/modules/4.19.104-microsoft-standard/build ]; then \
rm -f /lib/modules/4.19.104-microsoft-standard/build ; \
ln -s /home/<username>/WSL2-Linux-Kernel-4.19.104-microsoft-standard /lib/modules/4.19.104-microsoft-standard/build ; \
fi
cp -f ./modules.order /lib/modules/4.19.104-microsoft-standard/
cp -f ./modules.builtin /lib/modules/4.19.104-microsoft-standard/
make -f ./scripts/Makefile.modinst
scripts/Makefile.modinst:18: target '__modinst' does not exist
:
Makefile:1273: update target '_modinst_post' due to: _modinst_
echo ' DEPMOD 4.19.104-microsoft-standard'; /bin/bash ./scripts/depmod.sh /sbin/depmod 4.19.104-microsoft-standard
DEPMOD 4.19.104-microsoft-standard
<username>@HOME:~/WSL2-Linux-Kernel-4.19.104-microsoft-standard$ which depmod
/sbin/depmod
<username>@HOME:~/WSL2-Linux-Kernel-4.19.104-microsoft-standard$ depmod --version
kmod version 26
+XZ -ZLIB +OPENSSL -EXPERIMENTAL
@startergo maybe you don't need the custom kernel if you already have 4.19.104+
I got it now. Thanks:
[wsl2]
nestedVirtualization=true
debugConsole=true
pageReporting=true
kernelCommandLine=intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
cat /sys/module/kvm_intel/parameters/nested
Y
@startergo You didn't eventually compile the kernel?
I have put the same WSL2 config:
[wsl2]
nestedVirtualization=true
debugConsole=true
pageReporting=true
kernelCommandLine=intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
and I get:
cat /sys/module/kvm_intel/parameters/nested
Y
,but at the same time I get this:
dmesg | grep kvm
[ 0.000000] Command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[ 0.119133] Kernel command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[ 0.330695] kvm: no hardware support
[ 0.330696] kvm: no hardware support
and
egrep -c '(vmx|svm)' /proc/cpuinfo
0
My CPU supports both VTx and VTd.
Solved my problem by updating the windows preview build from this website: https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewadvanced
Solved my problem by updating the windows preview build from this website: https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewadvanced
Hi Rizary, Does was It solved for AMD or Intel CPU that preview build?
I would like to know whether AMD CPU's are supported with KVM on WSL2.
For me the kvm-ok command says:
Your CPU does not support KVM extensions
Even though I've succesfully used Hyper-v within windows and KVM within linux.
Which kernel would I need to compile? I'm trying:
WSL2-Linux-Kernel-4.19.128-microsoft-standard
I'm also using the following windows build:
19041.572
Does anyone know which kernel and which windows build I need to use in order to have KVM acceleration work in WSL2 for a AMD processor?
EDIT: I just read this guide: https://boxofcables.dev/accelerated-kvm-guests-on-wsl-2/
And at the start it has a note about AMD users that KVM support is still on it's way, does anyone know what the current status is of it?
@Simbaclaws I built a 5.8 kernel the other day with KVM AMD built in as well as zfs on the latest insiders build.
It appears to work fine for me.
Thank you @ameeno, I suppose this hasn't landed in the official release then. Or perhaps I just need newer kernel sources. I haven't tried an insiders build yet. I might go for that.
Did you build with a wsl2 based kernel? Or a custom one? Could you please send me a .config and tell me which kernel sources you used? I would like to test this on a official release as well as on a insider build.
@Simbaclaws it may work on official release if you have an intel CPU (I have AMD hence insiders)
my .wslconfig in home folder looks like this:
[wsl2]
kernel=c:\\Users\\Arun\\wsl2kernel5.8-kvm-zfs # my custom kernel
nestedVirtualization=true # needed for nested
localhostForwarding=true # already default
swap=0 # not needed but I use it.
On the kernel side of things, I built my own using the stable sources from kernel.org 5.8
I used a mixture of his guide: https://boxofcables.dev/accelerated-kvm-guests-on-wsl-2/
and his guide (for zfs, not relevant to you) https://wsl.dev/wsl2-kernel-zfs/ - Used 5.8 sources not 5.5 Interesting screenshot here ;) https://i.imgur.com/LFAGM8N.png
In a nutshell it works good for nested, and means I do not need to run a separate hyper-v vm.
Interestingly, the config files from the WSL2 Kernel sources repo from microsoft, where i pulled the configs from, seem to have KVM already switched on.
Here is my Kernel config file attached. but I have ADDED the zfs module. (You can delete)
Hi everyone! I've tried all methods to make KVM working inside Ubuntu (on WSL2) but I failed.
I have one strange behavior - to run WSL2 I have to run "Virtual machine platform" windows feature. But when I have it turn on I got information that my processor doesn't support virtualization (e.g. while I using Intel Processor Identification Utility - https://downloadcenter.intel.com/download/28539). When I I turn it off I got info that virtualization is working - but of course WSL2 is not working in that configuration. The same with VMware - when I turn on "Virtual machine platform" VT-x is stop working in VMware.
Is it normal that turning on "Virtual machine platform" break virtualization for other tools? Can it be the cause why I get "Your CPU does not support KVM extensions" message for kvm-ok?
Thanks in advance for any help!
@bartoliniii what's your CPU and what Windows build (winver.exe) are you running?
@bartoliniii make sure you have enabled svm (on amd) or VT-X on Intel inside your system BIOS.
Nested virtualization requires separate flags from straight virtualization
what's your CPU and what Windows build (winver.exe) are you running?
make sure you have enabled svm (on amd) or VT-X on Intel inside your system BIOS.
I've checked it twice and it is enabled. What's more it is working correctly while I disable "Virtual machine platform" windows feature - this indicates that the source of the problem is not in bios configuration.
Nested virtualization requires separate flags from straight virtualization
Do you mean the flag in the bios?
Here is how it looks in my bios configuration:
And when I turn off "Virtual machine platform" windows feature I see VT-X active (in Intel Processor Identification Utility):
@bartoliniii KVM won't work in WSL2 if you don't run at least Insider build 20175.
Is it normal that turning on "Virtual machine platform" breaks other virtualization solutions because it's a subset of Hyper-V and with it enabled your Windows session runs also under the hypervisor. Right now Hyper-V is the only VM solution that supports nested virtualization with WSL2 enabled. VMWare VM works with WSL2 enabled in latest version but it doesn't support nested virtualization yet.
Hi @ameeno. I try to have kvm working on my amd Win10 2004 (19041.572) inspired by your advices. But still no kvm-ok. I'm stucked... I tried to compile my kernel with KVM for Amd processors support as Module. Amd virt activated, nestedVirtualization=true in my wslconfig. 1) Is my /etc/modprobe.d/kvm-nested.conf ok?:)
nested=1
options kvm-amd enable_shadow_vmcs=1
options kvm-amd enable_apicv=1
options kvm-amd ept=1
2) Bellow, where I am:
rudy@RUDY-PC:/mnt/c/Users/rudy$ uname -ar
Linux RUDY-PC 5.8.18-microsoft-standard #3 SMP Sun Nov 1 22:52:16 CET 2020 x86_64 x86_64 x86_64 GNU/Linux
rudy@RUDY-PC:/mnt/c/Users/rudy$ sudo modprobe kvm_amd
modprobe: ERROR: could not insert 'kvm_amd': Operation not supported
rudy@RUDY-PC:/mnt/c/Users/rudy$ sudo kvm-ok
INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used
Thanks for your help
@rudyspano KVM won't work in WSL2 if you don't run a Dev Insider build 20175 or superior.
For amd nested you need to use Dev build insiders win10.
For intel you can use 2004 win 10 but for amd it must be insiders Dev build of win10.
If you are scared to use insiders win10 like I was as main system, you can install insiders build into hyper-v gen1 virtual machine.
And afterwards you can do VHD boot on your Windows.
This way you can dual boot normal windows +insiders
And after you have insiders, you can have fun with my earlier instructions.
@onomatopellan
KVM won't work in WSL2 if you don't run at least Insider build 20175.
Yes, I confirm that was the cause. Thanks for you help!
@rudyspano KVM won't work in WSL2 if you don't run a Dev Insider build 20175 or superior.
Does KVM work in WSL2 with the Win10 version 2004, or the insider build 20175 has to be used?
Currently I followed this guide: https://boxofcables.dev/accelerated-kvm-guests-on-wsl-2/
and I got the folllowing result on my Win 10 version 2004 with an Intel processor:
$ dmesg | grep kvm
[ 0.231668] kvm: no hardware support
@ameeno
For intel you can use 2004 win 10 but for amd it must be insiders Dev build of win10.
I have an Intel processor and am on the Win10 version 2004, but KVM does not work. Any pointers?
@ameeno
For intel you can use 2004 win 10 but for amd it must be insiders Dev build of win10.
I have an Intel processor and am on the Win10 version 2004, but KVM does not work. Any pointers?
Custom kernel with KVM enabled. Modprobe kvm_intel. Enable qemu libvirt ect.
WSL is a little tricky to work with as systemd + mod does not launch by default. But with the right kernel, the right modules. Virtualization enabled in the system BIOS. Nested enabled in the wsl config, KVM will work.
Try and find a nested virtualization guide on Google for WSL. 99% of the guides online target intel.
For intel you can use 2004 win 10 but for amd it must be insiders Dev build of win10.
I have an Intel processor and am on the Win10 version 2004, but KVM does not work. Any pointers?
Custom kernel with KVM enabled. Modprobe kvm_intel. Enable qemu libvirt ect.
@ameeno
I was using built-in KVM Intel driver before. This time I changed the KVM Intel to a module and do modprobe. But it still fails.
$ sudo modprobe kvm_intel [sudo] password for test: modprobe: ERROR: could not insert 'kvm_intel': Operation not supported
WSL is a little tricky to work with as systemd + mod does not launch by default. But with the right kernel, the right modules. Virtualization enabled in the system BIOS. Nested enabled in the wsl config, KVM will work.
Try and find a nested virtualization guide on Google for WSL. 99% of the guides online target intel.
I only find this guide: https://boxofcables.dev/accelerated-kvm-guests-on-wsl-2/
I wonder whether Win 10 Ver 2004 (19041.630) works with KVM on WSL2 because @onomatopellan said Win 10 Dev Insider is needed.
Could you please share more details? Thanks!
Part 1) nested t rue inside .wslconf part 2) custom kernel with kvm modules either built or built in part 3) virtualization enabled in bios.
Part 1) nested t rue inside .wslconf part 2) custom kernel with kvm modules either built or built in part 3) virtualization enabled in bios.
I can confirm that all of the 3 parts are enabled or followed. So I wonder whether it's caused by the official Win 10 ver 2004 does not work.
have you configured modprobe nested?
Not sure it is needed unless you are using docker / Nested kvm,
but.. to configure nested on the KVM itself
Configure kvm-intel
nano /etc/modprobe.d/kvm-nested.conf Paste:
options kvm-intel nested=1 options kvm-intel enable_shadow_vmcs=1 options kvm-intel enable_apicv=1 options kvm-intel ept=1
Now if you are using ubuntu distro in ubuntu, you could try and use the command sudo kvm-ok to check if kvm is working and active.
also i think its not sudo modprobe kvm_intel its actually, its actually sudo modprobe kvm
have you configured modprobe nested?
Not sure it is needed unless you are using docker / Nested kvm,
but.. to configure nested on the KVM itself
Configure kvm-intel
nano /etc/modprobe.d/kvm-nested.conf Paste:
options kvm-intel nested=1 options kvm-intel enable_shadow_vmcs=1 options kvm-intel enable_apicv=1 options kvm-intel ept=1
Now if you are using ubuntu distro in ubuntu, you could try and use the command sudo kvm-ok to check if kvm is working and active.
also i think its not sudo modprobe kvm_intel its actually, its actually sudo modprobe kvm
My WSL2 kernel menuconfig says KVM is built-in but kvm-intel is a module, but anyway I tried
$ cat /etc/modprobe.d/kvm-nested.conf options kvm-intel nested=1 options kvm-intel enable_shadow_vmcs=1 options kvm-intel enable_apicv=1 options kvm-intel ept=1 $ sudo modprobe kvm $ sudo modprobe kvm-intel modprobe: ERROR: could not insert 'kvm_intel': Operation not supported $ cat /sys/module/kvm_intel/parameters/nested cat: /sys/module/kvm_intel/parameters/nested: No such file or directory
Just incase it helps, here is my kernel & wslconfig file.
https://github.com/ameeno/wsl2-kernel-settings
You can put the kernel in d:\wsl\kernel\ and the .wslconfig file into %USERPROFILE%
alternatively, you can place the kernel anywhere you wish, but remember to update the .wslconfig file.
after the items are in the right place, in a powershell/cmd window do wsl --shutdown
and after a few seconds do wsl
.
once you are in your wsl distro try uname -a
. if you see kernel 5.9.1-microsoft-standard-wsl2 then you are good.
PS i have only tested my nested on AMD but I believe I built modules for both intel and AMD at build time.
also make sure you do touch /dev/kvm
and also try sudo kvm-ok
what does it tell you?
@lbmeng As per release notes nested virtualization was enabled by default since Dev build 20175.
Latest kernel 4.19.128 from Windows Update is already ready for KVM but in v2004 the WSL2 lightweight VM is always created without nested virtualization support (ExposeVirtualizationExtensions), so KVM won't work even with nestedVirtualization=true in the .wslconfig
file.
If you are in v2004 (build 19041) with Intel CPU you can still try the trick from this thread. This forces the ExposeVirtualizationExtensions to the WSL2 VM and thus KVM works.
As per release notes nested virtualization was enabled by default since Dev build 20175.
Latest kernel 4.19.128 from Windows Update is already ready for KVM but in v2004 the WSL2 lightweight VM is always created without nested virtualization support (ExposeVirtualizationExtensions), so KVM won't work even with nestedVirtualization=true in the
.wslconfig
file.If you are in v2004 (build 19041) with Intel CPU you can still try the trick from this thread. This forces the ExposeVirtualizationExtensions to the WSL2 VM and thus KVM works.
@onomatopellan Thank you very much. Indeed the trick solved the issue. The only thing I noticed that in order to get the WinDbg patch vmcompute.exe successfully we have to wait for WinDbg to complete the process of downloading symbols from Microsoft server otherwise WSL cannot be started.
$ sudo modprobe kvm-intel
[sudo] password for test:
$ lsmod
Module Size Used by
kvm_intel 233472 0
$ cat /sys/module/kvm_intel/parameters/nested
Y
Greetings,
I am trying to do Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true
On the wsl2vm itself so I can use KVM inside it.
However, I cannot do that, and I don't know why Get-VM does not see the wsl2vm so I cannot expose these vars.
how do I do it?