search

microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.27k stars 812 forks source link

Nested Virtualization for WSL2 VM #4193

Closed ameeno closed 3 years ago

ameeno commented 5 years ago

Greetings,

I am trying to do Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true

On the wsl2vm itself so I can use KVM inside it.

However, I cannot do that, and I don't know why Get-VM does not see the wsl2vm so I cannot expose these vars.

how do I do it?

lbmeng commented 3 years ago

Just incase it helps, here is my kernel & wslconfig file. https://github.com/ameeno/wsl2-kernel-settings

You can put the kernel in d:\wsl\kernel and the .wslconfig file into %USERPROFILE%

alternatively, you can place the kernel anywhere you wish, but remember to update the .wslconfig file.

after the items are in the right place, in a powershell/cmd window do wsl --shutdown

and after a few seconds do wsl. once you are in your wsl distro try uname -a. if you see kernel 5.9.1-microsoft-standard-wsl2 then you are good. PS i have only tested my nested on AMD but I believe I built modules for both intel and AMD at build time.

@ameeno

As @onomatopellan mentioned that for the official Win10 v2004 it still cannot support nested virtualization out of the box on either Intel or AMD. The trick has to be used in order to get KVM work.

Anyway, thank you all the same!

ameeno commented 3 years ago

@lbmeng @onomatopellan glad you guys got it sorted.

I use insiders build (as an AMD user) so for me the settings worked great 👍

Never tried nested on 2004 as i have an AMD cpu and knew it would not work.

Nesci28 commented 3 years ago

@ameeno would you mind sharing your boot script to launch the ubuntu QEMU from inside WSL2?

qemu-system-x86_64 \
    -drive file=ubuntu.qcow2,format=qcow2 \
    -drive file=groovy-desktop-amd64.iso,media=cdrom,readonly \
    -net nic -net user \
    -m 5172 \
    -vga qxl \
    --enable-kvm \
    -smp 4 \
    -cpu kvm64,+vmx,+vme,+msr,+x2apic,+hypervisor,+aes,+vmx-activity-hlt,+vmx-cr3-load-noexit,+vmx-cr3-store-noexit,+vmx-cr8-load-exit,+vmx-cr8-store-exit,+vmx-desc-exit,+vmx-entry-ia32e-mode,+vmx-entry-load-efer,+vmx-entry-load-pat,+vmx-entry-noload-debugctl,+vmx-ept,+vmx-ept-1gb,+vmx-ept-2mb,+vmx-ept-execonly,+vmx-eptad,+vmx-exit-ack-intr,+vmx-exit-load-efer,+vmx-exit-load-pat,+vmx-exit-nosave-debugctl,+vmx-exit-save-efer,+vmx-exit-save-pat,+vmx-exit-save-preemption-timer,+vmx-flexpriority,+vmx-hlt-exit,+vmx-intr-exit,+vmx-invept,+vmx-invept-all-context,+vmx-invept-single-context,+vmx-invept-single-context,+vmx-invept-single-context-noglobals,+vmx-invlpg-exit,+vmx-invvpid,+vmx-invvpid-all-context,+vmx-invvpid-single-addr,+vmx-io-bitmap,+vmx-io-exit,+vmx-monitor-exit,+vmx-movdr-exit,+vmx-msr-bitmap,+vmx-mwait-exit,+vmx-nmi-exit,+vmx-page-walk-4,+vmx-pause-exit,+vmx-pml,+vmx-preemption-timer,+vmx-rdpmc-exit,+vmx-rdtsc-exit,+vmx-secondary-ctls,+vmx-shadow-vmcs,+vmx-store-lma,+vmx-true-ctls,+vmx-tsc-offset,+vmx-unrestricted-guest,+vmx-vintr-pending,+vmx-vmwrite-vmexit-fields,+vmx-vnmi,+vmx-vnmi-pending,+vmx-vpid,+de,+pse,+tsc,+msr,+pae,+mce,+cx8,+apic,+sep,+mtrr,+pge,+mca,+cmov,+pat,+pse36,+clflush,+mmx,+fxsr,+sse,+sse2,+ss,+ht,+syscall,+nx,+pdpe1gb,+rdtscp,+lm,+pni,+pclmulqdq,+vmx,+ssse3,+fma,+cx16,+pcid,+sse4_1,+sse4_2,+movbe,+popcnt,+aes,+xsave,+avx,+f16c,+rdrand,+hypervisor,+lahf_lm,+abm,+3dnowprefetch,+ssbd,+ibpb,+stibp,+fsgsbase,+bmi1,+avx2,+smep,+bmi2,+erms,+invpcid,+rdseed,+adx,+smap,+clflushopt,+xsaveopt,+xsavec,+xgetbv1,+xsaves \

With this one, my nested virtualization doesnt work. When I take out the -cpu flags, the virtualization works but the QEMU screen flickers and QEMU crash after a couple of seconds.

Kagami commented 3 years ago

https://www.kernel.org/doc/Documentation/virt/kvm/running-nested-guests.rst nested=y should be default for 4.20+ kernels.

But as far as I can understand you only need to enable this for L0 host (Hyper-V in case of WSL). For L1 (WSL Linux) you need to enable nesting only if you want to have KVM available in L2 and spawn L3 (i.e. nested-in-nested), L2 can be accelerated because of nested virtualization support in L0, L1 nesting option doesn't change that.

mayohn commented 3 years ago

WSL2 Linux内核版本v4.19.104中有一些更改。配置文件已启用某些KVM功能。因此，现在可以编译内核并进行尝试。无需任何骇人听闻的步骤。

不行啊，老铁，我试了，无效

spersico commented 3 years ago

Do we have a timeline or some sort of idea of when AMD's Linux KVM guest support is going to get on the stable release? I want to try this out, but I don't want to get into the Windows Insider rings, I've had a bad experience with those.

With my Ryzen 5 1600 and my compiled kernel at 5.4.91, this still doesn't seems to work: image

ameeno commented 3 years ago

sorry I have not replied in a while. I have since moved my main rig away from windows insiders and virtualised win10 pro inside a proxmox VM. it’s a shame really./ nested amd is a must for me and I can't keep relying on an ever-updating insider's windows build to get a basic processor feature.

It’s a shame really. windows have the potential to be the greatest of all dev machines with wsl2, but not having nested available on AMD out of the box is a big issue. I figured I would use Insiders for a few months until the feature went public release. That was almost a year ago now and I do not see MS making nested on AMD a standard feature until way after 20h2.

Simbaclaws commented 3 years ago

Maybe the official 21H2 build in June will have this feature enabled. I'm hoping someone from microsoft can officially confirm this. They did announce AMD nested virt for hyper-v if I'm not mistaken, so going by that logic I think WSL will get it as well.

https://techcommunity.microsoft.com/t5/virtualization/amd-nested-virtualization-support/ba-p/1434841

Build version 19636 < 21H2 I suppose?

PLEASE make this a thing.

therealkenc commented 3 years ago

AMD nested virt for hyper-v

Note that Tech Community announcement mentions (quoth) "Linux KVM guest support will be coming in the future". The best place to plead would be the User Voice, which is still open. WSL2 will inherit whatever functionality is available from hypervisor platform.

Simbaclaws commented 3 years ago

Apparently 21H1 still doesn't have support for KVM AMD nested virtual machines. Let's hope 21H2 does... (which was supposed to be in june :/)

Apparently the public preview for 21H2 is there, but only for the azure cloud instances...

I just installed the update and build the WSL kernel from source.

I'm still getting:

INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used

Why is it that only intel cpu's are supported for this in windows? I'm not willing to switch my entire desktop build just to be able to get nested virt....

Also the link you send for the User Voice was deleted...

AMD nested virt for hyper-v

Note that Tech Community announcement mentions (quoth) "Linux KVM guest support will be coming in the future". The best place to plead would be the User Voice, which is still open. WSL2 will inherit whatever functionality is available from hypervisor platform.

ameeno commented 3 years ago

You need to be on Dev branch for AMD KVM. But that is rapidly moving to windows 11 and lots is broken. It may be possible to build a Dev iso from a particular build version and then freeze updates to use. But I believe it would expire after 180 days

Simbaclaws commented 3 years ago

yeah I'm just hoping for official releases. I don't like using dev builds for it. If it's in windows 11, I can wait for that as well :)

cyraid commented 3 years ago

There's gotta be a better way to keep my wsl2 up to date rather than converting my entire OS (that's already pretty established) to a new insider ring, etc. Can I just download a new package of WSL2 somewhere?

jseguillon commented 3 years ago

I can confirm this works in Win 11. I created a hyperv VM on my Windows 10 with ExposeVirtualizationExtension and was able to launch a nested VM with Kubevirt. A lot of inception but works great: Win10 -> HyperV Win11 -> WSL2 -> Minikube -> Kubvevirt 😊 EDIT: Intel cpu

crramirez commented 3 years ago

I can confirm this works in Win 11. I created a hyperv VM on my Windows 10 with ExposeVirtualizationExtension and was able to launch a nested VM with Kubevirt. A lot of inception but works great: Win10 -> HyperV Win11 -> WSL2 -> Minikube -> Kubvevirt 😊

Intel or AMD ?

jseguillon commented 3 years ago

@crramirez : Intel (edited my comment)

crramirez commented 2 years ago

Hello,

I tested with Windows 11 released version and the latest kernel update (not the WSL Store Preview) and I can confirm that the nested virtualization for AMD is enabled and working by default. The kvm_amd module is included in the kernel and started with nested=1. I tested it using livemedia-creator with qemu and the time to process dropped by half when yesterday I upgraded from Windows 10 to 11.

Regards

NNTin commented 2 years ago

@startergo You didn't eventually compile the kernel?

I have put the same WSL2 config:

[wsl2]
nestedVirtualization=true
debugConsole=true
pageReporting=true
kernelCommandLine=intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1

and I get:

cat /sys/module/kvm_intel/parameters/nested
Y

,but at the same time I get this:

dmesg | grep kvm
[    0.000000] Command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[    0.119133] Kernel command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[    0.330695] kvm: no hardware support
[    0.330696] kvm: no hardware support

and

egrep -c '(vmx|svm)' /proc/cpuinfo
0

My CPU supports both VTx and VTd.

I also got this far. For future people if you also got this far try enabling Windows Hypervisor Platform. https://superuser.com/questions/1510172/hyper-v-vs-virtual-machine-platform-vs-windows-hypervisor-platform-settings-in-p I had Hyper-V enabled. In BIOS Virtualization Technology an VT-d Feature were enabled but I kept getting 

[    0.330695] kvm: no hardware support
[    0.330696] kvm: no hardware support

After enabling Windows Hypervisor platform I finally got

[    0.193019] KVM: vmx: using Hyper-V Enlightened VMCS
[    0.195115] kvm: already loaded the other module
udovenkoav1981 commented 2 years ago

I have AMD 4800H, windows 11 21h2 (22000.856), wsl kernel 5.10.102.1-microsoft-standard-WSL2 dmesg | grep kvm [ 0.265398] kvm: no hardware support [ 0.266298] kvm: Nested Virtualization enabled [ 0.266306] SVM: kvm: Nested Paging enabled

cat /sys/module/kvm_amd/parameters/nested 1

nested virtualization NOT working, Qemu is very slow

CMingTseng commented 1 year ago

我可以確認這在 Win 11 中有效。我在Windows 10上創建了一個hyperv VM，並且能夠使用Kubevirt啟動一個嵌套VM。很多開始，但效果很好：ExposeVirtualizationExtension``Win10 -> HyperV Win11 -> WSL2 -> Minikube -> Kubvevirt 😊編輯：英特爾中央處理器 Dear Sir

How enable nested virtualization/KVm at Window 10 22H1/22H2 ?

THX

CMingTseng commented 1 year ago

@startergo你最終沒有編譯內核嗎？ 我已經放置了相同的 WSL2 配置：

[wsl2]
nestedVirtualization=true
debugConsole=true
pageReporting=true
kernelCommandLine=intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1

我得到：

cat /sys/module/kvm_intel/parameters/nested
Y

，但同時我得到這個：

dmesg | grep kvm
[    0.000000] Command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[    0.119133] Kernel command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[    0.330695] kvm: no hardware support
[    0.330696] kvm: no hardware support

和

egrep -c '(vmx|svm)' /proc/cpuinfo
0

我的 CPU 同時支援 VTx 和 VTd。

我也走到了這一步。對於未來的人們，如果你也走到了這一步，請嘗試啟用Windows虛擬機監控程序平臺。https://superuser.com/questions/1510172/hyper-v-vs-virtual-machine-platform-vs-windows-hypervisor-platform-settings-in-p 我啟用了Hyper-V。在 BIOS 虛擬化技術中啟用了 VT-d 功能，但我不斷得到

[    0.330695] kvm: no hardware support
[    0.330696] kvm: no hardware support

啟用Windows虛擬機管理程序平臺后，我終於得到了

[    0.193019] KVM: vmx: using Hyper-V Enlightened VMCS
[    0.195115] kvm: already loaded the other module

Dear Sir

Do you use Windows 10 22H1/22H2 ?

it can run https://github.com/budtmo/docker-android ?

THX

startergo commented 1 year ago

I have not used it in a while.

CMingTseng commented 1 year ago

@startergo & all

as this post https://learn.microsoft.com/zh-tw/windows/wsl/wsl-config

the ms still disable "nestedVirtualization" at win10

the laste version WSL (download from store)

at Win11 it can run nestedVirtualization

at win10 still not work

PS C:\Users\Lan> wsl --version WSL 版本： 1.0.3.0 核心版本： 5.15.79.1 WSLg 版本： 1.0.47 MSRDC 版本： 1.2.3575 Direct3D 版本： 1.606.4 DXCore 版本： 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows版本： 10.0.19045.2486

WinDbg PreView error log

(dd0.17ac): Break instruction exception - code 80000003 (first chance) ntdll!DbgBreakPoint: 00007ffa`b5650bb0 cc int 3 0:006> bp vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq Couldn't resolve error at 'vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq' onecore\com\combase\dcomrem\call.cxx(4236)\combase.dll!00007FFAB379B3AC: (caller: 00007FFAB379A208) ReturnHr(4) tid(1754) 800706BA RPC 伺服器無法使用。 (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance)

THX

crramirez commented 1 year ago

@CMingTseng, by any chance do you have an AMD processor? Because nested virtualization with AMD is not supported at all in Windows 10

ZeppLu commented 1 year ago

@startergo & all

as this post https://learn.microsoft.com/zh-tw/windows/wsl/wsl-config

the ms still disable "nestedVirtualization" at win10

the laste version WSL (download from store)

at Win11 it can run nestedVirtualization

at win10 still not work

PS C:\Users\Lan> wsl --version WSL 版本： 1.0.3.0 核心版本： 5.15.79.1 WSLg 版本： 1.0.47 MSRDC 版本： 1.2.3575 Direct3D 版本： 1.606.4 DXCore 版本： 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows版本： 10.0.19045.2486

WinDbg PreView error log

(dd0.17ac): Break instruction exception - code 80000003 (first chance) ntdll!DbgBreakPoint: 00007ffa`b5650bb0 cc int 3 0:006> bp vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq Couldn't resolve error at 'vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq' onecore\com\combase\dcomrem\call.cxx(4236)\combase.dll!00007FFAB379B3AC: (caller: 00007FFAB379A208) ReturnHr(4) tid(1754) 800706BA RPC 伺服器無法使用。 (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance)

THX

Indeed, the WinDBG hack still works on Windows 10, but only if you don't update wsl.

In short, if wsl --version shows versioning info, then you are bad to go. You should consider downgrading it by:

$pkg = Get-AppxPackage MicrosoftCorporationII.WindowsSubsystemforLinux -AllUsers
Remove-AppxPackage $pkg -AllUsers

Then the WinDBG hack should work as expected!

sarim commented 1 year ago

So I'm getting very slow performance using libvirt inside WSL. It is using nested virtualization. But still its slow, especially loading of grub, the loading kernel, loading initrd steps. Do others encountered similar issues?

wsl --version
WSL version: 1.3.17.0
Kernel version: 5.15.90.4-1
WSLg version: 1.0.57
MSRDC version: 1.2.4485
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25880.1000-230602-1350.main
Windows version: 10.0.22621.2215
dmesg | grep kvm
[    0.156436] kvm: no hardware support
[    0.329847] kvm: Nested Virtualization enabled
[    0.329851] SVM: kvm: Nested Paging enabled
[    0.329852] SVM: kvm: Hyper-V enlightened NPT TLB flush enabled
[    0.329853] SVM: kvm: Hyper-V Direct TLB Flush enabled

egrep -c '(vmx|svm)' /proc/cpuinfo
16

sudo virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

I'm running a simple debian image: debian-12-genericcloud-amd64.qcow2 It takes several minutes to load.

Simbaclaws commented 1 year ago

So I'm getting very slow performance using libvirt inside WSL. It is using nested virtualization. But still its slow, especially loading of grub, the loading kernel, loading initrd steps. Do others encountered similar issues?

wsl --version
WSL version: 1.3.17.0
Kernel version: 5.15.90.4-1
WSLg version: 1.0.57
MSRDC version: 1.2.4485
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25880.1000-230602-1350.main
Windows version: 10.0.22621.2215
dmesg | grep kvm
[    0.156436] kvm: no hardware support
[    0.329847] kvm: Nested Virtualization enabled
[    0.329851] SVM: kvm: Nested Paging enabled
[    0.329852] SVM: kvm: Hyper-V enlightened NPT TLB flush enabled
[    0.329853] SVM: kvm: Hyper-V Direct TLB Flush enabled

egrep -c '(vmx|svm)' /proc/cpuinfo
16

sudo virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

I'm running a simple debian image: debian-12-genericcloud-amd64.qcow2 It takes several minutes to load.

hey @sarim ,

What kind of hardware is this running on? Perhaps the cpu is missing specific instruction sets for virtualization acceleration. Since kvm is reporting no hardware support?

That's just from a quick glance through your supplied log.

Hopefully that can be helpful.

Other then that, I currently don't have the time/resources to validate your report. If anyone else can jump in to see wether this is reproducible?

sarim commented 1 year ago

@Simbaclaws Thanks for taking a look. The cpu is AMD Ryzen 7 3700X. It has full support for virtualization.

Ouput of lscpu from wsl.

lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 7 3700X 8-Core Processor
    CPU family:          23
    Model:               113
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            7200.03
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pd
                         pe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 m
                         ovbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoex
                         t perfctr_core ssbd ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1
                          xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmloa
                         d umip rdpid
Virtualization features:
  Virtualization:        AMD-V
  Hypervisor vendor:     Microsoft
  Virtualization type:   full
Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    16 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT enabled with STIBP protection
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected
Simbaclaws commented 1 year ago

@Simbaclaws Thanks for taking a look. The cpu is AMD Ryzen 7 3700X. It has full support for virtualization.

Ouput of lscpu from wsl.

lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 7 3700X 8-Core Processor
    CPU family:          23
    Model:               113
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            7200.03
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pd
                         pe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 m
                         ovbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoex
                         t perfctr_core ssbd ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1
                          xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmloa
                         d umip rdpid
Virtualization features:
  Virtualization:        AMD-V
  Hypervisor vendor:     Microsoft
  Virtualization type:   full
Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    16 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT enabled with STIBP protection
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

@sarim

perhaps some specific flags for acceleration are missing.

you can use this stackoverflow answer to do a quick test what is and is not supported by your cpu:

https://stackoverflow.com/questions/66277749/how-do-i-get-every-supported-cpu-flags-as-qemu-cpu-flags

not sure if that could yield any potential results.

you'll probably want to enable all flags for your cpu model in order to get better performance.

sarim commented 1 year ago

@Simbaclaws I'm already using cpu mode=host-passthrough in libvirt. So it already has all the flags of host cpu. So I don't think it would yield any beneficial result :/

CMingTseng commented 1 year ago

So I'm getting very slow performance using libvirt inside WSL. It is using nested virtualization. But still its slow, especially loading of grub, the loading kernel, loading initrd steps. Do others encountered similar issues?

wsl --version
WSL version: 1.3.17.0
Kernel version: 5.15.90.4-1
WSLg version: 1.0.57
MSRDC version: 1.2.4485
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25880.1000-230602-1350.main
Windows version: 10.0.22621.2215
dmesg | grep kvm
[    0.156436] kvm: no hardware support
[    0.329847] kvm: Nested Virtualization enabled
[    0.329851] SVM: kvm: Nested Paging enabled
[    0.329852] SVM: kvm: Hyper-V enlightened NPT TLB flush enabled
[    0.329853] SVM: kvm: Hyper-V Direct TLB Flush enabled

egrep -c '(vmx|svm)' /proc/cpuinfo
16

sudo virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

I'm running a simple debian image: debian-12-genericcloud-amd64.qcow2 It takes several minutes to load. Dear Sir

Do you run WSL2 at WIn11 ? or WIn 10

THX

Simbaclaws commented 1 year ago

@Simbaclaws I'm already using cpu mode=host-passthrough in libvirt. So it already has all the flags of host cpu. So I don't think it would yield any beneficial result :/

@sarim

you're right, host-passthrough should already do those things.

what about trying to get the cgroup backends available?

Perhaps you need cgroups v2.

Maybe have a look here? https://stackoverflow.com/questions/73021599/how-to-enable-cgroup-v2-in-wsl2

CMingTseng commented 1 year ago

@startergo & all as this post https://learn.microsoft.com/zh-tw/windows/wsl/wsl-config the ms still disable "nestedVirtualization" at win10 the laste version WSL (download from store) at Win11 it can run nestedVirtualization at win10 still not work PS C:\Users\Lan> wsl --version WSL 版本： 1.0.3.0 核心版本： 5.15.79.1 WSLg 版本： 1.0.47 MSRDC 版本： 1.2.3575 Direct3D 版本： 1.606.4 DXCore 版本： 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows版本： 10.0.19045.2486 WinDbg PreView error log (dd0.17ac): Break instruction exception - code 80000003 (first chance) ntdll!DbgBreakPoint: 00007ffa`b5650bb0 cc int 3 0:006> bp vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq Couldn't resolve error at 'vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq' onecore\com\combase\dcomrem\call.cxx(4236)\combase.dll!00007FFAB379B3AC: (caller: 00007FFAB379A208) ReturnHr(4) tid(1754) 800706BA RPC 伺服器無法使用。 (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance) THX

Indeed, the WinDBG hack still works on Windows 10, but only if you don't update wsl.

In short, if wsl --version shows versioning info, then you are bad to go. You should consider downgrading it by:

$pkg = Get-AppxPackage MicrosoftCorporationII.WindowsSubsystemforLinux -AllUsers
Remove-AppxPackage $pkg -AllUsers

Then the WinDBG hack should work as expected!

@ZeppLu

so we can not upgrade WSL2 version ?

THX

CMingTseng commented 1 year ago

2. Pass ExposeVirtualizationExtensions to the utility VM and boot a different kernel. Dear Sir

At WIndows 10 use the WSL2 (kernel version 5.15.79.1 )

How Pass ExposeVirtualizationExtensions to the utility VM and boot a different kernel. ?

THX

sarim commented 1 year ago

@CMingTseng windows 11. you can see my windows version in wsl --version output.

@Simbaclaws I actually already have cgroups v2 enabled. kernel command line to disable cgroupsv1 and mount cgroupsv2. I use it to use podman.

Simbaclaws commented 1 year ago

@CMingTseng windows 11. you can see my windows version in wsl --version output.

@Simbaclaws I actually already have cgroups v2 enabled. kernel command line to disable cgroupsv1 and mount cgroupsv2. I use it to use podman.

@sarim

I'm kind of running out of ideas at this point.

I do see the error message in your log saying it's missing the cgroup backend, but honestly don't know whether fixing it would yield better performance or not.

Hopefully someone else can come up with a solution to your problem.

I can not reproduce this right now since I'm on vacation.

sarim commented 1 year ago

@Simbaclaws Thanks for taking a look. But I suspect It could be bug with WSL, so I was seeking other people's experience with it. If other people's VM's are running fast, then it would be plausible to debug my configuration.

startergo commented 6 months ago

Indeed, the WinDBG hack still works on Windows 10, but only if you don't update wsl.

In short, if wsl --version shows versioning info, then you are bad to go. You should consider downgrading it by:

$pkg = Get-AppxPackage MicrosoftCorporationII.WindowsSubsystemforLinux -AllUsers
Remove-AppxPackage $pkg -AllUsers

Then the WinDBG hack should work as expected!

Correct! Thanks for the input. So, after applying the above version downgrade the WinDbg hack works.