search

xcp-ng / xcp

Entry point for issues and wiki. Also contains some scripts and sources.
https://xcp-ng.org
1.26k stars 74 forks source link

Domain crash when using nested virtualization #562

Closed jippi closed 4 months ago

jippi commented 2 years ago

Hello there,

I'm trying to run a KVM machine inside a Xen guest with "Nested Virtualisation" enabled.

When I start a KVM gust inside my Xen guest, the Xen guest immediately gets shut down my Xen and the error domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_update_guest_cr+0x516/0x6a0 (more details in log output section below).

The Xen guest is running fine up until I start a KVM guest in the Xen guest.

This behaviour is true for both my attempted macOS install, but also for Ubuntu and Debian KVM guests, so seems unlikely to be something specific to my original intent of virtualising macOS (server).

Not really sure what information y'all need to dig into this kind of issue, so I've tried to collect everything I could think of (with my limited ~2mo usage of Xen / XCP-ng)

The Xen guest is running Management agent 7.20.0-9

Various command outputs

$ xe vm-param-list uuid=66fc99e7-d4cb-3d2d-987e-875523b25e45

``` $ xe vm-param-list uuid=66fc99e7-d4cb-3d2d-987e-875523b25e45 uuid ( RO) : 66fc99e7-d4cb-3d2d-987e-875523b25e45 name-label ( RW): macOS-kvm name-description ( RW): macOS-kvm user-version ( RW): 1 is-a-template ( RW): false is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): parent ( RO): children ( RO): is-control-domain ( RO): false power-state ( RO): running memory-actual ( RO): 8589934592 memory-target ( RO): 0 memory-overhead ( RO): 73400320 memory-static-max ( RW): 8589934592 memory-dynamic-max ( RW): 8589934592 memory-dynamic-min ( RW): 4294967296 memory-static-min ( RW): 536870912 suspend-VDI-uuid ( RW): suspend-SR-uuid ( RW): VCPUs-params (MRW): VCPUs-max ( RW): 4 VCPUs-at-startup ( RW): 4 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): 6308be4b-4173-4d25-9263-3dade14e8634 hvm ( RO): true platform (MRW): timeoffset: 0; device-model: qemu-upstream-compat; secureboot: false; videoram: 16; exp-nested-hvm: true; vga: std; hpet: true; apic: true; device_id: 0001; nx: true; pae: true; viridian: false; acpi: 1 allowed-operations (SRO): changing_dynamic_range; migrate_send; pool_migrate; changing_VCPUs_live; suspend; hard_reboot; hard_shutdown; clean_reboot; clean_shutdown; pause; checkpoint; snapshot current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 648919d1-a841-401b-a408-f97db904b3c1 domain-type ( RW): hvm current-domain-type ( RO): hvm HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: c HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-f7fa3223-2c100800-00000121-0000000f-009c47ab-00000008-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000 last-boot-record ( RO): '' resident-on ( RO): 648919d1-a841-401b-a408-f97db904b3c1 affinity ( RW): other-config (MRW): base_template_name: Ubuntu Focal Fossa 20.04; import_task: OpaqueRef:d19e91d5-f217-4f60-86dd-8abb41464fba; mac_seed: 7cfb2795-4c1d-2bec-8775-d36f15cc20f6; install-methods: cdrom,nfs,http,ftp; linux_template: true dom-id ( RO): 36 recommendations ( RO): xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): 20220805T20:16:18Z install-time ( RO): 20220805T19:09:54Z VCPUs-number ( RO): 4 VCPUs-utilisation (MRO): 0: 0.004; 1: 0.004; 2: 0.006; 3: 0.007 os-version (MRO): name: Ubuntu 22.04.1 LTS; uname: 5.15.0-43-generic; distro: ubuntu; major: 22; minor: 04 PV-drivers-version (MRO): major: 7; minor: 20; micro: 0; build: 9 PV-drivers-up-to-date ( RO) [DEPRECATED]: true memory (MRO): disks (MRO): VBDs (SRO): 995ca6c8-a55d-1b3d-65d9-2ec3e0d03f7b; b8b13607-6de9-fe30-422e-faec06e5328f networks (MRO): 0/ip: 192.168.20.123; 0/ipv4/0: 192.168.20.123; 0/ipv6/0: fe80::ac62:ad9e:1121:fbf9 PV-drivers-detected ( RO): true other (MRO): feature-balloon: 1; feature-vcpu-hotplug: 1; feature-reboot: 1; feature-poweroff: 1; feature-suspend: 1; has-vendor-device: 0; platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1 live ( RO): true guest-metrics-last-updated ( RO): 20220805T20:17:44Z can-use-hotplug-vbd ( RO): unspecified can-use-hotplug-vif ( RO): unspecified cooperative ( RO) [DEPRECATED]: true tags (SRW): appliance ( RW): snapshot-schedule ( RW): is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): hardware-platform-version ( RO): 0 has-vendor-device ( RW): false requires-reboot ( RO): false reference-label ( RO): ubuntu-20.04 bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d ```

$ cat /sys/hypervisor/properties/capabilities

``` $ cat /sys/hypervisor/properties/capabilities xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 ```

$ xe diagnostic-vm-status uuid=66fc99e7-d4cb-3d2d-987e-875523b25e45

``` $ xe diagnostic-vm-status uuid=66fc99e7-d4cb-3d2d-987e-875523b25e45 uuid ( RO) : 66fc99e7-d4cb-3d2d-987e-875523b25e45 name-label ( RW): macOS-kvm power-state ( RO): running possible-hosts ( RO): 648919d1-a841-401b-a408-f97db904b3c1 Checking to see whether disks are attachable uuid ( RO) : 995ca6c8-a55d-1b3d-65d9-2ec3e0d03f7b vdi-uuid ( RO): empty ( RO): true device ( RO): xvdd userdevice ( RW): 3 mode ( RW): RO type ( RW): CD attachable ( RO): true storage-lock ( RO): false uuid ( RO) : b8b13607-6de9-fe30-422e-faec06e5328f vdi-uuid ( RO): bbe0910f-4f8f-4321-9c28-f183e95a3e62 empty ( RO): false device ( RO): xvda userdevice ( RW): 0 mode ( RW): RW type ( RW): Disk attachable ( RO): true storage-lock ( RO): false Checking to see whether VM can boot on each host Nimbus : OK ```

$ xl info

``` $ xl info host : xen-nimbus release : 4.19.0+1 version : #1 SMP Thu Jan 13 12:55:45 CET 2022 machine : x86_64 nr_cpus : 16 max_cpu_id : 15 nr_nodes : 1 cores_per_socket : 8 threads_per_core : 2 cpu_mhz : 2904.033 hw_caps : bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c67af:00000008:00000100 virt_caps : pv hvm hvm_directio pv_directio hap shadow iommu_hap_pt_share total_memory : 64892 free_memory : 1593 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 13 xen_extra : .4-9.24.1 xen_version : 4.13.4-9.24.1 xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : 6e2fc128eb1a, pq 61b8eb0f0db4 xen_commandline : dom0_mem=4272M,max:4272M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 sched-gran=cpu cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28) cc_compile_by : mockbuild cc_compile_domain : [unknown] cc_compile_date : Wed Jul 13 09:56:11 CEST 2022 build_id : c22784fa0790531ccac1d0717cd9cfd83a78ac27 xend_config_format : 4 ```

$ cat /proc/cpuinfo (in xcp-ng vm)

``` $ cat /proc/cpuinfo processor : 15 vendor_id : GenuineIntel cpu family : 6 model : 165 model name : Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz stepping : 5 microcode : 0xf0 cpu MHz : 2904.032 cache size : 16384 KB physical id : 0 siblings : 16 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt xsaveopt xsavec xgetbv1 arch_capabilities bugs : spectre_v1 spectre_v2 spec_store_bypass bogomips : 5808.06 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: ```

$ xl dmesg

From multiple crashes, included multiple in case it would show y'all something useful ``` $ xl dmesg XEN) [ 2611.733209] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_update_guest_cr+0x516/0x6a0 (XEN) [ 2611.733210] Domain 18 (vcpu#0) crashed on cpu#10: (XEN) [ 2611.733212] ----[ Xen-4.13.4-9.24.1 x86_64 debug=n Not tainted ]---- (XEN) [ 2611.733213] CPU: 10 (XEN) [ 2611.733213] RIP: 0010:[] (XEN) [ 2611.733214] RFLAGS: 0000000000000002 CONTEXT: hvm guest (d18v0) (XEN) [ 2611.733215] rax: 0000000000000660 rbx: 00000000756e6547 rcx: 0000000000000000 (XEN) [ 2611.733216] rdx: 0000000000000000 rsi: 00000000fffd1844 rdi: 0000000000005042 (XEN) [ 2611.733217] rbp: 00000000fffcc000 rsp: ffffb89b46ad7bd0 r8: 0000000000000000 (XEN) [ 2611.733217] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [ 2611.733218] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 2611.733218] r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) [ 2611.733219] cr3: 0000000109d4e003 cr2: 0000000000000000 (XEN) [ 2611.733219] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 2611.733220] ds: 0008 es: 0008 fs: 0008 gs: 0008 ss: 0008 cs: 0010 (XEN) [ 2803.225273] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_update_guest_cr+0x516/0x6a0 (XEN) [ 2803.225274] Domain 20 (vcpu#3) crashed on cpu#15: (XEN) [ 2803.225275] ----[ Xen-4.13.4-9.24.1 x86_64 debug=n Not tainted ]---- (XEN) [ 2803.225276] CPU: 15 (XEN) [ 2803.225276] RIP: 0010:[] (XEN) [ 2803.225277] RFLAGS: 0000000000000002 CONTEXT: hvm guest (d20v3) (XEN) [ 2803.225278] rax: 0000000000000660 rbx: 00000000756e6547 rcx: 0000000000000000 (XEN) [ 2803.225279] rdx: 0000000000000000 rsi: 00000000fffd1844 rdi: 0000000000005042 (XEN) [ 2803.225280] rbp: 00000000fffcc000 rsp: ffffb18a05917bf0 r8: 0000000000000000 (XEN) [ 2803.225280] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [ 2803.225281] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 2803.225281] r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) [ 2803.225282] cr3: 0000000112e34001 cr2: 0000000000000000 (XEN) [ 2803.225282] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 2803.225283] ds: 0008 es: 0008 fs: 0008 gs: 0008 ss: 0008 cs: 0010 (XEN) [ 3158.419555] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_update_guest_cr+0x516/0x6a0 (XEN) [ 3158.419556] Domain 21 (vcpu#2) crashed on cpu#14: (XEN) [ 3158.419558] ----[ Xen-4.13.4-9.24.1 x86_64 debug=n Not tainted ]---- (XEN) [ 3158.419558] CPU: 14 (XEN) [ 3158.419559] RIP: 0010:[] (XEN) [ 3158.419560] RFLAGS: 0000000000000002 CONTEXT: hvm guest (d21v2) (XEN) [ 3158.419561] rax: 0000000000000660 rbx: 00000000756e6547 rcx: 0000000000000000 (XEN) [ 3158.419562] rdx: 0000000000000000 rsi: 00000000fffd1844 rdi: 0000000000005042 (XEN) [ 3158.419562] rbp: 00000000fffcc000 rsp: ffffb6c241987c50 r8: 0000000000000000 (XEN) [ 3158.419563] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [ 3158.419563] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 3158.419564] r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) [ 3158.419565] cr3: 00000001130e4004 cr2: 0000000000000000 (XEN) [ 3158.419565] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 3158.419566] ds: 0008 es: 0008 fs: 0008 gs: 0008 ss: 0008 cs: 0010 (XEN) [ 3423.961388] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_update_guest_cr+0x516/0x6a0 (XEN) [ 3423.961389] Domain 24 (vcpu#1) crashed on cpu#5: (XEN) [ 3423.961391] ----[ Xen-4.13.4-9.24.1 x86_64 debug=n Not tainted ]---- (XEN) [ 3423.961392] CPU: 5 (XEN) [ 3423.961392] RIP: 0010:[] (XEN) [ 3423.961393] RFLAGS: 0000000000000002 CONTEXT: hvm guest (d24v1) (XEN) [ 3423.961394] rax: 0000000000000660 rbx: 00000000756e6547 rcx: 0000000000000000 (XEN) [ 3423.961395] rdx: 0000000000000000 rsi: 00000000fffd1844 rdi: 0000000000005042 (XEN) [ 3423.961396] rbp: 00000000fffcc000 rsp: ffffb70a81ec7bc0 r8: 0000000000000000 (XEN) [ 3423.961396] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [ 3423.961397] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 3423.961397] r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) [ 3423.961398] cr3: 0000000102ae4001 cr2: 0000000000000000 (XEN) [ 3423.961398] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 3423.961399] ds: 0008 es: 0008 fs: 0008 gs: 0008 ss: 0008 cs: 0010 (XEN) [ 4155.974455] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_update_guest_cr+0x516/0x6a0 (XEN) [ 4155.974457] Domain 33 (vcpu#2) crashed on cpu#9: (XEN) [ 4155.974458] ----[ Xen-4.13.4-9.24.1 x86_64 debug=n Not tainted ]---- (XEN) [ 4155.974459] CPU: 9 (XEN) [ 4155.974460] RIP: 0010:[] (XEN) [ 4155.974460] RFLAGS: 0000000000000002 CONTEXT: hvm guest (d33v2) (XEN) [ 4155.974461] rax: 0000000000000660 rbx: 00000000756e6547 rcx: 0000000000000000 (XEN) [ 4155.974462] rdx: 0000000000000000 rsi: 00000000fffd1844 rdi: 0000000000005042 (XEN) [ 4155.974463] rbp: 00000000fffcc000 rsp: ffffb1f401fefc40 r8: 0000000000000000 (XEN) [ 4155.974463] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [ 4155.974464] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 4155.974464] r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) [ 4155.974465] cr3: 00000001115b6001 cr2: 0000000000000000 (XEN) [ 4155.974465] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 4155.974466] ds: 0008 es: 0008 fs: 0008 gs: 0008 ss: 0008 cs: 0010 (XEN) [ 4651.736623] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_update_guest_cr+0x516/0x6a0 (XEN) [ 4651.736625] Domain 34 (vcpu#3) crashed on cpu#0: (XEN) [ 4651.736627] ----[ Xen-4.13.4-9.24.1 x86_64 debug=n Not tainted ]---- (XEN) [ 4651.736628] CPU: 0 (XEN) [ 4651.736628] RIP: 0010:[] (XEN) [ 4651.736629] RFLAGS: 0000000000000002 CONTEXT: hvm guest (d34v3) (XEN) [ 4651.736631] rax: 0000000000000660 rbx: 00000000756e6547 rcx: 0000000000000000 (XEN) [ 4651.736632] rdx: 0000000000000000 rsi: 00000000fffd1844 rdi: 0000000000005042 (XEN) [ 4651.736633] rbp: 00000000fffcc000 rsp: ffffb8e1c18bfc80 r8: 0000000000000000 (XEN) [ 4651.736633] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [ 4651.736634] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 4651.736635] r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) [ 4651.736635] cr3: 0000000118956005 cr2: 0000000000000000 (XEN) [ 4651.736636] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 4651.736637] ds: 0008 es: 0008 fs: 0008 gs: 0008 ss: 0008 cs: 0010 (XEN) [ 5194.432335] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_update_guest_cr+0x516/0x6a0 (XEN) [ 5194.432336] Domain 35 (vcpu#2) crashed on cpu#7: (XEN) [ 5194.432339] ----[ Xen-4.13.4-9.24.1 x86_64 debug=n Not tainted ]---- (XEN) [ 5194.432339] CPU: 7 (XEN) [ 5194.432340] RIP: 0008:[] (XEN) [ 5194.432341] RFLAGS: 0000000000000002 CONTEXT: hvm guest (d35v2) (XEN) [ 5194.432343] rax: 0000000000000020 rbx: 0000000003b4d000 rcx: 00000000f7fa3223 (XEN) [ 5194.432343] rdx: 0000000006000000 rsi: 000000000008b000 rdi: 0000000000000000 (XEN) [ 5194.432344] rbp: 0000000001000000 rsp: ffffbea68183bbb0 r8: 0000000000000000 (XEN) [ 5194.432345] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [ 5194.432346] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 5194.432347] r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002060 (XEN) [ 5194.432348] cr3: 00000001072d0005 cr2: 0000000000000000 (XEN) [ 5194.432348] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 5194.432349] ds: 0018 es: 0018 fs: 0018 gs: 0018 ss: 0018 cs: 0008 ```

$ dpkg -l | grep -i qemu (on KVM host)

``` $ dpkg -l|grep -i qemu ii ipxe-qemu 1.21.1+git-20220113.fbbdc3926-0ubuntu1 all PXE boot firmware - ROM images for qemu ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu4 all PXE boot firmware - Compat EFI ROM images for qemu ii libvirt-daemon-driver-qemu 8.0.0-1ubuntu7.1 amd64 Virtualization daemon QEMU connection driver ii qemu 1:6.2+dfsg-2ubuntu6.3 amd64 fast processor emulator, dummy package ii qemu-block-extra 1:6.2+dfsg-2ubuntu6.3 amd64 extra block backend modules for qemu-system and qemu-utils ii qemu-system-common 1:6.2+dfsg-2ubuntu6.3 amd64 QEMU full system emulation binaries (common files) ii qemu-system-data 1:6.2+dfsg-2ubuntu6.3 all QEMU full system emulation (data files) ii qemu-system-gui 1:6.2+dfsg-2ubuntu6.3 amd64 QEMU full system emulation binaries (user interface and audio support) ii qemu-system-x86 1:6.2+dfsg-2ubuntu6.3 amd64 QEMU full system emulation binaries (x86) ii qemu-utils 1:6.2+dfsg-2ubuntu6.3 amd64 QEMU utilities ```

KVM machine definition

```xml ubuntu22.04 68c72998-fdc2-4c18-bd07-18f773b1e0a4 1048576 1048576 2 hvm destroy restart destroy /usr/bin/qemu-system-x86_64

olivierlambert commented 2 years ago

Hi,

Nested virtualization is very complicated and not that widely used. Sadly, this combo makes bug hunt very hard, while having relatively little incentive to fix them due to nature of use cases.

I think your best chance would be to try asking on xen-devel mailing list about this, and see if there's a Xen developer willing to investigate further.

ddelnano commented 6 months ago

I recently tried to run KVM within Xen and received the same stack trace as above on XCP-ng 8.2.1.

I reached out to the Xen maintainers in Matrix to better understand the status of nested virt and if what I was trying should be supported (link). They mentioned that the earlier work to support nested virt was somewhat experimental, but they have plans to revamp the implementation.

Since I had interest in trying to debug this issue furhter, I wanted to see if the latest Xen release ran into the same issue. I compiled Xen 4.18 from source and tried the same KVM test. Rather than receiving the error above, I was able to run my VM workload some of the time. The times where it ran, it executed as expected while the times where it failed Xen crashed the KVM guest (different stack trace).

It's worth noting for my use case, I'm launching short lived processes (google test running BPF tests on different kernels). So the stability of my workload is less important since it's short lived.

Xen vm config file ``` type = "hvm" # Guest name name = "pixie hvm" # 128-bit UUID for the domain as a hexadecimal number. # Use "uuidgen" to generate one if required. # The default behavior is to generate a new UUID each time the guest is started. #uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" # Enable Microsoft Hyper-V compatibile paravirtualisation / # enlightenment interfaces. Turning this on can improve Windows guest # performance and is therefore recommended #viridian = 1 # Initial memory allocation (MB) memory = 4096 # Maximum memory (MB) # If this is greater than `memory' then the slack will start ballooned # (this assumes guest kernel support for ballooning) #maxmem = 512 # Number of VCPUS vcpus = 2 device_model_version = 'qemu-xen' acpi = 1 boot = 'cd' serial = 'pty' vga = "stdvga" videoram = 64 # Network devices # A list of 'vifspec' entries as described in # docs/misc/xl-network-configuration.markdown vif = [ '' ] # Disk Devices # A list of `diskspec' entries as described in # docs/misc/xl-disk-configuration.txt disk = ['phy:/dev/mapper/pixie--data-pixie--lv--data,xvda,rw', 'file:/home/ddelnano/vm/xen/ubuntu-22.04.3-live-server-amd64.iso,xvdb:cdrom,r'] # Guest VGA console configuration, either SDL or VNC #sdl = 1 vnc = 1 vnclisten = "127.0.0.1" vncdisplay = 0 vncpasswd = "secret" hap=1 nestedhvm=1 ```
xl dmesg output ``` (XEN) domain_crash called from arch/x86/hvm/vmx/vmx.c:1591 (XEN) Domain 8 (vcpu#1) crashed on cpu#36: (XEN) ----[ Xen-4.18.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 36 (XEN) RIP: 0008:[] (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (d8v1) (XEN) rax: 0000000000000020 rbx: 0000000002492000 rcx: 00000000c7fa2223 (XEN) rdx: 0000000006000000 rsi: 0000000000013b00 rdi: 0000000000000000 (XEN) rbp: 0000000000100000 rsp: ffffa45c8076bba0 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002060 (XEN) cr3: 00000001020f2005 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) ds: 0018 es: 0018 fs: 0018 gs: 0018 ss: 0018 cs: 0008 ```
michael-newsrx commented 4 months ago

This is a very long standing issue dating back at least 6 years now. See also:

Should all the nested virtualization fail tickets get merged?

stormi commented 4 months ago

I think it's not a problem if several tickets exist, as long as they reference each other. Individual situations may have different useful details.