bysui
commented
8 months ago Thanks for your debugging and report!
Sorry, I don't have an AMD Zen 2 in my environment. However, I can easily reproduce the issue on an Intel platform.
In PVM, only PVM mode is allowed to run on the hardware, while non-PVM mode is emulated. When transitioning from non-PVM mode to PVM mode, the pvm->msr_star is set to the current CS selector in try_to_convert_to_pvm_mode(). And The CS pushed on the stack during kernel exception delivery is obtained from pvm->msr_star in do_pvm_supervisor_exception
According to the Linux boot protocol, when reaching startup_64(), the operating mode is already in 64-bit long mode, and the Rust VMM seems to set the CS selector as 1 * 8 for booting, which corresponds to __KERNEL32_CS in Linux. Therefore, the pvm->msr_star is set to the value of __KERNEL32_CS. Although the later booting code updates the CS selector to __KERNEL_CS, we don't update pvm->msr_star in the pvm_set_segment() callback. As a result, before the MSR_STAR setup, the pushed CS is always __KERNEL32_CS. Unfortunately, we haven't encountered any unsupported MSR during our testing before the real event entry setup.
The fix may look like this:
diff --git a/arch/x86/kvm/pvm/pvm.c b/arch/x86/kvm/pvm/pvm.c
index 35bdcf9b977d..0f1af1c0e528 100644
--- a/arch/x86/kvm/pvm/pvm.c
+++ b/arch/x86/kvm/pvm/pvm.c
@@ -1318,6 +1318,9 @@ static void pvm_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int
goto invalid_change;
if (cpl == 0 && !var->l)
pvm->non_pvm_mode = true;
+ if (cpl == 0 && !pvm->non_pvm_mode)
+ pvm->msr_star = ((u64)pvm->segments[VCPU_SREG_CS].selector << 32) |
+ ((u64)__USER32_CS << 48);
}
break;
case VCPU_SREG_LDTR:However, I will discuss the issue with Lai Jiangshan later to come up with a formal fix.
Champ-Goblem
zhuangel
Description
Boot demo VM on AMD Zen 2 virtual machine (which PCID is disabled) hangs.
Step to reproduce
Build PVM host kernel and PVM guest kernel Following the guide pvm-get-started-with-kata.md, install PVM host kernel in AMD Zen 2 virtual machine.
PVM VM resource from Guide cloud-hypervisor v37 VM image from Guide
Start PVM VM Start PVM VM on AMD Zen 2 virtual machine cloud-hypervisor.v37 `
--api-socket ch.sock \--log-file vmm.log \--cpus boot=1 \--kernel vmlinux.virt-pvm-guest \--cmdline 'console=ttyS0 root=/dev/vda1 rw clocksource=kvm-clock pti=off' \--memory size=1G,hugepages=off,shared=false,prefault=off \--disk id=disk_0,path=ubuntu-22.04-pvm-kata.raw \-v --console off --serial ttyThe PVM VM hangs There is no output on serial, and VMM process CPU usage is very low(there is no dead loop), then I enable all kvm tracepoint, found there is msr_read emulate failed(index 0xc0011020, MSR_AMD64_LS_CFG), and PVM inject GP to PVM VM. Then PVM VM hangs in early_fixup_exception, because of the CS is __KERNEL32_CS.
vcpu0-3676813 [000] d..1. 275659.054772: kvm_exit: vcpu 0 reason GP excp rip 0xffffd97f81051232 info1 0x000000000000000d info2 0x0000000000000000 intr_info 0x0000000d error_code 0x00000000vcpu0-3676813 [000] ..... 275659.054774: kvm_emulate_insn: 0:ffffd97f81051232:0f 32 (prot64)vcpu0-3676813 [000] ..... 275659.054774: kvm_msr: msr_read c0011020 = 0x0 (#GP)vcpu0-3676813 [000] ..... 275659.054775: kvm_inj_exception: #GP (0x0)Workaround
I am not sure why __KERNEL32_CS in early_fixup_exception, maybe there should be some detection xen_pv_domain for PVM kernel, so that PVM kernel could finished fixup process. And I could workaround the issue with following fix
--- a/arch/x86/mm/extable.c+++ b/arch/x86/mm/extable.c@@ -322,7 +322,7 @@ void __init early_fixup_exception(struct pt_regs *regs, int trapnr)* the 486 DX works this way.* Xen pv domains are not using the default __KERNEL_CS.*/- if (!xen_pv_domain() && regs->cs != __KERNEL_CS)+ if (!xen_pv_domain() && regs->cs != __KERNEL_CS && regs->cs != __KERNEL32_CS)goto fail;