bysui
commented
5 months ago Hi @pojntfx , Thank you for your CI/CD testing for PVM. We really appreciate it.
Actually, there is a major problem with live migration between different hosts currently. As can be seen from the error log of Cloud Hypervisor, the restoration of 'MSR_PVM_LINEAR_ADDRESS_RANGE' (MSR 0x4b564df0) has failed. The reason behind this is that the hypervisor needs to reserve a range in the vmalloc area. However, different hosts may have different range allocations, leading to the failure of MSR restoration. Since there is not enough space in the 4-level paging mode kernel address space, it becomes challenging to find an available fixed range. Hence, I had to employ a workaround to reserve the same range on the second host in my testing environment.
diff --git a/arch/x86/kvm/pvm/host_mmu.c b/arch/x86/kvm/pvm/host_mmu.c
index 35e97f4f7055..047e7679fe2d 100644
--- a/arch/x86/kvm/pvm/host_mmu.c
+++ b/arch/x86/kvm/pvm/host_mmu.c
@@ -35,8 +35,11 @@ static int __init guest_address_space_init(void)
return -1;
}
- pvm_va_range_l4 = get_vm_area_align(DEFAULT_RANGE_L4_SIZE, PT_L4_SIZE,
- VM_ALLOC|VM_NO_GUARD);
+ //pvm_va_range_l4 = get_vm_area_align(DEFAULT_RANGE_L4_SIZE, PT_L4_SIZE,
+ // VM_ALLOC|VM_NO_GUARD);
+ pvm_va_range_l4 = __get_vm_area_caller(DEFAULT_RANGE_L4_SIZE, VM_ALLOC|VM_NO_GUARD,
+ VMALLOC_END - DEFAULT_RANGE_L4_SIZE, VMALLOC_END,
+ __builtin_return_address(0));
if (!pvm_va_range_l4)
return -1;
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 6e4b95f24bd8..bf89f9184b62 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -2622,6 +2622,7 @@ struct vm_struct *__get_vm_area_caller(unsigned long size, unsigned long flags,
return __get_vm_area_node(size, 1, PAGE_SHIFT, flags, start, end,
NUMA_NO_NODE, GFP_KERNEL, caller);
}
+EXPORT_SYMBOL_GPL(__get_vm_area_caller);I've tried to use the provided host config file, but I cannot reproduce the issue. Without the workaround, the restoration fails, and a new VM boots instead. However, with the workaround in place, the restoration is successful on the second host.
From the host kernel crash log, it appears that the first problem is due to a NULL pointer access in pvm_vcpu_run(). Could you please provide the 'kvm-pvm.ko' file so that I can examine which variable is being accessed?
Additionally, based on the log, it seems that the issue may be related to XSAVE features. The faulting instruction in restore_fpregs_from_fpstate() is xrestores64, which causes the FPU restore to fail and triggers a #GP exception. Furthermore, in fixup_exception(), there is an attempt to restore the init_fpstate FPU context, which also calls restore_fpregs_from_fpstate(). This results in repeated faults and eventually leads to a task stack overflow. To gather more debugging information, could you please add the following code?
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 29413cb2f090..72b2a0964df8 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -5517,12 +5517,20 @@ static void kvm_vcpu_ioctl_x86_get_xsave2(struct kvm_vcpu *vcpu,
*/
u64 supported_xcr0 = vcpu->arch.guest_supported_xcr0 |
XFEATURE_MASK_FPSSE;
+ union fpregs_state *ustate = (void *)state;
if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
return;
fpu_copy_guest_fpstate_to_uabi(&vcpu->arch.guest_fpu, state, size,
supported_xcr0, vcpu->arch.pkru);
+
+ pr_info("during getting:\n guest xcr0: %llx, host xcr0: %llx, supported_xcr0: %llx\n",
+ vcpu->arch.xcr0, host_xcr0, supported_xcr0);
+ pr_info("guest pkru: %x, host pkru: %x\n",
+ vcpu->arch.pkru, vcpu->arch.host_pkru);
+ pr_info("xfeatures: %llx, xcomp_bv: %llx\n",
+ ustate->xsave.header.xfeatures, ustate->xsave.header.xcomp_bv);
}
static void kvm_vcpu_ioctl_x86_get_xsave(struct kvm_vcpu *vcpu,
@@ -5535,9 +5543,17 @@ static void kvm_vcpu_ioctl_x86_get_xsave(struct kvm_vcpu *vcpu,
static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
struct kvm_xsave *guest_xsave)
{
+ union fpregs_state *ustate = (void *)guest_xsave->region;
+
if (fpstate_is_confidential(&vcpu->arch.guest_fpu))
return 0;
+ pr_info("during settting:\n guest xcr0: %llx, host xcr0: %llx, supported_xcr0: %llx\n",
+ vcpu->arch.xcr0, host_xcr0, kvm_caps.supported_xcr0);
+ pr_info("guest pkru: %x, host pkru: %x\n",
+ vcpu->arch.pkru, vcpu->arch.host_pkru);
+ pr_info("xfeatures: %llx, xcomp_bv: %llx\n",
+ ustate->xsave.header.xfeatures, ustate->xsave.header.xcomp_bv);
return fpu_copy_uabi_to_guest_fpstate(&vcpu->arch.guest_fpu,
guest_xsave->region,
kvm_caps.supported_xcr0,
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 271dcb2deabc..21403b6e12a6 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -6,6 +6,7 @@
#include <xen/xen.h>
#include <asm/fpu/api.h>
+#include <asm/fpu/xcr.h>
#include <asm/sev.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
@@ -121,8 +122,18 @@ static bool ex_handler_sgx(const struct exception_table_entry *fixup,
static bool ex_handler_fprestore(const struct exception_table_entry *fixup,
struct pt_regs *regs)
{
+ static bool once;
regs->ip = ex_fixup_addr(fixup);
+ if (boot_cpu_has(X86_FEATURE_XSAVE) && !once) {
+ struct xregs_state *state = (void *)regs->di;
+
+ once = true;
+ pr_info("xcr0 is %llx\n", xgetbv(XCR_XFEATURE_ENABLED_MASK));
+ pr_info("xfeatures: %llx, xcomp_bv: %llx\n",
+ state->header.xfeatures, state->header.xcomp_bv);
+ }
+
WARN_ONCE(1, "Bad FPU state detected at %pB, reinitializing FPU registers.",
(void *)instruction_pointer(regs));
pojntfx
Description
We're testing PVM on AWS EC2 and we're running into some issues with restoring snapshots on EC2 (
c5.largein particular, but this also occurs on all other non-bare metal instance types - both Intel and AMD). Snapshot restores work well on GCP non-bare metal hosts with Cloud Hypervisor (and on bare metal hosts with the Intel KVM module unloaded and the PVM module loaded), but they lead to host kernel panics when used on AWS EC2. There are also some error messages (see further down) related to it not being able to set MSRs for the guest (a similar error occurs for Firecracker's snapshot restores, too!) in the load step before the kernel crashes.Note that this only happens when the snapshot is resumed from a different host than the one that it was created on, so when a snapshot is created on host A, moved from host A to host B and resumed on host B. Both instance types in this are the same (same CPU etc.). It does not occur if the snapshot is simply resumed from the same host that it was created on.
Reproduction
Snapshot/Restore
To reproduce (
35.89.175.21is the first EC2 instance,34.214.167.180is the second EC2 host, andrsyncis used to sync the snapshot, rootfs etc. between the VMs):Live Migration
The same issue also occurs with Cloud Hypervisor's live migration:
Full Kernel Panic
Expand section
```plaintext [ 346.852957] BUG: kernel NULL pointer dereference, address: 0000000000000000 [ 346.881359] #PF: supervisor read access in kernel mode [ 346.905711] #PF: error_code(0x0000) - not-present page [ 346.930199] PGD 0 P4D 0 [ 346.941589] Oops: 0000 [#1] PREEMPT SMP NOPTI [ 346.962726] CPU: 0 PID: 1083 Comm: vcpu0 Not tainted 6.7.0-rc6-pvm-host-fedora-baremetal #1 [ 347.003980] Hardware name: Amazon EC2 c5.large/, BIOS 1.0 10/16/2017 [ 347.033695] RIP: 0010:pvm_vcpu_run+0x415/0x560 [kvm_pvm] [ 347.058894] Code: f8 01 0f 85 ad fe ff ff 48 8b 93 d0 19 00 00 48 8b 83 a8 1a 00 00 80 e6 fd 48 83 bb 18 1b 00 00 00 48 89 93 d0 19 00 00 74 12 <48> 8b 00 25 00 02 00 00 48 09 d0 48 89 83 d0 19 00 00 48 b8 33 00 [ 347.151883] RSP: 0000:ffffa0cc01d43d98 EFLAGS: 00010006 [ 347.174132] RAX: 0000000000000000 RBX: ffff8bfccae00000 RCX: 000000000000000e [ 347.208690] RDX: 0000000000010012 RSI: fffffe3de5515f58 RDI: fffffe3de5515f58 [ 347.242816] RBP: ffffa0cc01d43db0 R08: 0000000000000000 R09: 0000000000000000 [ 347.276942] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 347.312976] R13: 0000000000000001 R14: 0000000000000000 R15: ffff8bfccae00038 [ 347.346938] FS: 0000000000000000(0000) GS:ffff8bfcf2600000(0000) knlGS:fffff0001f400000 [ 347.387738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 347.412850] CR2: 0000000000000000 CR3: 0000000104d04005 CR4: 00000000007706f0 [ 347.447733] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 347.481654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 347.516084] PKRU: 00000000 [ 347.528191] Call Trace: [ 347.539797]Cloud Hypervisor Error Log
Firecracker Error Log