How to use EPTP switching VMFUNC

[QinJingjie]

I have learned the code about EPTP switching, and I found the vcpu_vmfunc function is called in ksm_introspect_start(Introspect.c), and it is called in ksm_ioctl(Main_linux.c). But I don't know understand how to call ksm_ioctl, and how to use EPTP switching VMFUNC.

[asamy]

ksm_ioctl() is called by the kernel when a user application calls ioctl (see um/um.c). You use EPTP switching VMFUNC as follows:

vcpu_vmfunc(EPTP_NORMAL, 0);

Which will switch current EPTP (EPT_POINTER in VMCS) to the EPTP_NORMAL index from the EPTP list provided (eptp_list in struct ept (see ksm.h) aka EPTP_LIST_POINTER in VMCS, see ept_create_ptr and vcpu_run in vcpu.c.)

[QinJingjie]

Is um.c a sample of user application using ksm? And does VMFUNC only be used within guest? I tried to add vmfunc to kvm and used it through an ioctl but failed. I don't understand how ksm using vmfunc while being a hypervisor like kvm.

[asamy]

Yes, and yes. How did it "fail"? How are you setting it up? Are you trying to nest KVM in KSM or are you just trying to port VMFUNC to KVM? Also, VMFUNC is supported on Broadwell and above, not all processors will support it, it's a pretty new feature, i'd say.

[QinJingjie]

My processor can support EPTP-Switching. And I tried to port VMFUNC to KVM by using

__asm __volatile(ASM_VMX_VMFUNC "; setna %0"
             : "=q" (error) : "c" (eptp), "a" (func)
             : "cc")"

and called it in the guest through an ioctl, but it failed, seemed like the instruction can't execute in KVM. How can I nest KVM in KSM? And According to my understanding, KSM is deployed on the host, acted as a hypervisor which can replace KVM. Is that right? So I'm wondering how KSM execute VMFUNC outside the guest.

[asamy]

It needs to be enabled also in VMX control fields (secondary control field, EPTP_LIST_ADDRESS, EPTP_INDEX), etc... You don't just call it like that and expect it to work magically with a VM on...

I don't even know why you are using an ioctl to call it. VMFUNC can be called from CPL 3 (i.e. any user application).

KSM doesn't execute VMFUNC outside guest, it does it INSIDE guest, it cannot be executed in host, it's a guest only functionality. and no, KSM cannot nest KVM, neither can KVM and still provide the functionality KSM requires.

[asamy]

If you're new to Hardware assisted virtualization, I highly recommend you read this: Documentation/SPEC.rst

I wrote it specifically for this reason. Should you find anything confusing, feel free to comment here and I will explain.

[QinJingjie]

I find ksm_ioctl(Main_linux.c) has an ioctl "KSM_IOCTL_INTRO_START", and it will call ksm_introspect_start() --> vcpu_vmfunc() --> __vmx_vmfunc(), which execute VMFUNC instruction. I think this process is happened in the VMM(host), so I conjectured the VMFUNC is execute ouside guest(in the host). Does these functions run in the VMM(host)?

[asamy]

No... __vmx_vmfunc runs inside guest, it will only exit to host if either are true:

1. The function specified is not supported (i.e. not EPTP switching)
2. The EPTP index is too high (>511)

If VM functions are not supported though, KSM will enable emulation via a VMCALL (that also runs inside guest), which will exit to host and then it will switch to the specified EPTP (see vcpu_handle_vmcall and vcpu_emulate_vmfunc in exit.c).

ksm_ioctl is executed inside guest, not host... The kernel will only call it should a userspace application calls ioctl.

[QinJingjie]

Is KSM running in the host or in the guest? I thought it is running in the host before, so I thought mistakenly that the vmfunc is called in the host.

[asamy]

Please read this document: Documentation/SPEC.rst it will help you understand how KSM works and give you a general overview of how hardware-assisted virtualization works.

KSM acts as both guest and host, as host when it enters __vmx_entrypoint, as guest when it first starts up and when ksm_ioctl (and other callbacks like reboot/hotplug/sleep) are called.