Open solemnify opened 3 years ago
I have interest in this issue, but can you explain more detail about this issue?
qemu/migration/rdma.c
and linux/arch/x86/kvm/krdma.c
, But I'm curious about which part of the code does really makes RDMA transaction.Thanks.
I think the original authors of GiantVM chose infiniband, which is used a lot in reality among RDMA networks, to implement the idea of GiantVM paper quickly.
The core of GiantVM's network implementation resides in linux/arch/x86/kvm (dsm.c, ivy.c, krdma.c). Start with the kvm_dsm_init() code that initializes the network function in dsm.c (eg. netwwork_ops.send = krdma_send; etc.). If you look at the kvm_dsm_threadfn() function, which is a thread launched by kvm_dsm_init(), there is kvm_dsm_handle_req(), which acts as the main server for message processing, and follow this function. GiantVM sends DSM level messages to handle page fault handling, and this client role is performed by ivy_kvm_dsm_page_fault(). The paired server role is played by ivy_kvm_dsm_handle_req(). There are two core functions for DSM implementation in ivy.c. (ivy_kvm_dsm_page_fault(), ivy_kvm_dsm_handle_req()).
If you follow the above sequence, you can specifically check the implementation of the low-level network that the high-level functions call.
At QEMU level, look at interrupt-router.c.
Source patched to be compilable with TCP option on real machine. If you are compiling with TCP, please refer to the following.
[Caution] GiantVM Linux-DSM-4.18 is best suited for RDMA, and the TCP code is unstable. If the real machine is ready, it is recommended to use RDMA as before.
[TCP version HOW TO] Related Links: https://github.com/ememos/GiantVM/commit/bf9a4d58e13b287084e3b46dd50f1f364107f141#diff-9b97f4915db62a6d015609eb70af82c0ef0370b4c82af74d2b9260df84f4a033
To compile the above patch, edit the following in arch/x86/include/asm/kvm_host.h and then compile.
(before) //#define USE_KTCP_NETWORK
(after)
//#define USE_KRDMA_NETWORK
Also, once the network of Linux-DSM-4.18 is determined, the QEMU code must be changed consistently (interrupt-router.c). (before) //#define ROUTER_CONNECTION_TCP
(after)
//#define ROUTER_CONNECTION_RDMA
I am also interested in solving this problem. Also, I think the performance can be improved by bypassing the TCP stack over BPF. it was discussed with @Taeung, but still it is not clear yet.
Only Infiniband support seems stable. For more easy of use and wide adoption of the GiantVM, ethernet support is a must.