search

ememos / GiantVM

9 stars 9 forks source link

TCP (Ethernet) support is not stable. #2

Open solemnify opened 3 years ago

solemnify commented 3 years ago

Only Infiniband support seems stable. For more easy of use and wide adoption of the GiantVM, ethernet support is a must.

ChoKyuWon commented 3 years ago

I have interest in this issue, but can you explain more detail about this issue?

  1. Why does GiantVM only support InfiniBand? Why it does not support RoCE or iWARP?
  2. Which code handle RDMA between two VM? I found qemu/migration/rdma.c and linux/arch/x86/kvm/krdma.c, But I'm curious about which part of the code does really makes RDMA transaction.

Thanks.

ememos commented 3 years ago

  1. I think the original authors of GiantVM chose infiniband, which is used a lot in reality among RDMA networks, to implement the idea of GiantVM paper quickly.

  2. The core of GiantVM's network implementation resides in linux/arch/x86/kvm (dsm.c, ivy.c, krdma.c). Start with the kvm_dsm_init() code that initializes the network function in dsm.c (eg. netwwork_ops.send = krdma_send; etc.). If you look at the kvm_dsm_threadfn() function, which is a thread launched by kvm_dsm_init(), there is kvm_dsm_handle_req(), which acts as the main server for message processing, and follow this function. GiantVM sends DSM level messages to handle page fault handling, and this client role is performed by ivy_kvm_dsm_page_fault(). The paired server role is played by ivy_kvm_dsm_handle_req(). There are two core functions for DSM implementation in ivy.c. (ivy_kvm_dsm_page_fault(), ivy_kvm_dsm_handle_req()).

    If you follow the above sequence, you can specifically check the implementation of the low-level network that the high-level functions call.

At QEMU level, look at interrupt-router.c.

ememos commented 3 years ago

Source patched to be compilable with TCP option on real machine. If you are compiling with TCP, please refer to the following.

[Caution] GiantVM Linux-DSM-4.18 is best suited for RDMA, and the TCP code is unstable. If the real machine is ready, it is recommended to use RDMA as before.

[TCP version HOW TO] Related Links: https://github.com/ememos/GiantVM/commit/bf9a4d58e13b287084e3b46dd50f1f364107f141#diff-9b97f4915db62a6d015609eb70af82c0ef0370b4c82af74d2b9260df84f4a033

To compile the above patch, edit the following in arch/x86/include/asm/kvm_host.h and then compile.

(before) //#define USE_KTCP_NETWORK

define USE_KRDMA_NETWORK

(after)

define USE_KTCP_NETWORK

//#define USE_KRDMA_NETWORK

Also, once the network of Linux-DSM-4.18 is determined, the QEMU code must be changed consistently (interrupt-router.c). (before) //#define ROUTER_CONNECTION_TCP

define ROUTER_CONNECTION_RDMA

(after)

define ROUTER_CONNECTION_TCP

//#define ROUTER_CONNECTION_RDMA

ParkHanbum commented 3 years ago

I am also interested in solving this problem. Also, I think the performance can be improved by bypassing the TCP stack over BPF. it was discussed with @Taeung, but still it is not clear yet.