search

redn-io / RedN

Arbitrary offloads for RDMA NICs
https://www.redn.io/
84 stars 20 forks source link

How can the rdma_metadata be made to accommodate more SGEs? #3

Open LiYuTingxxn opened 2 years ago

LiYuTingxxn commented 2 years ago

Hi,

The send and recv operations fail if there are four or more send meta->sge entries in the post get req async function at line 571 of hash bench.c and four or more recv meta->sge entries in the offload hash function.

LiYuTingxxn commented 2 years ago

When initializing send meta and recv meta, as described below, I set 4 ibv sge.

struct rdma_metadata send_meta = (struct rdma_metadata ) calloc(1, sizeof(struct rdma_metadata) + 4 * sizeof(struct ibv_sge));

struct rdma_metadata recv_meta = (struct rdma_metadata ) calloc(1, sizeof(struct rdma_metadata) + 4 * sizeof(struct ibv_sge));

ChengjunJia commented 2 years ago

@LiYuTingxxn @wreda I tried to run the RedN source code. I wanted to run for the scenario where the hash collision happens or multiple visits for the linked list so I can get the performance for a complex offloaded function. However,

  1. I changed parameter key from 1000 to 1004 for function post_get_req_sync, but it stuck for hash_bench or linkedlist_bench.
  2. I tried the REDN_PARALLEL/REDN_SEQUENTIAL for the hash_bench, but it still failed.
  3. I only succeeded for the basic hash_bench with REDN_SINGLE, but I got a bad performance with 50-ile latency as 42us and 99.9-ile as 39ms (The read latency is 3.5-7.3us for our testbed).

I am not sure whether my environment is not correct with OFED 4.9 (The author used 4.7), or if there are some bugs in the code because there are many '#if 0/1'. So I want to know whether you have run the linked_list successfully in your testbed. And I will appreciate it if you can give me some advice for the experiment reproduction.

Thanks.