search

redn-io / RedN

Arbitrary offloads for RDMA NICs
https://www.redn.io/
84 stars 20 forks source link

Reproduce RedN #1

Closed chaojin0310 closed 2 years ago

chaojin0310 commented 2 years ago

Hi! I‘m trying to reproduce RedN these days, but I've encountered a problem. When I run ./hash_bench -i bond0(bond0 is the name of a CX5 NIC) at the server side (whose IP address is 11.167.65.208) and run ./hash_bench 11.167.65.208 1 -i bond0 (a CX5 NIC, too) at the client side, I get failed to get remote memory address; no metadata available for region at the server side. My system version is CentOS 7.7.1908 and the kernel version is Linux 3.10.0-1062.el7.x86_64. I compile the MLNX_OFED-4.7-1.0.0.1 driver with the ./mlnxofedinstall command. Could you please tell me your system config of your experiments? I have attached the error information below. Thank you very much!

server:

$./hash_bench -i bond0
Mapping dram memory: size 268265456 bytes
Mapping dram memory: size 268265456 bytes
DEBUG[tid:51028][src/rdma/connection.c:993]: initializing RC module
[RDMA-Server] Listening on port 12345 for connections. interrupt (^C) to exit.
DEBUG[tid:51028][src/rdma/agent.c:123]: attempting to add connection to 11.167.65.208:12345
DEBUG[tid:51028][src/rdma/connection.c:56]: adding connection on socket #0
[RDMA-Client] Creating connection (status:pending) to 11.167.65.208:12345 on sockfd 0
---- Initializing hashmap ----
bucket addr 140713055117312
bucket[0] key=232 addr=140713055117323
bucket[1] key=233 addr=140713055379478
bucket[2] key=234 addr=140713055641633
bucket[3] key=235 addr=140713055903788
bucket[4] key=236 addr=140713056165943
bucket[5] key=237 addr=140713056428098
bucket[6] key=238 addr=140713056690253
bucket[7] key=239 addr=140713056952408
bucket[8] key=240 addr=140713057214563
bucket[9] key=241 addr=140713057476718
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[0]: RDMA_CM_EVENT_ADDR_RESOLVED
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[2]: RDMA_CM_EVENT_ROUTE_RESOLVED
DEBUG[tid:51031][src/rdma/connection.c:324]: initializing rdma device-0
creating background thread to poll completions (blocking)
DEBUG[tid:51031][src/rdma/connection.c:459]: Creating QP for sock #0 [SendQ - size: 1024] [RecvQ - size: 1024] flags 0
DEBUG[tid:51031][src/rdma/connection.c:535]: trigger pre-connection callback
DEBUG[tid:51031][src/rdma/mr.c:75]: [sockfd 0] registering 2 memory regions & 2 send/rcv buffers
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa4faa6000, len:268265456, rkey:105370, lkey:105370]
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #1 with addr:140712786845696 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa3face000, len:268265456, rkey:99757, lkey:99757]
Registering msg buffers with size: 304
DEBUG[tid:51031][src/rdma/mr.c:185]: registered msg_send_mr[addr:7ffa38040000, len:304]
DEBUG[tid:51031][src/rdma/mr.c:200]: registered msg_rcv_mr[addr:7ffa38041000, len:304]
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[4]: RDMA_CM_EVENT_CONNECT_REQUEST
DEBUG[tid:51031][src/rdma/connection.c:56]: adding connection on socket #1
DEBUG[tid:51031][src/rdma/connection.c:118]: private data 0x7ffa38040190 (len: given 56 expected 152)
DEBUG[tid:51031][src/rdma/mr.c:208]: [sockfd 1] updating remote mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:208]: [sockfd 1] updating remote mr #1 with addr:140712786845696 and size:268265456
creating background thread to poll completions (blocking)
DEBUG[tid:51031][src/rdma/connection.c:459]: Creating QP for sock #1 [SendQ - size: 1024] [RecvQ - size: 1024] flags 0
DEBUG[tid:51031][src/rdma/connection.c:564]: trigger pre-connection callback
DEBUG[tid:51031][src/rdma/mr.c:75]: [sockfd 1] registering 2 memory regions & 2 send/rcv buffers
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa4faa6000, len:268265456, rkey:105370, lkey:105370]
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #1 with addr:140712786845696 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa3face000, len:268265456, rkey:99757, lkey:99757]
Registering msg buffers with size: 304
DEBUG[tid:51031][src/rdma/mr.c:185]: registered msg_send_mr[addr:7ffa38074000, len:304]
DEBUG[tid:51031][src/rdma/mr.c:200]: registered msg_rcv_mr[addr:7ffa38075000, len:304]
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[9]: RDMA_CM_EVENT_ESTABLISHED
modify state for socket #0 from 0 to 2
DEBUG[tid:51031][src/rdma/connection.c:576]: trigger post-connection callback
Connection established [sockfd:0] [qpnum: 5027]
ADDING PEER SOCKET 0 (type: 2)
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[9]: RDMA_CM_EVENT_ESTABLISHED
modify state for socket #1 from 0 to 2
DEBUG[tid:51031][src/rdma/connection.c:576]: trigger post-connection callback
Connection established [sockfd:1] [qpnum: 5028]
ADDING PEER SOCKET 1 (type: 2)
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[4]: RDMA_CM_EVENT_CONNECT_REQUEST
DEBUG[tid:51031][src/rdma/connection.c:56]: adding connection on socket #2
DEBUG[tid:51031][src/rdma/connection.c:118]: private data 0x7ffa380735b0 (len: given 56 expected 152)
DEBUG[tid:51031][src/rdma/mr.c:208]: [sockfd 2] updating remote mr #0 with addr:140338352910336 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:208]: [sockfd 2] updating remote mr #1 with addr:140338084638720 and size:268265456
creating background thread to poll completions (blocking)
DEBUG[tid:51031][src/rdma/connection.c:459]: Creating QP for sock #2 [SendQ - size: 1024] [RecvQ - size: 1024] flags 8
DEBUG[tid:51031][src/rdma/connection.c:564]: trigger pre-connection callback
DEBUG[tid:51031][src/rdma/mr.c:75]: [sockfd 2] registering 2 memory regions & 2 send/rcv buffers
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa4faa6000, len:268265456, rkey:105370, lkey:105370]
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #1 with addr:140712786845696 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa3face000, len:268265456, rkey:99757, lkey:99757]
Registering msg buffers with size: 304
DEBUG[tid:51031][src/rdma/mr.c:185]: registered msg_send_mr[addr:7ffa380a8000, len:304]
DEBUG[tid:51031][src/rdma/mr.c:200]: registered msg_rcv_mr[addr:7ffa380a9000, len:304]
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[9]: RDMA_CM_EVENT_ESTABLISHED
modify state for socket #2 from 0 to 2
DEBUG[tid:51031][src/rdma/connection.c:576]: trigger post-connection callback
Connection established [sockfd:2] [qpnum: 5029]
ADDING PEER SOCKET 2 (type: 3)
DEBUG[tid:51031][src/rdma/agent.c:123]: attempting to add connection to 11.167.65.208:12345
DEBUG[tid:51031][src/rdma/connection.c:56]: adding connection on socket #3
[RDMA-Client] Creating connection (status:pending) to 11.167.65.208:12345 on sockfd 3
input id 0 to offload_hash
Setting sockfds [client: 2 worker: 3]
DEBUG[tid:51494][src/rdma/mr.c:319]: connection for sockfd 3 isn't currently active; sleeping for 1 sec...
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[0]: RDMA_CM_EVENT_ADDR_RESOLVED
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[2]: RDMA_CM_EVENT_ROUTE_RESOLVED
creating background thread to poll completions (blocking)
DEBUG[tid:51031][src/rdma/connection.c:459]: Creating QP for sock #3 [SendQ - size: 1024] [RecvQ - size: 1024] flags 8
DEBUG[tid:51031][src/rdma/connection.c:535]: trigger pre-connection callback
DEBUG[tid:51031][src/rdma/mr.c:75]: [sockfd 3] registering 2 memory regions & 2 send/rcv buffers
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa4faa6000, len:268265456, rkey:105370, lkey:105370]
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #1 with addr:140712786845696 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa3face000, len:268265456, rkey:99757, lkey:99757]
Registering msg buffers with size: 304
DEBUG[tid:51031][src/rdma/mr.c:185]: registered msg_send_mr[addr:7ffa380dc000, len:304]
DEBUG[tid:51031][src/rdma/mr.c:200]: registered msg_rcv_mr[addr:7ffa380dd000, len:304]
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[4]: RDMA_CM_EVENT_CONNECT_REQUEST
DEBUG[tid:51031][src/rdma/connection.c:56]: adding connection on socket #4
DEBUG[tid:51031][src/rdma/connection.c:118]: private data 0x7ffa380a7ee0 (len: given 56 expected 152)
DEBUG[tid:51031][src/rdma/mr.c:208]: [sockfd 4] updating remote mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:208]: [sockfd 4] updating remote mr #1 with addr:140712786845696 and size:268265456
creating background thread to poll completions (blocking)
DEBUG[tid:51031][src/rdma/connection.c:459]: Creating QP for sock #4 [SendQ - size: 1024] [RecvQ - size: 1024] flags 8
DEBUG[tid:51031][src/rdma/connection.c:564]: trigger pre-connection callback
DEBUG[tid:51031][src/rdma/mr.c:75]: [sockfd 4] registering 2 memory regions & 2 send/rcv buffers
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa4faa6000, len:268265456, rkey:105370, lkey:105370]
DEBUG[tid:51031][src/rdma/mr.c:82]: registering mr #1 with addr:140712786845696 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:129]: registered local_mr[addr:7ffa3face000, len:268265456, rkey:99757, lkey:99757]
Registering msg buffers with size: 304
DEBUG[tid:51031][src/rdma/mr.c:185]: registered msg_send_mr[addr:7ffa38110000, len:304]
DEBUG[tid:51031][src/rdma/mr.c:200]: registered msg_rcv_mr[addr:7ffa38111000, len:304]
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[9]: RDMA_CM_EVENT_ESTABLISHED
DEBUG[tid:51031][src/rdma/connection.c:214]: private data 0x7ffa380dbe10 (len: given 196 expected 56)
DEBUG[tid:51031][src/rdma/mr.c:208]: [sockfd 3] updating remote mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:51031][src/rdma/mr.c:208]: [sockfd 3] updating remote mr #1 with addr:140712786845696 and size:268265456
modify state for socket #3 from 0 to 2
DEBUG[tid:51031][src/rdma/connection.c:576]: trigger post-connection callback
Connection established [sockfd:3] [qpnum: 5030]
ADDING PEER SOCKET 3 (type: 4)
DEBUG[tid:51031][src/rdma/connection.c:521]: received event[9]: RDMA_CM_EVENT_ESTABLISHED
modify state for socket #4 from 0 to 2
DEBUG[tid:51031][src/rdma/connection.c:576]: trigger post-connection callback
Connection established [sockfd:4] [qpnum: 5031]
ADDING PEER SOCKET 4 (type: 4)
offload hash with id 0
performing hash offload [client: 2 worker: 3]
DEBUG[tid:51494][src/rdma/verbs.c:583]: POST --> WAIT (SEND WR #1) [send_fd:3 wait_fd:2 wait_idx:1]
updating scur_post 0 by 1 (original size 2)
DEBUG[tid:51494][src/rdma/verbs.c:766]: POST --> SEND_ENABLE(WR#2) [master = 3] [worker = 3] [idx = 7]
updating scur_post 1 by 1 (original size 2)
DEBUG[tid:51494][src/rdma/verbs.c:1463]: Find ctrl seg for wr_id: 2
found wr_id. [seg 1] idx #1 opcode 23 [UNDEFINED]
remote start: 140713055117312 end: 140713323382768
DEBUG[tid:51494][src/rdma/verbs.c:128]: POST --> RDMA_READ (opcode 4) (SEND WR 3) [local addr 7ffa4faa6000 remote addr 7ffa4faa6000]
updating scur_post 2 by 1 (original size 4)
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
DEBUG[tid:51494][src/rdma/mr.c:354]: mr metadata for sockfd 0 haven't yet been received; sleeping for 1 sec...
failed to get remote memory address; no metadata available for region
 [error code: 0]

client side:

$./hash_bench 11.167.65.208 2 -i bond0
Mapping dram memory: size 268265456 bytes
Mapping dram memory: size 268265456 bytes
DEBUG[tid:61765][src/rdma/connection.c:993]: initializing RC module
[RDMA-Server] Listening on port 12345 for connections. interrupt (^C) to exit.
DEBUG[tid:61765][src/rdma/agent.c:123]: attempting to add connection to 11.167.65.209:12345
DEBUG[tid:61765][src/rdma/connection.c:56]: adding connection on socket #0
[RDMA-Client] Creating connection (status:pending) to 11.167.65.209:12345 on sockfd 0
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[0]: RDMA_CM_EVENT_ADDR_RESOLVED
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[2]: RDMA_CM_EVENT_ROUTE_RESOLVED
DEBUG[tid:61768][src/rdma/connection.c:324]: initializing rdma device-0
creating background thread to poll completions (blocking)
DEBUG[tid:61768][src/rdma/connection.c:459]: Creating QP for sock #0 [SendQ - size: 1024] [RecvQ - size: 1024] flags 0
DEBUG[tid:61768][src/rdma/connection.c:535]: trigger pre-connection callback
DEBUG[tid:61768][src/rdma/mr.c:75]: [sockfd 0] registering 2 memory regions & 2 send/rcv buffers
DEBUG[tid:61768][src/rdma/mr.c:82]: registering mr #0 with addr:140338352910336 and size:268265456
DEBUG[tid:61768][src/rdma/mr.c:129]: registered local_mr[addr:7fa311ac7000, len:268265456, rkey:89217, lkey:89217]
DEBUG[tid:61768][src/rdma/mr.c:82]: registering mr #1 with addr:140338084638720 and size:268265456
DEBUG[tid:61768][src/rdma/mr.c:129]: registered local_mr[addr:7fa301aef000, len:268265456, rkey:76576, lkey:76576]
Registering msg buffers with size: 304
DEBUG[tid:61768][src/rdma/mr.c:185]: registered msg_send_mr[addr:7fa2fc040000, len:304]
DEBUG[tid:61768][src/rdma/mr.c:200]: registered msg_rcv_mr[addr:7fa2fc041000, len:304]
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[4]: RDMA_CM_EVENT_CONNECT_REQUEST
DEBUG[tid:61768][src/rdma/connection.c:56]: adding connection on socket #1
DEBUG[tid:61768][src/rdma/connection.c:118]: private data 0x7fa2fc040190 (len: given 56 expected 152)
DEBUG[tid:61768][src/rdma/mr.c:208]: [sockfd 1] updating remote mr #0 with addr:140338352910336 and size:268265456
DEBUG[tid:61768][src/rdma/mr.c:208]: [sockfd 1] updating remote mr #1 with addr:140338084638720 and size:268265456
creating background thread to poll completions (blocking)
DEBUG[tid:61768][src/rdma/connection.c:459]: Creating QP for sock #1 [SendQ - size: 1024] [RecvQ - size: 1024] flags 0
DEBUG[tid:61768][src/rdma/connection.c:564]: trigger pre-connection callback
DEBUG[tid:61768][src/rdma/mr.c:75]: [sockfd 1] registering 2 memory regions & 2 send/rcv buffers
DEBUG[tid:61768][src/rdma/mr.c:82]: registering mr #0 with addr:140338352910336 and size:268265456
DEBUG[tid:61768][src/rdma/mr.c:129]: registered local_mr[addr:7fa311ac7000, len:268265456, rkey:89217, lkey:89217]
DEBUG[tid:61768][src/rdma/mr.c:82]: registering mr #1 with addr:140338084638720 and size:268265456
DEBUG[tid:61768][src/rdma/mr.c:129]: registered local_mr[addr:7fa301aef000, len:268265456, rkey:76576, lkey:76576]
Registering msg buffers with size: 304
DEBUG[tid:61768][src/rdma/mr.c:185]: registered msg_send_mr[addr:7fa2fc074000, len:304]
DEBUG[tid:61768][src/rdma/mr.c:200]: registered msg_rcv_mr[addr:7fa2fc075000, len:304]
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[9]: RDMA_CM_EVENT_ESTABLISHED
modify state for socket #0 from 0 to 2
DEBUG[tid:61768][src/rdma/connection.c:576]: trigger post-connection callback
Connection established [sockfd:0] [qpnum: 3876]
DEBUG[tid:61765][src/rdma/agent.c:123]: attempting to add connection to 11.167.65.208:12345
ADDING PEER SOCKET 0 (type: 2)
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[9]: RDMA_CM_EVENT_ESTABLISHED
modify state for socket #1 from 0 to 2
DEBUG[tid:61768][src/rdma/connection.c:576]: trigger post-connection callback
Connection established [sockfd:1] [qpnum: 3877]
ADDING PEER SOCKET 1 (type: 2)
DEBUG[tid:61765][src/rdma/connection.c:56]: adding connection on socket #2
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[0]: RDMA_CM_EVENT_ADDR_RESOLVED
[RDMA-Client] Creating connection (status:pending) to 11.167.65.208:12345 on sockfd 2
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[2]: RDMA_CM_EVENT_ROUTE_RESOLVED
creating background thread to poll completions (blocking)
DEBUG[tid:61768][src/rdma/connection.c:459]: Creating QP for sock #2 [SendQ - size: 1024] [RecvQ - size: 1024] flags 8
DEBUG[tid:61768][src/rdma/connection.c:535]: trigger pre-connection callback
DEBUG[tid:61768][src/rdma/mr.c:75]: [sockfd 2] registering 2 memory regions & 2 send/rcv buffers
DEBUG[tid:61768][src/rdma/mr.c:82]: registering mr #0 with addr:140338352910336 and size:268265456
DEBUG[tid:61768][src/rdma/mr.c:129]: registered local_mr[addr:7fa311ac7000, len:268265456, rkey:89217, lkey:89217]
DEBUG[tid:61768][src/rdma/mr.c:82]: registering mr #1 with addr:140338084638720 and size:268265456
DEBUG[tid:61768][src/rdma/mr.c:129]: registered local_mr[addr:7fa301aef000, len:268265456, rkey:76576, lkey:76576]
Registering msg buffers with size: 304
DEBUG[tid:61768][src/rdma/mr.c:185]: registered msg_send_mr[addr:7fa2fc0a8000, len:304]
DEBUG[tid:61768][src/rdma/mr.c:200]: registered msg_rcv_mr[addr:7fa2fc0a9000, len:304]
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[9]: RDMA_CM_EVENT_ESTABLISHED
DEBUG[tid:61768][src/rdma/connection.c:214]: private data 0x7fa2fc073d80 (len: given 196 expected 56)
DEBUG[tid:61768][src/rdma/mr.c:208]: [sockfd 2] updating remote mr #0 with addr:140713055117312 and size:268265456
DEBUG[tid:61768][src/rdma/mr.c:208]: [sockfd 2] updating remote mr #1 with addr:140712786845696 and size:268265456
modify state for socket #2 from 0 to 2
DEBUG[tid:61768][src/rdma/connection.c:576]: trigger post-connection callback
Connection established [sockfd:2] [qpnum: 3878]
ADDING PEER SOCKET 2 (type: 3)
Starting benchmark ...
--> Send GET [key 1000 addr 140713055117312]
DEBUG[tid:61765][src/rdma/verbs.c:1006]: POST --> RDMA_SEND_IMM (SEND WR 1) [send_fd: 2 batch_size: 1]
updating scur_post 0 by 1 (original size 3)
DEBUG[tid:61765][src/rdma/verbs.c:714]: POST --> SEND_ENABLE(WR#1) [master = 0] [worker = 2]
updating scur_post 0 by 1 (original size 2)
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[10]: RDMA_CM_EVENT_DISCONNECTED
DEBUG[tid:61768][src/rdma/connection.c:586]: trigger disconnection callback
Connection terminated [sockfd:2]
DEBUG[tid:61768][src/rdma/connection.c:1277]: terminating connection on socket #2
modify state for socket #2 from 2 to -1
DEBUG[tid:61768][src/rdma/connection.c:521]: received event[15]: RDMA_CM_EVENT_TIMEWAIT_EXIT
DEBUG[tid:61768][src/rdma/connection.c:1318]: clearing connection metadata for socket #2
DEBUG[tid:61768][src/rdma/connection.c:1343]: deregistering msg_send_mr[addr:7fa2fc0a8000, len:304]
DEBUG[tid:61768][src/rdma/connection.c:1345]: deregistering msg_rcv_mr[addr:7fa2fc0a9000, len:304]
wreda commented 2 years ago

It seems the memory region keys aren't being passed correctly for socket 0. I have attempted a fix for this. Can you pull/rebuild RedN and try again?

chaojin0310 commented 2 years ago

RedN runs correctly now with ./hash_bench <peer_ip> 1, but stucks with ./hash_bench <peer_ip> 2 (i.e., iteration=2).

wreda commented 2 years ago

Thanks! The hash benchmark should now work with larger iteration counts. Let me know if you're still running into issues.

chaojin0310 commented 2 years ago

Thanks for your help! I want to measure throughput of different verbs. I modified some code in hash_bench.c, but I got several errors. Could you please upload a testing code example when evaluating verb throughput in Table 3 in RedN's paper? It would help a lot.

wreda commented 2 years ago

I'm planning on releasing throughput benchmarks for RedN soon (likely end of May or a bit after).

You can of course tweak the existing examples to benchmark throughput but, as you noted, it will require some engineering.