ChengjunJia
commented
2 years ago I hope that the following messages would be helpful. 使用RedN源码，优先使用hash这个测试用例，其他测试用例可能存在问题。 Anyone who wants to run RedN can use the hash micro benchmark. The code for the linkedlist may have some bugs, while the test cases for memcached can not run (Makefiles are missed).
From the hash_bench.c, the last command is IBV_RECEIVE_SG(client, recv_meta, worker_wq_mr->lkey); The command is for the cross channel communication from the client qp (receive the message from another server) to the worker qp (a sequence of WQEs to lookup, i.e. the core design in RedN). The client qp wants to modify some WQEs in the worker qp, so the key is for the visit of wq_buffer. worker_wq_mr->lkey is from register_wq(worker, worker), which calls the ibv_reg_mr to register the ibv_ex_get_wq_buffer of the worker qp. Finally, the client qp uses the key with the ability to modify the worker qp's WQEs (i.e. a memory region).
The main problem for the linkedlist bench is the Memory Region Protection. It uses the key mr_local_key(worker, mr_get_sq_idx(worker)) for the client qp to modify the worker qp's WQEs, but the key is for the DATA region! Thus, the first WQE in client qp, whose sockfd is 2, fails due to the local protection error.
In theory, the disable_wqe_checks.sh should work for the closure of the protection check, but it fails for my server (From the output, there is no change for the NIC register configuration). I am not sure whether my NIC firmware is wrong or the OFED version has problems (I have to use the latest 4.9 to adapt to my OS). If you have similar problems, I recommend the hash micro-benchmark instead of others.
authwork
Hello! I'm trying to run linkedlist_bench these days. But there were some errors when I run it. Could you please tell me how to solve them? I have attached the error information below. I set the OFFLOAD_COUNT=1 and LIST_SIZE-=1. Thank you very much！
SERVER: DEBUG[tid:15641][src/rdma/verbs.c:714]: POST --> SEND_ENABLE(WR#1) [master = 0] [worker = 3] updating scur_post 0 by 1 (original size 2) DEBUG[tid:15641][src/rdma/verbs.c:1112]: POST --> RECV (WR #99) [send_fd: 2, qp_num 5323] DEBUG[tid:15641][src/rdma/verbs.c:1115]: ----------- sge0 [addr 7f46d8339129, length 3] mlx5: fs240: got completion with error: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000 00000000 00000000 9d003304 000014cb 00006ae2 COMPLETION FAILURE on sockfd 2 (SEND WR #99) status[4] = local protection error
CLINET: Starting benchmark ... --> Send GET [key 1000] DEBUG[tid:72599][src/rdma/verbs.c:1006]: POST --> RDMA_SEND_IMM (SEND WR 1) [send_fd: 2 batch_size: 1] updating scur_post 0 by 1 (original size 2) DEBUG[tid:72599][src/rdma/verbs.c:714]: POST --> SEND_ENABLE(WR#1) [master = 0] [worker = 2] updating scur_post 0 by 1 (original size 2) mlx5: fs237: got completion with error: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000 00000000 00000000 00008914 0b000ded 000059d2 COMPLETION FAILURE on sockfd 2 (SEND WR #1) status[11] = remote operation error DEBUG[tid:72602][src/rdma/connection.c:521]: received event[10]: RDMA_CM_EVENT_DISCONNECTED DEBUG[tid:72602][src/rdma/connection.c:586]: trigger disconnection callback Connection terminated [sockfd:2] DEBUG[tid:72602][src/rdma/connection.c:1277]: terminating connection on socket #2 modify state for socket #2 from 2 to -1 DEBUG[tid:72602][src/rdma/connection.c:521]: received event[15]: RDMA_CM_EVENT_TIMEWAIT_EXIT DEBUG[tid:72602][src/rdma/connection.c:1318]: clearing connection metadata for socket #2 DEBUG[tid:72602][src/rdma/connection.c:1343]: deregistering msg_send_mr[addr:7f39840a9000, len:304] DEBUG[tid:72602][src/rdma/connection.c:1345]: deregistering msg_rcv_mr[addr:7f39840aa000, len:304]