Open syspro4 opened 3 years ago
Hi @syspro4. Thanks for your interest.
The current version of eRPC does not support the Raw transport anymore. I will mark it as broken in the CMakeLists.txt to avoid future confusion. If you wish to use the Raw transport, please use this version (https://github.com/erpc-io/eRPC/releases/tag/v0.1). You'll need to install an old version of Mellanox OFED (4.4 or older).
I'd recommend using eRPC's DPDK transport instead with DPDK 19.11 LTS. That will require installing Mellanox userspace drivers from rdma_core
instead of Mellanox OFED.
Thanks for the reply.
Regarding support for latest OFED drivers how difficult it is to fix the issue? Can it be possible to change the Experimental IBVERBs and use the standard IBVERBs? If the changes are straight forward then I can fix it with some help.
It's not difficult if we ignore the kDumb
flag in raw_transport.h
. That flag enables optimizations like multi-packet RX queue and "overrunning CQ", which aren't really necessary. As you said, most of the work here involves replacing experimental verbs with standard ibverbs.
I will be happy to test and merge a PR that migrates eRPC's raw transport to standard ibverbs, while ignoring the kDumb
flag.
Great! Thanks for the help! I will give it a try and let you know.
Hi, Please find attached patch which has a fix for raw transport issue. With this I am able to run hello_server & hello_client without any issue.
Following are the Mellanox versions I used: Server: MLNX_OFED_LINUX-4.7-3.2.9.0 Client: MLNX_OFED_LINUX-5.0-2.1.8.0
Note: In the patch I have kept kDumb code commented so that in the future we can fix it if needed. I can remove the kDumb code if it is not needed.
TODO: I am trying to understand the benchmark test and will run them later.
Questions:
Please let me know if any changes needed.
Thanks, Vaibhav
Thanks so much! I've tested that it seems to work in the latency benchmark with rdma_core. May I suggest the following:
verbs_common.h
(to count only active devices) are not needed. eRPC takes a phy_port
argument, which for verbs-based transports corresponds to the index of the device as listed by ibv_devinfo
, which includes inactive devices.If you are unable to, I will make the required changes soon and merge.
Answers to your questions:
fake_transport.h
and fake_transport.cc
) that can be used to implement a UDP sockets--based (ideally with AF_XDP) transport. I can test and merge if someone contributes this.server_rate
benchmark is best suited to get high RPCs/sec. apps/server_rate/config
allows configuring the number of server and client threads, and you can run multiple client processes to saturate the server.Thank you for testing the patch. I will sure address your comments and repost the patch.
I tried the server_rate app & I fixed some compilation issue and then ran ./scripts/do.sh from serve & client but I see following messages and I am not sure what to make out of the messages:
Note: I enabled LOG_LEVEL=info
On server side I only see following messages continuously. I do not see any progress "thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00"
On client side I see following message & nothing is happening after that: "Process 1, thread 15: Connected. Starting work."
Detailed logs: ====================== server side ==================================== server_dev :: ~/eRPC » cat scripts/autorun_process_file 192.168.2.222 31850 0 192.168.2.223 31850 0 server_dev :: ~/eRPC »
server_dev :: ~/eRPC » ./scripts/do.sh 0 0 Installing modded drivers do.sh: Launching process 0 on NUMA node 0 46:105830 INFOR: eRPC Nexus: Launching 0 background threads. 50:099920 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX] 50:100049 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 0]. Issue: None. Sending response. 50:100065 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 0]. 50:100441 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX] 50:100500 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 0]. Issue: None. Sending response. 50:100514 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 1]. 50:100858 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX] 50:100925 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 1]. Issue: None. Sending response. 50:100941 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 0]. 50:101425 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX] 50:101486 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 0]. Issue: None. Sending response. 50:101510 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 2]. 50:101917 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX] 50:101976 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 1]. Issue: None. Sending response. 50:101990 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 1]. 50:102902 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX] 50:102945 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX] 50:102971 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 1]. Issue: None. Sending response. 50:102988 INFOR: 50:102995 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 2]. Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 2]. Issue: None. Sending response. 50:103029 INFOR: Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 0]. 50:103376 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX] 50:103419 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX] 50:103433 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 2]. Issue: None. Sending response. 50:103463 INFOR: 50:103452 INFOR: Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 1]. eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX] 50:103521 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 2]. Issue: None. Sending response. 50:103537 INFOR: 50:103540 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 3]. Issue: None. Sending response. Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 2]. 50:103558 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 0]. 50:104667 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX] 50:104710 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX] 50:104726 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 3]. Issue: None. Sending response. 50:104754 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 1]. 50:104811 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 3]. Issue: None. Sending response. . . . 50:134880 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 13, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 14]. eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX] Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 15]. 50:134909 INFOR: 50:134927 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 15, S: 2]. Issue: None. Sending response. 50:134945 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX] Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 15]. 50:134980 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 15, S: 3]. Issue: None. Sending response. 50:134996 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 15]. thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00
====================== client side ==================================== client_dev :: ~/eRPC » cat scripts/autorun_process_file 192.168.2.222 31850 0 192.168.2.223 31850 0 client_dev :: ~/eRPC »
client_dev :: ~/eRPC » ./scripts/do.sh 1 0 Installing modded drivers do.sh: Launching process 1 on NUMA node 0 1 192.168.2.222 31850 02 192.168.2.223 31850 01 192.168.2.222 31850 02 192.168.2.223 31850 050:067736 INFOR: eRPC Nexus: Launching 0 background threads. 50:067751 INFOR: eRPC Nexus: Launching session management thread on core 23. 50:067887 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 192.168.2.223. 50:099929 INFOR: 50:100390 INFOR: 50:100396 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 0] Rpc 7: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]. 50:100426 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 1] 50:100754 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 0] Rpc 0: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX]. 50:100977 INFOR: RawTransport created for Rpc ID 15. Device mlx4_0/eth5, port 2. IPv4 192.168.2.223, MAC f4:52:14:74:d9:e1. Datapath UDP port 31897. 50:101279 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 2] 50:101417 INFOR: Rpc 1: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]. 50:101933 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 1] 50:102400 INFOR: 50:102402 INFOR: Rpc 0: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]. Rpc 7: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX]. 50:102455 INFOR: ERPC: Vaibhav num_devices 1 50:102752 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 2] 50:102798 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 0] eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 11] 50:144159 INFOR: 50:144167 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 11] for session 0. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 11] 50:144257 INFOR: 50:144259 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 11] for session 1. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 10] 50:144278 INFOR: 50:144280 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 10] for session 2. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 10] 50:144296 INFOR: 50:144297 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 10] for session 3. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 6, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 11] 50:144313 INFOR: 50:144319 INFOR: Rpc 6: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 11] for session 2. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 6, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 11] 50:155156 INFOR: 50:155162 INFOR: Rpc 6: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 11] for session 3. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 12] 50:155177 INFOR: 50:155186 INFOR: Rpc 12: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 12] for session 0. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 12] 50:166160 INFOR: 50:166165 INFOR: Rpc 12: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 12] for session 1. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 12]
50:210277 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 15] for session 0. Issue: None. Session connected. 50:210288 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 15] for session 1. Issue: None. Session connected. 50:210297 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 15] for session 2. Issue: None. Session connected. 50:210307 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 15] for session 3. Issue: None. Session connected. 50:210307 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 15] for session 3. Issue: None. Session connected. Process 1, thread 7: Connected. Starting work. Process 1, thread 0: Connected. Starting work. Process 1, thread 1: Connected. Starting work. thread_id: median_us 5th_us 99th_us 999th_us Mops Process 1, thread 5: Connected. Starting work. Process 1, thread 14: Connected. Starting work. Process 1, thread 4: Connected. Starting work. Process 1, thread 3: Connected. Starting work. Process 1, thread 8: Connected. Starting work. Process 1, thread 10: Connected. Starting work. Process 1, thread 2: Connected. Starting work. Process 1, thread 6: Connected. Starting work. Process 1, thread 11: Connected. Starting work. Process 1, thread 12: Connected. Starting work. Process 1, thread 9: Connected. Starting work. Process 1, thread 13: Connected. Starting work. Process 1, thread 15: Connected. Starting work.
I pushed the required compilation fixes to server_rate.cc
. It works on my cluster with the raw transport.
In your case, I suspect that the datapath packets are getting dropped somewhere. You could try re-compiling eRPC with detailed packet-level logging (ex: cmake . -DLOG_LEVEL=trace
). Then when you run the server_rate
client and server, they will create files in /tmp
with detailed per-packet logs.
You are correct. In the log I see packet drop error. I am getting it for hello_server/client test too.
Log: /tmp/erpc_trace_31850-rpc_0: 83:467073 TRACE: Rpc 0, lsn 0 (server_dev): TX [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Slot [num_tx 0, num_rx 0]. 83:467120 TRACE: eRPC RawTransport: Sending packet (idx = 0, drop = 0). SGE #1 72B, SGE #2 = 0B. pkthdr = [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Frame header = [ETH: dst f4:52:14:5e:60:e0, src f4:52:14:74:d9:e1, eth_type 2048], [IPv4: ihl 5, version 4, ecn 1, tot_len 58, id 0, frag_off 0, ttl 128, protocol 17, check 0, src IP 192.168.2.223, dst IP 192.168.2.222], [UDP: src_port 31882, dst_port 31882, len 38, check 0]. 83:472158 REORD: Rpc 0, lsn 0 (server_dev): Pkt loss suspected for req 8 ([num_tx 1, num_rx 0]). Action: Retransmitting requests. 83:472166 TRACE: Rpc 0, lsn 0 (server_dev): TX [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Slot [num_tx 0, num_rx 0]. 83:472175 TRACE: eRPC RawTransport: Sending packet (idx = 0, drop = 0). SGE #1 72B, SGE #2 = 0B. pkthdr = [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Frame header = [ETH: dst f4:52:14:5e:60:e0, src f4:52:14:74:d9:e1, eth_type 2048], [IPv4: ihl 5, version 4, ecn 1, tot_len 58, id 0, frag_off 0, ttl 128, protocol 17, check 0, src IP 192.168.2.223, dst IP 192.168.2.222], [UDP: src_port 31882, dst_port 31882, len 38, check 0].
I did rping test between server & client and it is working fine.
server_dev :: ~/eRPC 1 » rping -s -C 10 -v <<< server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrst server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstu server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuv server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvw server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwx server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxy server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyzA server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10 server_dev :: ~/eRPC »
client_dev :: ~/eRPC » /usr/bin/rping -c -a 192.168.2.222 -C 10 -v ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuv ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvw ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwx ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxy ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyzA client DISCONNECT EVENT... server_dev :: ~/eRPC »
Please help.
Thanks
Hi, I tried with different set of server/client but I see same "Pkt loss suspected for req 8" error in the trace log file. All these servers can do rping without any issue. Can you please help?
Thanks!
Couple of suggestions:
Could you try building with eRPC's DPDK transport and see if the packets go through? There are detailed instructions in the README to use DPDK with eRPC (in the section on Azure).
One issue could be that the switch or NIC drops packets because the Raw transport sets CRC = 0. The DPDK transport does the right thing and sets valid CRC.
The ultimate way to debug this is a a bit complicated: You'll need to construct the exact Ethernet frame that eRPC is sending in a simpler test application (the /tmp logs should help in doing this), and modify the packet fields until it goes through. That should shed some light on the packet drops.
Ah, I just remembered that you were able to run the hello_world test, so CRC is likely not an issue. Still, it might be worth trying with the DPDK transport. Other suggestions:
server_rate/config
to one thread per process?Thanks for the suggestions.
Actually earlier hello_world also did not work. I retested it and I see same pkt loss message.
When you tested my patch on your setup did you changed anything in your code base?
I tried erpc dpdk but that is giving following errors.
server_dev :: ~/eRPC » ./build/hello_server 6:616156 INFOR: eRPC Nexus: Launching 0 background threads. 6:616202 INFOR: eRPC Nexus: Launching session management thread on core 23. 6:617241 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 192.168.2.222. 6:617452 INFOR: DPDK transport for Rpc 0 initializing DPDK EAL. EAL: Detected 32 lcore(s) EAL: Detected 2 NUMA nodes EAL: Auto-detected process type: PRIMARY EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.2 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.3 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:81:00.0 on NUMA socket 1 EAL: probe driver: 15b3:1003 net_mlx4 PMD: net_mlx4: PCI information matches, using device "mlx4_0" (VF: false) PMD: net_mlx4: 2 port(s) detected PMD: net_mlx4: port 1 MAC address is f4:52:14:5e:60:e0 PMD: net_mlx4: port 2 MAC address is f4:52:14:5e:60:e1 6:947277 WARNG: Running as primary DPDK process. eRPC DPDK daemon is not running. 6:947316 INFOR: DPDK transport for Rpc 0 got QP 0 6:947354 INFOR: dev_info.rx_desc_lim.nb_max 0 kNumRxRingEntries 4096 <========================== Added this debug print terminate called after throwing an instance of 'std::runtime_error' what(): Device RX ring too small [2] 10374 abort ./build/hello_server
server_dev :: ~/eRPC »
I tried running dpdk-helloworld test program to check if dpdk is configured and working correctly and it is working fine.
server_dev :: ~/dpdk-stable-19.11.5/build » examples/dpdk-helloworld -l 0-3 -n 4 EAL: Detected 32 lcore(s) /** EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.2 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.3 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:81:00.0 on NUMA socket 1 EAL: probe driver: 15b3:1003 net_mlx4 hello from core 1 hello from core 2 hello from core 3 hello from core 0 server_dev :: ~/dpdk-stable-19.11.5/build »
It's strange that the port reports zero as the max number of RX descriptors. Could you try:
testpmd -- -i
info port 0
This should show the number of RX ring descriptors detected by testpmd.
Also: I see your system has two CX3 ports. Are both connected to the switch? If not, please ensure that eRPC is using the correct port.
Thanks for the reply. I see following info:
testpmd> show port info 0
Infos for port 0 MAC address: F4:52:14:5E:60:E0 Device name: mlx4_0 port 1 Driver name: net_mlx4 Connect to socket: 1 memory allocation on the socket: 1 Link status: up Link speed: 40000 Mbps Link duplex: full-duplex MTU: 1500 Promiscuous mode: enabled Allmulticast mode: disabled Maximum number of MAC addresses: 128 Maximum number of MAC addresses of hash filtering: 0 VLAN offload: strip off, filter off, extend off, qinq strip off Hash key size in bytes: 40 Supported RSS offload flow types: ipv4 ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv6 ipv6-frag ipv6-tcp ipv6-udp ipv6-other user defined 15 user defined 16 user defined 17 Minimum size of RX buffer: 32 Maximum configurable length of RX packet: 65536 Maximum configurable size of LRO aggregated packet: 0 Current number of RX queues: 1 Max possible RX queues: 1024 Max possible number of RXDs per queue: 65535 Min possible number of RXDs per queue: 0 RXDs number alignment: 1 Current number of TX queues: 1 Max possible TX queues: 1024 Max possible number of TXDs per queue: 65535 Min possible number of TXDs per queue: 0 TXDs number alignment: 1 Max segment number per packet: 65535 Max segment number per MTU/TSO: 65535 testpmd>
testpmd> show config rxtx io packet forwarding packets/burst=32 nb forwarding cores=1 - nb forwarding ports=2 port 0: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=0 - RX free threshold=0 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=0 - TX free threshold=0 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=0 port 1: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=0 - RX free threshold=0 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=0 - TX free threshold=0 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=0 testpmd>
I don't see the ring descriptors in "info port" command but in "show config rxtx" I see 0 for descriptor value.
Regarding the CX3 ports, yes I have a card with 2 ports and the program is picking up correct port. If we manually select the port then how can I do that?
Thanks!
Hi,
I am trying to compile latest eRPC and getting some compilation errors. Can you please help?
Thanks!
Following are the setup details:
[root@dev_mc eRPC]# cat /etc/release | grep release cat: /etc/lsb-release.d: Is a directory Oracle Linux Server release 8.4 Red Hat Enterprise Linux release 8.4 (Ootpa) Oracle Linux Server release 8.4 [root@dev_mc eRPC]#
[root@dev_mc eRPC]# ofed_info -s MLNX_OFED_LINUX-5.4-1.0.3.0: [root@dev_mc eRPC]# [root@dev_mc eRPC]# lspci | grep Mellanox 04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] 04:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] [root@dev_mc eRPC]#
[root@dev_mc eRPC]# cmake . -DPERF=ON -DTRANSPORT=raw; make -j . . .
[ 48%] Building CXX object CMakeFiles/erpc.dir/src/util/numautils.cc.o [ 49%] Building CXX object CMakeFiles/erpc.dir/src/transport_impl/fake/fake_transport.cc.o [ 50%] Building CXX object CMakeFiles/erpc.dir/src/util/tls_registry.cc.o [ 51%] Linking C shared library libhdr_histogram.so [ 51%] Built target hdr_histogram [ 52%] Linking C static library libhdr_histogram_static.a [ 52%] Built target hdr_histogram_static /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::tx_burst(const erpc::Transport::tx_burst_item_t, size_t)’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:12:40: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘msg_buffer’; did you mean ‘msgbuffer’? const MsgBuffer msg_buffer = item.msg_buffer; ^(&msgbuffer->buf[offset]);
^~~
buf
/root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:50:62: error: ‘const class erpc::MsgBuffer’ has no member named ‘data_size’; did you mean ‘datasize’?
sgl[1].length = (std::min)(kMaxDataPerPkt, msg_buffer->data_size - offset);
^ eth_hdr = reinterpret_cast<eth_hdrt*>(pkthdr->headroom);
^
~~~~~ msgbuffer /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:27:14: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pktidx’? if (item.pkt_idx == 0) { ^~~ pktidx /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:32:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer’? sgl[0].lkey = msgbuffer->buffer.lkey; ^~buffer /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:43:46: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pktidx’? pkthdr = msg_buffer->get_pkthdr_n(item.pkt_idx); ^~~ pktidx /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:46:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer’? sgl[0].lkey = msgbuffer->buffer.lkey; ^~buffer /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:48:28: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pktidx’? size_t offset = item.pkt_idx kMaxDataPerPkt; ^~~ pktidx /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:49:61: error: ‘const class erpc::MsgBuffer’ has no member named ‘buf’; did you mean ‘buf’? sgl[1].addr = reinterpret_cast~~~~ datasize /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:51:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer’? sgl[1].lkey = msgbuffer->buffer.lkey; ^~buffer /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:60:21: error: ‘struct erpc::pkthdrt’ has no member named ‘headroom’; did you mean ‘headroom’? memcpy(&pkthdr->headroom[0], item.routing_info, hdr_copysz); ^~~~ headroom /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:60:39: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘routing_info’; did you mean ‘routinginfo’? memcpy(&pkthdr->headroom[0], item.routing_info, hdr_copy_sz); ^~~~ routinginfo /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:62:26: error: ‘const struct erpc::Transport::tx_burst_itemt’ has no member named ‘drop’; did you mean ‘drop’? if (kTesting && item.drop) { ^~~~ drop_ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:64:60: error: ‘struct erpc::pkthdrt’ has no member named ‘headroom’; did you mean ‘headroom’? auto~~~ headroom /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:69:48: error: ‘struct erpc::pkthdrt’ has no member named ‘headroom’; did you mean ‘headroom’? reinterpret_cast<ipv4_hdr_t>(&pkthdr->headroom[sizeof(eth_hdrt)]); ^~~~ headroom /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::tx_flush()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:106:19: error: ‘huge_alloc’ was not declared in this scope Buffer buffer = huge_alloc->alloc(pkt_size); // Get a registered buffer ^~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:106:19: note: suggested alternative: ‘hugealloc’ Buffer buffer = huge_alloc->alloc(pkt_size); // Get a registered buffer ^~~~~~ hugealloc /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:109:17: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf’? memset(buffer.buf, 0, pktsize); ^~~ buf /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:110:53: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf’? auto pkthdr = reinterpret_cast<pkthdrt*>(buffer.buf); ^~~ buf /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:139:24: error: ‘class erpc::Buffer’ has no member named ‘lkey’; did you mean ‘lkey’? sgl[0].lkey = buffer.lkey; ^~~~ lkey_ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:159:3: error: ‘testing’ was not declared in this scope testing.tx_flush_count++; ^~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:159:3: note: suggested alternative: ‘testing’ testing.tx_flushcount++; ^~~ testing /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘size_t erpc::RawTransport::rx_burst()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:171:52: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf’? reinterpret_cast<pkthdr_t>(&ring_extent.buf[recv_head kRecvSize]); ^~~ buf_ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::post_recvs(size_t)’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:200:24: error: invalid use of incomplete type ‘struct erpc::ibv_exp_wq_family’ int ret = wq_family->recv_burst(wq, &mp_recv_sge[mp_sge_idx], 1); ^~ In file included from /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:3: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.h:202:10: note: forward declaration of ‘struct erpc::ibv_exp_wq_family’ struct ibv_exp_wq_family *wq_family; ^~~~~ At global scope: cc1plus: error: unrecognized command line option ‘-Wno-keyword-macro’ [-Werror] cc1plus: error: unrecognized command line option ‘-Wno-nested-anon-types’ [-Werror] cc1plus: all warnings being treated as errors make[2]: [CMakeFiles/erpc.dir/build.make:454: CMakeFiles/erpc.dir/src/transport_impl/raw/raw_transport_datapath.cc.o] Error 1 make[2]: Waiting for unfinished jobs.... /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In destructor ‘erpc::RawTransport::~RawTransport()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:58:40: error: aggregate ‘erpc::RawTransport::~RawTransport()::ibv_exp_release_intf_params rel_intf_params’ has incomplete type and cannot be defined struct ibv_exp_release_intf_params rel_intf_params; ^~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:61:9: error: ‘ibv_exp_release_intf’ was not declared in this scope ibv_exp_release_intf(resolve.ib_ctx, wq_family, &rel_intf_params) == 0, ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:61:9: note: suggested alternative: ‘ibv_exp_release_intf_params’ ibv_exp_release_intf(resolve.ib_ctx, wq_family, &rel_intf_params) == 0, ^~~~~~~~ ibv_exp_release_intf_params /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:64:17: error: ‘ibv_exp_destroy_flow’ was not declared in this scope exit_assert(ibv_exp_destroy_flow(recv_flow) == 0, ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:64:17: note: suggested alternative: ‘ibv_destroy_flow’ exit_assert(ibv_exp_destroy_flow(recv_flow) == 0, ^~~~~~~~ ibv_destroy_flow /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:70:17: error: ‘ibv_exp_destroy_rwq_ind_table’ was not declared in this scope exit_assert(ibv_exp_destroy_rwq_ind_table(ind_tbl) == 0, ^~~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:70:17: note: suggested alternative: ‘ibv_destroy_rwq_ind_table’ exit_assert(ibv_exp_destroy_rwq_ind_table(ind_tbl) == 0, ^~~~~~~~~ ibv_destroy_rwq_ind_table /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:73:17: error: ‘ibv_exp_destroy_wq’ was not declared in this scope exit_assert(ibv_exp_destroy_wq(wq) == 0, "Failed to destroy WQ"); ^~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:73:17: note: suggested alternative: ‘ibv_destroy_wq’ exit_assert(ibv_exp_destroy_wq(wq) == 0, "Failed to destroy WQ"); ^~~~~~ ibv_destroy_wq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:75:17: error: ‘ibv_exp_destroy_flow’ was not declared in this scope exit_assert(ibv_exp_destroy_flow(recv_flow) == 0, ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:75:17: note: suggested alternative: ‘ibv_destroy_flow’ exit_assert(ibv_exp_destroy_flow(recv_flow) == 0, ^~~~~~~~ ibv_destroy_flow /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In member function ‘void erpc::RawTransport::init_basic_qp()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:125:31: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_cq_init_attr cq_init_attr’ has incomplete type and cannot be defined struct ibv_exp_cq_init_attr cq_init_attr; ^~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:127:13: error: ‘ibv_exp_create_cq’ was not declared in this scope send_cq = ibv_exp_create_cq(resolve.ib_ctx, kSQDepth, nullptr, nullptr, 0, ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:127:13: note: suggested alternative: ‘ibv_create_cq’ send_cq = ibv_exp_create_cq(resolve.ib_ctx, kSQDepth, nullptr, nullptr, 0, ^~~~~ ibv_create_cq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:138:31: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_qp_init_attr qp_init_attr’ has incomplete type and cannot be defined struct ibv_exp_qp_init_attr qp_init_attr; ^~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:140:28: error: ‘IBV_EXP_QP_INIT_ATTR_PD’ was not declared in this scope qp_init_attr.comp_mask = IBV_EXP_QP_INIT_ATTR_PD; ^~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:140:28: note: suggested alternative: ‘IBV_QP_INIT_ATTR_PD’ qp_init_attr.comp_mask = IBV_EXP_QP_INIT_ATTR_PD; ^~~~~~~ IBV_QP_INIT_ATTR_PD /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:152:8: error: ‘ibv_exp_create_qp’ was not declared in this scope qp = ibv_exp_create_qp(resolve.ib_ctx, &qp_init_attr); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:152:8: note: suggested alternative: ‘ibv_create_qp’ qp = ibv_exp_create_qp(resolve.ib_ctx, &qp_init_attr); ^~~~~ ibv_create_qp /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:155:26: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_qp_attr qp_attr’ has incomplete type and cannot be defined struct ibv_exp_qp_attr qp_attr; ^~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:159:13: error: ‘ibv_exp_modify_qp’ was not declared in this scope rt_assert(ibv_exp_modify_qp(qp, &qp_attr, IBV_QP_STATE | IBV_QP_PORT) == 0); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:159:13: note: suggested alternative: ‘ibv_modify_qp’ rt_assert(ibv_exp_modify_qp(qp, &qp_attr, IBV_QP_STATE | IBV_QP_PORT) == 0); ^~~~~ ibv_modify_qp /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In member function ‘void erpc::RawTransport::init_mp_recv_qp()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:175:31: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_cq_init_attr cq_init_attr’ has incomplete type and cannot be defined struct ibv_exp_cq_init_attr cq_init_attr; ^~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:177:13: error: ‘ibv_exp_create_cq’ was not declared in this scope recv_cq = ibv_exp_create_cq(resolve.ib_ctx, kRecvCQDepth / 2, nullptr, ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:177:13: note: suggested alternative: ‘ibv_create_cq’ recv_cq = ibv_exp_create_cq(resolve.ib_ctx, kRecvCQDepth / 2, nullptr, ^~~~~ ibv_create_cq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:182:26: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_cq_attr cq_attr’ has incomplete type and cannot be defined struct ibv_exp_cq_attr cq_attr; ^~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:184:23: error: ‘IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS’ was not declared in this scope cq_attr.comp_mask = IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS; ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:184:23: note: suggested alternative: ‘IBV_QP_INIT_ATTR_CREATE_FLAGS’ cq_attr.comp_mask = IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS; ^~~~~~~~ IBV_QP_INIT_ATTR_CREATE_FLAGS /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:185:26: error: ‘IBV_EXP_CQ_IGNORE_OVERRUN’ was not declared in this scope cq_attr.cq_cap_flags = IBV_EXP_CQ_IGNORE_OVERRUN; ^~~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:185:26: note: suggested alternative: ‘IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN’ cq_attr.cq_cap_flags = IBV_EXP_CQ_IGNORE_OVERRUN; ^~~~~~~~~ IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:50: error: ‘IBV_EXP_CQ_CAP_FLAGS’ was not declared in this scope rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0); ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:50: note: suggested alternative: ‘IBV_QP_ACCESS_FLAGS’ rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0); ^~~~~~~~ IBV_QP_ACCESS_FLAGS /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:13: error: ‘ibv_exp_modify_cq’ was not declared in this scope rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:13: note: suggested alternative: ‘ibv_modify_cq’ rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0); ^~~~~ ibv_modify_cq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:188:31: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_wq_init_attr wq_init_attr’ has incomplete type and cannot be defined struct ibv_exp_wq_init_attr wq_init_attr; ^~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:191:26: error: ‘IBV_EXP_WQT_RQ’ was not declared in this scope wq_init_attr.wq_type = IBV_EXP_WQT_RQ; ^~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:191:26: note: suggested alternative: ‘IBV_WQT_RQ’ wq_init_attr.wq_type = IBV_EXP_WQT_RQ; ^~~~~~ IBV_WQT_RQ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:197:29: error: ‘IBV_EXP_CREATE_WQ_MP_RQ’ was not declared in this scope wq_init_attr.comp_mask |= IBV_EXP_CREATE_WQ_MP_RQ; ^~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:197:29: note: suggested alternative: ‘IBV_QP_RATE_LIMIT’ wq_init_attr.comp_mask |= IBV_EXP_CREATE_WQ_MP_RQ; ^~~~~~~ IBV_QP_RATE_LIMIT /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:198:34: error: ‘IBV_EXP_MP_RQ_NO_SHIFT’ was not declared in this scope wq_init_attr.mp_rq.use_shift = IBV_EXP_MP_RQ_NO_SHIFT; ^~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:198:34: note: suggested alternative: ‘IBV_WC_IP_CSUM_OK_SHIFT’ wq_init_attr.mp_rq.use_shift = IBV_EXP_MP_RQ_NO_SHIFT; ^~~~~~ IBV_WC_IP_CSUM_OK_SHIFT /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:201:8: error: ‘ibv_exp_create_wq’ was not declared in this scope wq = ibv_exp_create_wq(resolve.ib_ctx, &wq_init_attr); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:201:8: note: suggested alternative: ‘ibv_create_wq’ wq = ibv_exp_create_wq(resolve.ib_ctx, &wq_init_attr); ^~~~~ ibv_create_wq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:205:26: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_wq_attr wq_attr’ has incomplete type and cannot be defined struct ibv_exp_wq_attr wq_attr; ^~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:207:23: error: ‘IBV_EXP_WQ_ATTR_STATE’ was not declared in this scope wq_attr.attr_mask = IBV_EXP_WQ_ATTR_STATE; ^~~~~/root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:207:23: note: suggested alternative: ‘IBV_WQ_ATTR_STATE’ wq_attr.attr_mask = IBV_EXP_WQ_ATTR_STATE; ^~~~~IBV_WQ_ATTR_STATE /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:208:22: error: ‘IBV_EXP_WQS_RDY’ was not declared in this scope wq_attr.wq_state = IBV_EXP_WQS_RDY; ^~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:208:22: note: suggested alternative: ‘IBV_WQS_RDY’ wq_attr.wq_state = IBV_EXP_WQS_RDY; ^~~~~~~ IBV_WQS_RDY /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:209:13: error: ‘ibv_exp_modify_wq’ was not declared in this scope rt_assert(ibv_exp_modify_wq(wq, &wq_attr) == 0, "Failed to ready WQ"); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:209:13: note: suggested alternative: ‘ibv_modify_wq’ rt_assert(ibv_exp_modify_wq(wq, &wq_attr) == 0, "Failed to ready WQ"); ^~~~~ ibv_modify_wq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:8: error: use of enum ‘ibv_exp_query_intf_status’ without previous declaration enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK; ^~~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:48: error: ‘IBV_EXP_INTF_STAT_OK’ was not declared in this scope enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK; ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:48: note: suggested alternative: ‘IBV_QP_CUR_STATE’ enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK; ^~~~~~~~ IBV_QP_CUR_STATE ~