erpc-io / eRPC

Efficient RPCs for datacenter networks
Other
861 stars 140 forks source link

Failed to compile 'raw' transport #66

Open syspro4 opened 3 years ago

syspro4 commented 3 years ago

Hi,

I am trying to compile latest eRPC and getting some compilation errors. Can you please help?

Thanks!

Following are the setup details:

[root@dev_mc eRPC]# cat /etc/release | grep release cat: /etc/lsb-release.d: Is a directory Oracle Linux Server release 8.4 Red Hat Enterprise Linux release 8.4 (Ootpa) Oracle Linux Server release 8.4 [root@dev_mc eRPC]#

[root@dev_mc eRPC]# ofed_info -s MLNX_OFED_LINUX-5.4-1.0.3.0: [root@dev_mc eRPC]# [root@dev_mc eRPC]# lspci | grep Mellanox 04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] 04:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] [root@dev_mc eRPC]#

[root@dev_mc eRPC]# cmake . -DPERF=ON -DTRANSPORT=raw; make -j . . .

[ 48%] Building CXX object CMakeFiles/erpc.dir/src/util/numautils.cc.o [ 49%] Building CXX object CMakeFiles/erpc.dir/src/transport_impl/fake/fake_transport.cc.o [ 50%] Building CXX object CMakeFiles/erpc.dir/src/util/tls_registry.cc.o [ 51%] Linking C shared library libhdr_histogram.so [ 51%] Built target hdr_histogram [ 52%] Linking C static library libhdr_histogram_static.a [ 52%] Built target hdr_histogram_static /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::tx_burst(const erpc::Transport::tx_burst_item_t, size_t)’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:12:40: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘msg_buffer’; did you mean ‘msgbuffer’? const MsgBuffer msg_buffer = item.msg_buffer; ^~~~~~ msgbuffer /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:27:14: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pktidx’? if (item.pkt_idx == 0) { ^~~ pktidx /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:32:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer’? sgl[0].lkey = msgbuffer->buffer.lkey; ^~ buffer /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:43:46: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pktidx’? pkthdr = msg_buffer->get_pkthdr_n(item.pkt_idx); ^~~ pktidx /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:46:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer’? sgl[0].lkey = msgbuffer->buffer.lkey; ^~ buffer /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:48:28: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘pkt_idx’; did you mean ‘pktidx’? size_t offset = item.pkt_idx kMaxDataPerPkt; ^~~ pktidx /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:49:61: error: ‘const class erpc::MsgBuffer’ has no member named ‘buf’; did you mean ‘buf’? sgl[1].addr = reinterpret_cast(&msgbuffer->buf[offset]); ^~~ buf /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:50:62: error: ‘const class erpc::MsgBuffer’ has no member named ‘data_size’; did you mean ‘datasize’? sgl[1].length = (std::min)(kMaxDataPerPkt, msg_buffer->data_size - offset); ^~~~~ datasize /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:51:33: error: ‘const class erpc::MsgBuffer’ has no member named ‘buffer’; did you mean ‘buffer’? sgl[1].lkey = msgbuffer->buffer.lkey; ^~ buffer /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:60:21: error: ‘struct erpc::pkthdrt’ has no member named ‘headroom’; did you mean ‘headroom’? memcpy(&pkthdr->headroom[0], item.routing_info, hdr_copysz); ^~~~ headroom /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:60:39: error: ‘const struct erpc::Transport::tx_burst_item_t’ has no member named ‘routing_info’; did you mean ‘routinginfo’? memcpy(&pkthdr->headroom[0], item.routing_info, hdr_copy_sz); ^~~~ routinginfo /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:62:26: error: ‘const struct erpc::Transport::tx_burst_itemt’ has no member named ‘drop’; did you mean ‘drop’? if (kTesting && item.drop) { ^~~~ drop_ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:64:60: error: ‘struct erpc::pkthdrt’ has no member named ‘headroom’; did you mean ‘headroom’? auto eth_hdr = reinterpret_cast<eth_hdrt*>(pkthdr->headroom); ^~~~ headroom /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:69:48: error: ‘struct erpc::pkthdrt’ has no member named ‘headroom’; did you mean ‘headroom’? reinterpret_cast<ipv4_hdr_t>(&pkthdr->headroom[sizeof(eth_hdrt)]); ^~~~ headroom /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::tx_flush()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:106:19: error: ‘huge_alloc’ was not declared in this scope Buffer buffer = huge_alloc->alloc(pkt_size); // Get a registered buffer ^~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:106:19: note: suggested alternative: ‘hugealloc’ Buffer buffer = huge_alloc->alloc(pkt_size); // Get a registered buffer ^~~~~~ hugealloc /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:109:17: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf’? memset(buffer.buf, 0, pktsize); ^~~ buf /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:110:53: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf’? auto pkthdr = reinterpret_cast<pkthdrt*>(buffer.buf); ^~~ buf /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:139:24: error: ‘class erpc::Buffer’ has no member named ‘lkey’; did you mean ‘lkey’? sgl[0].lkey = buffer.lkey; ^~~~ lkey_ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:159:3: error: ‘testing’ was not declared in this scope testing.tx_flush_count++; ^~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:159:3: note: suggested alternative: ‘testing’ testing.tx_flushcount++; ^~~ testing /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘size_t erpc::RawTransport::rx_burst()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transportdatapath.cc:171:52: error: ‘class erpc::Buffer’ has no member named ‘buf’; did you mean ‘buf’? reinterpret_cast<pkthdr_t>(&ring_extent.buf[recv_head kRecvSize]); ^~~ buf_ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc: In member function ‘void erpc::RawTransport::post_recvs(size_t)’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:200:24: error: invalid use of incomplete type ‘struct erpc::ibv_exp_wq_family’ int ret = wq_family->recv_burst(wq, &mp_recv_sge[mp_sge_idx], 1); ^~ In file included from /root/dev/test/eRPC/src/transport_impl/raw/raw_transport_datapath.cc:3: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.h:202:10: note: forward declaration of ‘struct erpc::ibv_exp_wq_family’ struct ibv_exp_wq_family *wq_family; ^~~~~ At global scope: cc1plus: error: unrecognized command line option ‘-Wno-keyword-macro’ [-Werror] cc1plus: error: unrecognized command line option ‘-Wno-nested-anon-types’ [-Werror] cc1plus: all warnings being treated as errors make[2]: [CMakeFiles/erpc.dir/build.make:454: CMakeFiles/erpc.dir/src/transport_impl/raw/raw_transport_datapath.cc.o] Error 1 make[2]: Waiting for unfinished jobs.... /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In destructor ‘erpc::RawTransport::~RawTransport()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:58:40: error: aggregate ‘erpc::RawTransport::~RawTransport()::ibv_exp_release_intf_params rel_intf_params’ has incomplete type and cannot be defined struct ibv_exp_release_intf_params rel_intf_params; ^~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:61:9: error: ‘ibv_exp_release_intf’ was not declared in this scope ibv_exp_release_intf(resolve.ib_ctx, wq_family, &rel_intf_params) == 0, ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:61:9: note: suggested alternative: ‘ibv_exp_release_intf_params’ ibv_exp_release_intf(resolve.ib_ctx, wq_family, &rel_intf_params) == 0, ^~~~~~~~ ibv_exp_release_intf_params /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:64:17: error: ‘ibv_exp_destroy_flow’ was not declared in this scope exit_assert(ibv_exp_destroy_flow(recv_flow) == 0, ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:64:17: note: suggested alternative: ‘ibv_destroy_flow’ exit_assert(ibv_exp_destroy_flow(recv_flow) == 0, ^~~~~~~~ ibv_destroy_flow /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:70:17: error: ‘ibv_exp_destroy_rwq_ind_table’ was not declared in this scope exit_assert(ibv_exp_destroy_rwq_ind_table(ind_tbl) == 0, ^~~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:70:17: note: suggested alternative: ‘ibv_destroy_rwq_ind_table’ exit_assert(ibv_exp_destroy_rwq_ind_table(ind_tbl) == 0, ^~~~~~~~~ ibv_destroy_rwq_ind_table /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:73:17: error: ‘ibv_exp_destroy_wq’ was not declared in this scope exit_assert(ibv_exp_destroy_wq(wq) == 0, "Failed to destroy WQ"); ^~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:73:17: note: suggested alternative: ‘ibv_destroy_wq’ exit_assert(ibv_exp_destroy_wq(wq) == 0, "Failed to destroy WQ"); ^~~~~~ ibv_destroy_wq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:75:17: error: ‘ibv_exp_destroy_flow’ was not declared in this scope exit_assert(ibv_exp_destroy_flow(recv_flow) == 0, ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:75:17: note: suggested alternative: ‘ibv_destroy_flow’ exit_assert(ibv_exp_destroy_flow(recv_flow) == 0, ^~~~~~~~ ibv_destroy_flow /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In member function ‘void erpc::RawTransport::init_basic_qp()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:125:31: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_cq_init_attr cq_init_attr’ has incomplete type and cannot be defined struct ibv_exp_cq_init_attr cq_init_attr; ^~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:127:13: error: ‘ibv_exp_create_cq’ was not declared in this scope send_cq = ibv_exp_create_cq(resolve.ib_ctx, kSQDepth, nullptr, nullptr, 0, ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:127:13: note: suggested alternative: ‘ibv_create_cq’ send_cq = ibv_exp_create_cq(resolve.ib_ctx, kSQDepth, nullptr, nullptr, 0, ^~~~~ ibv_create_cq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:138:31: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_qp_init_attr qp_init_attr’ has incomplete type and cannot be defined struct ibv_exp_qp_init_attr qp_init_attr; ^~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:140:28: error: ‘IBV_EXP_QP_INIT_ATTR_PD’ was not declared in this scope qp_init_attr.comp_mask = IBV_EXP_QP_INIT_ATTR_PD; ^~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:140:28: note: suggested alternative: ‘IBV_QP_INIT_ATTR_PD’ qp_init_attr.comp_mask = IBV_EXP_QP_INIT_ATTR_PD; ^~~~~~~ IBV_QP_INIT_ATTR_PD /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:152:8: error: ‘ibv_exp_create_qp’ was not declared in this scope qp = ibv_exp_create_qp(resolve.ib_ctx, &qp_init_attr); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:152:8: note: suggested alternative: ‘ibv_create_qp’ qp = ibv_exp_create_qp(resolve.ib_ctx, &qp_init_attr); ^~~~~ ibv_create_qp /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:155:26: error: aggregate ‘erpc::RawTransport::init_basic_qp()::ibv_exp_qp_attr qp_attr’ has incomplete type and cannot be defined struct ibv_exp_qp_attr qp_attr; ^~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:159:13: error: ‘ibv_exp_modify_qp’ was not declared in this scope rt_assert(ibv_exp_modify_qp(qp, &qp_attr, IBV_QP_STATE | IBV_QP_PORT) == 0); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:159:13: note: suggested alternative: ‘ibv_modify_qp’ rt_assert(ibv_exp_modify_qp(qp, &qp_attr, IBV_QP_STATE | IBV_QP_PORT) == 0); ^~~~~ ibv_modify_qp /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc: In member function ‘void erpc::RawTransport::init_mp_recv_qp()’: /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:175:31: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_cq_init_attr cq_init_attr’ has incomplete type and cannot be defined struct ibv_exp_cq_init_attr cq_init_attr; ^~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:177:13: error: ‘ibv_exp_create_cq’ was not declared in this scope recv_cq = ibv_exp_create_cq(resolve.ib_ctx, kRecvCQDepth / 2, nullptr, ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:177:13: note: suggested alternative: ‘ibv_create_cq’ recv_cq = ibv_exp_create_cq(resolve.ib_ctx, kRecvCQDepth / 2, nullptr, ^~~~~ ibv_create_cq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:182:26: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_cq_attr cq_attr’ has incomplete type and cannot be defined struct ibv_exp_cq_attr cq_attr; ^~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:184:23: error: ‘IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS’ was not declared in this scope cq_attr.comp_mask = IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS; ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:184:23: note: suggested alternative: ‘IBV_QP_INIT_ATTR_CREATE_FLAGS’ cq_attr.comp_mask = IBV_EXP_CQ_ATTR_CQ_CAP_FLAGS; ^~~~~~~~ IBV_QP_INIT_ATTR_CREATE_FLAGS /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:185:26: error: ‘IBV_EXP_CQ_IGNORE_OVERRUN’ was not declared in this scope cq_attr.cq_cap_flags = IBV_EXP_CQ_IGNORE_OVERRUN; ^~~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:185:26: note: suggested alternative: ‘IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN’ cq_attr.cq_cap_flags = IBV_EXP_CQ_IGNORE_OVERRUN; ^~~~~~~~~ IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:50: error: ‘IBV_EXP_CQ_CAP_FLAGS’ was not declared in this scope rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0); ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:50: note: suggested alternative: ‘IBV_QP_ACCESS_FLAGS’ rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0); ^~~~~~~~ IBV_QP_ACCESS_FLAGS /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:13: error: ‘ibv_exp_modify_cq’ was not declared in this scope rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:186:13: note: suggested alternative: ‘ibv_modify_cq’ rt_assert(ibv_exp_modify_cq(recv_cq, &cq_attr, IBV_EXP_CQ_CAP_FLAGS) == 0); ^~~~~ ibv_modify_cq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:188:31: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_wq_init_attr wq_init_attr’ has incomplete type and cannot be defined struct ibv_exp_wq_init_attr wq_init_attr; ^~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:191:26: error: ‘IBV_EXP_WQT_RQ’ was not declared in this scope wq_init_attr.wq_type = IBV_EXP_WQT_RQ; ^~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:191:26: note: suggested alternative: ‘IBV_WQT_RQ’ wq_init_attr.wq_type = IBV_EXP_WQT_RQ; ^~~~~~ IBV_WQT_RQ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:197:29: error: ‘IBV_EXP_CREATE_WQ_MP_RQ’ was not declared in this scope wq_init_attr.comp_mask |= IBV_EXP_CREATE_WQ_MP_RQ; ^~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:197:29: note: suggested alternative: ‘IBV_QP_RATE_LIMIT’ wq_init_attr.comp_mask |= IBV_EXP_CREATE_WQ_MP_RQ; ^~~~~~~ IBV_QP_RATE_LIMIT /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:198:34: error: ‘IBV_EXP_MP_RQ_NO_SHIFT’ was not declared in this scope wq_init_attr.mp_rq.use_shift = IBV_EXP_MP_RQ_NO_SHIFT; ^~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:198:34: note: suggested alternative: ‘IBV_WC_IP_CSUM_OK_SHIFT’ wq_init_attr.mp_rq.use_shift = IBV_EXP_MP_RQ_NO_SHIFT; ^~~~~~ IBV_WC_IP_CSUM_OK_SHIFT /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:201:8: error: ‘ibv_exp_create_wq’ was not declared in this scope wq = ibv_exp_create_wq(resolve.ib_ctx, &wq_init_attr); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:201:8: note: suggested alternative: ‘ibv_create_wq’ wq = ibv_exp_create_wq(resolve.ib_ctx, &wq_init_attr); ^~~~~ ibv_create_wq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:205:26: error: aggregate ‘erpc::RawTransport::init_mp_recv_qp()::ibv_exp_wq_attr wq_attr’ has incomplete type and cannot be defined struct ibv_exp_wq_attr wq_attr; ^~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:207:23: error: ‘IBV_EXP_WQ_ATTR_STATE’ was not declared in this scope wq_attr.attr_mask = IBV_EXP_WQ_ATTR_STATE; ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:207:23: note: suggested alternative: ‘IBV_WQ_ATTR_STATE’ wq_attr.attr_mask = IBV_EXP_WQ_ATTR_STATE; ^~~~~ IBV_WQ_ATTR_STATE /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:208:22: error: ‘IBV_EXP_WQS_RDY’ was not declared in this scope wq_attr.wq_state = IBV_EXP_WQS_RDY; ^~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:208:22: note: suggested alternative: ‘IBV_WQS_RDY’ wq_attr.wq_state = IBV_EXP_WQS_RDY; ^~~~~~~ IBV_WQS_RDY /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:209:13: error: ‘ibv_exp_modify_wq’ was not declared in this scope rt_assert(ibv_exp_modify_wq(wq, &wq_attr) == 0, "Failed to ready WQ"); ^~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:209:13: note: suggested alternative: ‘ibv_modify_wq’ rt_assert(ibv_exp_modify_wq(wq, &wq_attr) == 0, "Failed to ready WQ"); ^~~~~ ibv_modify_wq /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:8: error: use of enum ‘ibv_exp_query_intf_status’ without previous declaration enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK; ^~~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:48: error: ‘IBV_EXP_INTF_STAT_OK’ was not declared in this scope enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK; ^~~~~~~~ /root/dev/test/eRPC/src/transport_impl/raw/raw_transport.cc:212:48: note: suggested alternative: ‘IBV_QP_CUR_STATE’ enum ibv_exp_query_intf_status intf_status = IBV_EXP_INTF_STAT_OK; ^~~~~~~~ IBV_QP_CUR_STATE ~

anujkaliaiitd commented 3 years ago

Hi @syspro4. Thanks for your interest.

The current version of eRPC does not support the Raw transport anymore. I will mark it as broken in the CMakeLists.txt to avoid future confusion. If you wish to use the Raw transport, please use this version (https://github.com/erpc-io/eRPC/releases/tag/v0.1). You'll need to install an old version of Mellanox OFED (4.4 or older).

I'd recommend using eRPC's DPDK transport instead with DPDK 19.11 LTS. That will require installing Mellanox userspace drivers from rdma_core instead of Mellanox OFED.

syspro4 commented 3 years ago

Thanks for the reply.

Regarding support for latest OFED drivers how difficult it is to fix the issue? Can it be possible to change the Experimental IBVERBs and use the standard IBVERBs? If the changes are straight forward then I can fix it with some help.

anujkaliaiitd commented 3 years ago

It's not difficult if we ignore the kDumb flag in raw_transport.h. That flag enables optimizations like multi-packet RX queue and "overrunning CQ", which aren't really necessary. As you said, most of the work here involves replacing experimental verbs with standard ibverbs.

I will be happy to test and merge a PR that migrates eRPC's raw transport to standard ibverbs, while ignoring the kDumb flag.

syspro4 commented 3 years ago

Great! Thanks for the help! I will give it a try and let you know.

syspro4 commented 3 years ago

Hi, Please find attached patch which has a fix for raw transport issue. With this I am able to run hello_server & hello_client without any issue.

Following are the Mellanox versions I used: Server: MLNX_OFED_LINUX-4.7-3.2.9.0 Client: MLNX_OFED_LINUX-5.0-2.1.8.0

Note: In the patch I have kept kDumb code commented so that in the future we can fix it if needed. I can remove the kDumb code if it is not needed.

TODO: I am trying to understand the benchmark test and will run them later.

Questions:

  1. Will eRPC work on standard NICs (without DPDK)?
  2. I want to run RPC test which gives these numbers. "(Performance for small 32-byte RPCs: ~10M RPCs/sec with one CPU core, 60--80M RPCs/sec with one NIC.)" Can you please let me know how to run this test?

Please let me know if any changes needed.

Thanks, Vaibhav

eRPC_raw_transport_patch.tar.gz

anujkaliaiitd commented 3 years ago

Thanks so much! I've tested that it seems to work in the latency benchmark with rdma_core. May I suggest the following:

If you are unable to, I will make the required changes soon and merge.

Answers to your questions:

syspro4 commented 3 years ago

Thank you for testing the patch. I will sure address your comments and repost the patch.

I tried the server_rate app & I fixed some compilation issue and then ran ./scripts/do.sh from serve & client but I see following messages and I am not sure what to make out of the messages:

Note: I enabled LOG_LEVEL=info

On server side I only see following messages continuously. I do not see any progress "thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00"

On client side I see following message & nothing is happening after that: "Process 1, thread 15: Connected. Starting work."

Detailed logs: ====================== server side ==================================== server_dev :: ~/eRPC » cat scripts/autorun_process_file 192.168.2.222 31850 0 192.168.2.223 31850 0 server_dev :: ~/eRPC »

server_dev :: ~/eRPC » ./scripts/do.sh 0 0 Installing modded drivers do.sh: Launching process 0 on NUMA node 0 46:105830 INFOR: eRPC Nexus: Launching 0 background threads. 50:099920 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX] 50:100049 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 0]. Issue: None. Sending response. 50:100065 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 0]. 50:100441 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX] 50:100500 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 0]. Issue: None. Sending response. 50:100514 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 1]. 50:100858 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX] 50:100925 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 1]. Issue: None. Sending response. 50:100941 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 0]. 50:101425 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX] 50:101486 INFOR: Rpc 0: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 0]. Issue: None. Sending response. 50:101510 INFOR: Rpc 0: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 2]. 50:101917 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX] 50:101976 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 1]. Issue: None. Sending response. 50:101990 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 1]. 50:102902 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX] 50:102945 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX] 50:102971 INFOR: Rpc 1: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 1]. Issue: None. Sending response. 50:102988 INFOR: 50:102995 INFOR: Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 2]. Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 2]. Issue: None. Sending response. 50:103029 INFOR: Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 0]. 50:103376 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX] 50:103419 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX] 50:103433 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 2]. Issue: None. Sending response. 50:103463 INFOR: 50:103452 INFOR: Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 1]. eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX] 50:103521 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 2]. Issue: None. Sending response. 50:103537 INFOR: 50:103540 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 7, S: 3]. Issue: None. Sending response. Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 2]. 50:103558 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 0]. 50:104667 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX] 50:104710 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX] 50:104726 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 0, S: 3]. Issue: None. Sending response. 50:104754 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 1]. 50:104811 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 1, S: 3]. Issue: None. Sending response. . . . 50:134880 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 13, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 14]. eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX] Rpc 1: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 15]. 50:134909 INFOR: 50:134927 INFOR: Rpc 2: Received connect request from [H: 192.168.2.223:31850, R: 15, S: 2]. Issue: None. Sending response. 50:134945 INFOR: eRPC SM thread: Received SM packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: XX] Rpc 2: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 15]. 50:134980 INFOR: Rpc 3: Received connect request from [H: 192.168.2.223:31850, R: 15, S: 3]. Issue: None. Sending response. 50:134996 INFOR: Rpc 3: Sending packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 15, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 15]. thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 3: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 2: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 0: 0.00 M/s. rx batch -1.00, tx batch -1.00 thread 1: 0.00 M/s. rx batch -1.00, tx batch -1.00

====================== client side ==================================== client_dev :: ~/eRPC » cat scripts/autorun_process_file 192.168.2.222 31850 0 192.168.2.223 31850 0 client_dev :: ~/eRPC »

client_dev :: ~/eRPC » ./scripts/do.sh 1 0 Installing modded drivers do.sh: Launching process 1 on NUMA node 0 1 192.168.2.222 31850 02 192.168.2.223 31850 01 192.168.2.222 31850 02 192.168.2.223 31850 050:067736 INFOR: eRPC Nexus: Launching 0 background threads. 50:067751 INFOR: eRPC Nexus: Launching session management thread on core 23. 50:067887 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 192.168.2.223. 50:099929 INFOR: 50:100390 INFOR: 50:100396 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 0] Rpc 7: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]. 50:100426 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 1] 50:100754 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 0] Rpc 0: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: XX]. 50:100977 INFOR: RawTransport created for Rpc ID 15. Device mlx4_0/eth5, port 2. IPv4 192.168.2.223, MAC f4:52:14:74:d9:e1. Datapath UDP port 31897. 50:101279 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 2] 50:101417 INFOR: Rpc 1: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]. 50:101933 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 1, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 1] 50:102400 INFOR: 50:102402 INFOR: Rpc 0: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: XX]. Rpc 7: Sending packet [Connect request], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: XX]. 50:102455 INFOR: ERPC: Vaibhav num_devices 1 50:102752 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 0, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 2] 50:102798 INFOR: eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 7, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 0] eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 11] 50:144159 INFOR: 50:144167 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 11] for session 0. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 11] 50:144257 INFOR: 50:144259 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 11] for session 1. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 10] 50:144278 INFOR: 50:144280 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 10] for session 2. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 11, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 10] 50:144296 INFOR: 50:144297 INFOR: Rpc 11: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 10] for session 3. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 6, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 11] 50:144313 INFOR: 50:144319 INFOR: Rpc 6: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 11] for session 2. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 6, S: 3], server: [H: 192.168.2.222:31850, R: 3, S: 11] 50:155156 INFOR: 50:155162 INFOR: Rpc 6: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 11] for session 3. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 0], server: [H: 192.168.2.222:31850, R: 0, S: 12] 50:155177 INFOR: 50:155186 INFOR: Rpc 12: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 12] for session 0. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 1], server: [H: 192.168.2.222:31850, R: 1, S: 12] 50:166160 INFOR: 50:166165 INFOR: Rpc 12: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 12] for session 1. Issue: None. Session connected. eRPC SM thread: Received SM packet [Connect response], [No error], client: [H: 192.168.2.223:31850, R: 12, S: 2], server: [H: 192.168.2.222:31850, R: 2, S: 12]

50:210277 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 0, S: 15] for session 0. Issue: None. Session connected. 50:210288 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 1, S: 15] for session 1. Issue: None. Session connected. 50:210297 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 2, S: 15] for session 2. Issue: None. Session connected. 50:210307 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 15] for session 3. Issue: None. Session connected. 50:210307 INFOR: Rpc 15: Received connect response from [H: 192.168.2.222:31850, R: 3, S: 15] for session 3. Issue: None. Session connected. Process 1, thread 7: Connected. Starting work. Process 1, thread 0: Connected. Starting work. Process 1, thread 1: Connected. Starting work. thread_id: median_us 5th_us 99th_us 999th_us Mops Process 1, thread 5: Connected. Starting work. Process 1, thread 14: Connected. Starting work. Process 1, thread 4: Connected. Starting work. Process 1, thread 3: Connected. Starting work. Process 1, thread 8: Connected. Starting work. Process 1, thread 10: Connected. Starting work. Process 1, thread 2: Connected. Starting work. Process 1, thread 6: Connected. Starting work. Process 1, thread 11: Connected. Starting work. Process 1, thread 12: Connected. Starting work. Process 1, thread 9: Connected. Starting work. Process 1, thread 13: Connected. Starting work. Process 1, thread 15: Connected. Starting work.


anujkaliaiitd commented 3 years ago

I pushed the required compilation fixes to server_rate.cc. It works on my cluster with the raw transport.

In your case, I suspect that the datapath packets are getting dropped somewhere. You could try re-compiling eRPC with detailed packet-level logging (ex: cmake . -DLOG_LEVEL=trace). Then when you run the server_rate client and server, they will create files in /tmp with detailed per-packet logs.

syspro4 commented 3 years ago

You are correct. In the log I see packet drop error. I am getting it for hello_server/client test too.

Log: /tmp/erpc_trace_31850-rpc_0: 83:467073 TRACE: Rpc 0, lsn 0 (server_dev): TX [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Slot [num_tx 0, num_rx 0]. 83:467120 TRACE: eRPC RawTransport: Sending packet (idx = 0, drop = 0). SGE #1 72B, SGE #2 = 0B. pkthdr = [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Frame header = [ETH: dst f4:52:14:5e:60:e0, src f4:52:14:74:d9:e1, eth_type 2048], [IPv4: ihl 5, version 4, ecn 1, tot_len 58, id 0, frag_off 0, ttl 128, protocol 17, check 0, src IP 192.168.2.223, dst IP 192.168.2.222], [UDP: src_port 31882, dst_port 31882, len 38, check 0]. 83:472158 REORD: Rpc 0, lsn 0 (server_dev): Pkt loss suspected for req 8 ([num_tx 1, num_rx 0]). Action: Retransmitting requests. 83:472166 TRACE: Rpc 0, lsn 0 (server_dev): TX [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Slot [num_tx 0, num_rx 0]. 83:472175 TRACE: eRPC RawTransport: Sending packet (idx = 0, drop = 0). SGE #1 72B, SGE #2 = 0B. pkthdr = [type REQ, dsn 0, reqn 8, pktn 0, msz 16, magic 11]. Frame header = [ETH: dst f4:52:14:5e:60:e0, src f4:52:14:74:d9:e1, eth_type 2048], [IPv4: ihl 5, version 4, ecn 1, tot_len 58, id 0, frag_off 0, ttl 128, protocol 17, check 0, src IP 192.168.2.223, dst IP 192.168.2.222], [UDP: src_port 31882, dst_port 31882, len 38, check 0].

I did rping test between server & client and it is working fine.

server_dev :: ~/eRPC 1 » rping -s -C 10 -v <<< server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrst server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstu server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuv server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvw server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwx server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxy server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyzA server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10 server_dev :: ~/eRPC »

client_dev :: ~/eRPC » /usr/bin/rping -c -a 192.168.2.222 -C 10 -v ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuv ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvw ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwx ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxy ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyz ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[]^_abcdefghijklmnopqrstuvwxyzA client DISCONNECT EVENT... server_dev :: ~/eRPC »

Please help.

Thanks

syspro4 commented 3 years ago

Hi, I tried with different set of server/client but I see same "Pkt loss suspected for req 8" error in the trace log file. All these servers can do rping without any issue. Can you please help?

Thanks!

anujkaliaiitd commented 3 years ago

Couple of suggestions:

anujkaliaiitd commented 3 years ago

Ah, I just remembered that you were able to run the hello_world test, so CRC is likely not an issue. Still, it might be worth trying with the DPDK transport. Other suggestions:

syspro4 commented 3 years ago

Thanks for the suggestions.

Actually earlier hello_world also did not work. I retested it and I see same pkt loss message.

When you tested my patch on your setup did you changed anything in your code base?

syspro4 commented 3 years ago

I tried erpc dpdk but that is giving following errors.

server_dev :: ~/eRPC » ./build/hello_server 6:616156 INFOR: eRPC Nexus: Launching 0 background threads. 6:616202 INFOR: eRPC Nexus: Launching session management thread on core 23. 6:617241 INFOR: eRPC Nexus: Created with management UDP port 31850, hostname 192.168.2.222. 6:617452 INFOR: DPDK transport for Rpc 0 initializing DPDK EAL. EAL: Detected 32 lcore(s) EAL: Detected 2 NUMA nodes EAL: Auto-detected process type: PRIMARY EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.2 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.3 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:81:00.0 on NUMA socket 1 EAL: probe driver: 15b3:1003 net_mlx4 PMD: net_mlx4: PCI information matches, using device "mlx4_0" (VF: false) PMD: net_mlx4: 2 port(s) detected PMD: net_mlx4: port 1 MAC address is f4:52:14:5e:60:e0 PMD: net_mlx4: port 2 MAC address is f4:52:14:5e:60:e1 6:947277 WARNG: Running as primary DPDK process. eRPC DPDK daemon is not running. 6:947316 INFOR: DPDK transport for Rpc 0 got QP 0 6:947354 INFOR: dev_info.rx_desc_lim.nb_max 0 kNumRxRingEntries 4096 <========================== Added this debug print terminate called after throwing an instance of 'std::runtime_error' what(): Device RX ring too small [2] 10374 abort ./build/hello_server

server_dev :: ~/eRPC »

I tried running dpdk-helloworld test program to check if dpdk is configured and working correctly and it is working fine.

server_dev :: ~/dpdk-stable-19.11.5/build » examples/dpdk-helloworld -l 0-3 -n 4 EAL: Detected 32 lcore(s) /** EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.2 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:06:00.3 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:81:00.0 on NUMA socket 1 EAL: probe driver: 15b3:1003 net_mlx4 hello from core 1 hello from core 2 hello from core 3 hello from core 0 server_dev :: ~/dpdk-stable-19.11.5/build »

anujkaliaiitd commented 3 years ago

It's strange that the port reports zero as the max number of RX descriptors. Could you try:

This should show the number of RX ring descriptors detected by testpmd.

Also: I see your system has two CX3 ports. Are both connected to the switch? If not, please ensure that eRPC is using the correct port.

syspro4 commented 3 years ago

Thanks for the reply. I see following info:

testpmd> show port info 0

Infos for port 0 MAC address: F4:52:14:5E:60:E0 Device name: mlx4_0 port 1 Driver name: net_mlx4 Connect to socket: 1 memory allocation on the socket: 1 Link status: up Link speed: 40000 Mbps Link duplex: full-duplex MTU: 1500 Promiscuous mode: enabled Allmulticast mode: disabled Maximum number of MAC addresses: 128 Maximum number of MAC addresses of hash filtering: 0 VLAN offload: strip off, filter off, extend off, qinq strip off Hash key size in bytes: 40 Supported RSS offload flow types: ipv4 ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv6 ipv6-frag ipv6-tcp ipv6-udp ipv6-other user defined 15 user defined 16 user defined 17 Minimum size of RX buffer: 32 Maximum configurable length of RX packet: 65536 Maximum configurable size of LRO aggregated packet: 0 Current number of RX queues: 1 Max possible RX queues: 1024 Max possible number of RXDs per queue: 65535 Min possible number of RXDs per queue: 0 RXDs number alignment: 1 Current number of TX queues: 1 Max possible TX queues: 1024 Max possible number of TXDs per queue: 65535 Min possible number of TXDs per queue: 0 TXDs number alignment: 1 Max segment number per packet: 65535 Max segment number per MTU/TSO: 65535 testpmd>

testpmd> show config rxtx io packet forwarding packets/burst=32 nb forwarding cores=1 - nb forwarding ports=2 port 0: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=0 - RX free threshold=0 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=0 - TX free threshold=0 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=0 port 1: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=0 - RX free threshold=0 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=0 - TX free threshold=0 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=0 testpmd>

I don't see the ring descriptors in "info port" command but in "show config rxtx" I see 0 for descriptor value.

Regarding the CX3 ports, yes I have a card with 2 ports and the program is picking up correct port. If we manually select the port then how can I do that?

Thanks!