wxdwfc
commented
5 years ago Hi,
Thanks for trying ROCC !
Basically, ROCC is guaranteed to work correctly, if the condition is satisfied. I think the main problem is that: RWorker assumes each machine has the same number of worker threads. I see that you create 3 workers on client, while the server has 4. So the 4th thread on the server will try to connect to the QP created on the 4th thread at the client (since each QP can only be connected once). So it stucks.
The simple solution is to remove this code, and manually connect QPs using the connect() method in RLib.You can reference the LibRDMA’s readme to manually connect QPs.(https://github.com/wxdwfc/rlib/ https://github.com/wxdwfc/rlib/)
- Xingda
在 2019年2月20日，下午6:22，Alchem Lab notifications@github.com 写道：
Hi rocc developers,
I am trying to use the rocc framework in my own code to support rdma-based communication. Basically I am trying to use the RWorker class to as the base class in my own code to model thread creation and routine scheduling. My thread class inherits the RWorker class just like bench_workers did in rocc's code. However, I met with some difficulties using the rocc framework as well as the librdma library. As one simple demo, I started one server node and one client node. The server node spawns 4 RWorkers and the client node spawns 3 RWorkers. At the time when the client finished initializing all the 4 RWorkers, some of the server workers stuck in the rdmaio::RdmaCtrl::link_connect_qps() function and cannot connect qps successfully to the other node and thus they retry forever (see the while(1) loop in the link_connect_qps function). Essentially, the PreConnector::get_send_socket() function called by the Qp::connect_rc() function will always return a negative socket value will will cause the next retry. Even the recv_thread spawned by the librdma library are good in consistently accepting new tcp connection requests, get_send_socket function is consistently fail. I noticed that link_connect_qps() function will retry every 200ms until all qps in the cluster are linked. Is this guranteed to work correctly? In my case, it indeed connects forever. I am wondering if you guys have any idea to help me solve this issue. Thank you!
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/SJTU-IPADS/drtmh/issues/3, or mute the thread https://github.com/notifications/unsubscribe-auth/ADNREo4GPY_uFbEj2TjkjuJatTIXzt6Aks5vPRO5gaJpZM4bEwwD.
albertghtoun
Hi rocc developers,
I am trying to use the rocc framework in my own code to support rdma-based communication. Basically I am trying to use the RWorker class to as the base class in my own code to model thread creation and routine scheduling. My thread class inherits the RWorker class just like bench_workers did in rocc's code. However, I met with some difficulties using the rocc framework as well as the librdma library. As one simple demo, I started one server node and one client node. The server node spawns 4 RWorkers and the client node spawns 3 RWorkers. At the time when the client finished initializing all the 4 RWorkers, some of the server workers stuck in the rdmaio::RdmaCtrl::link_connect_qps() function and cannot connect qps successfully to the other node and thus they retry forever (see the while(1) loop in the link_connect_qps function). Essentially, the PreConnector::get_send_socket() function called by the Qp::connect_rc() function will always return a negative socket value and thus will cause the next retry. Even the recv_thread spawned by the librdma library are good in consistently accepting new tcp connection requests, get_send_socket function consistently fail. I noticed that link_connect_qps() function will retry every 200ms until all qps in the cluster are linked. Is this guranteed to work correctly? In my case, it indeed connects forever. I am wondering if you guys have any idea to help me solve this issue. Thank you!