Closed Nik-Sch closed 4 years ago
cobbwho
commented
4 years ago Hi, I have a problem similar to yours. Did you find a solution?
When I test softroce with ib_write_bw, I can get the right average bandwidth after the test. But then the network card seems to crash, I can't use commands like ifconfig or rxe_cfg, and ssh is still alive. The reboot command does not work, but a physical restart works, just like you mentioned.
Nik-Sch
commented
4 years ago No I was not able to solve it. I was also having some bandwidth problems in the VirtualBox Environment and as the virtualization was just early testing I just proceeded to "real" servers and the problem was gone. In general I experienced that these kernel modules were not perfectly working in virtualized environment but I cannot directly show any dmesg outputs or similar. I have also seen a lot of stability improvements in the new Mellanox OFED drivers (4.7) which render rxe_cfg deprecated in favor of other tools provided (Mellanox docs).
cobbwho
commented
4 years ago Hi, bro. Thank you so much for your prompt answer. Although I still don't know why it caused the crash, I found a way to avoid the computer crash.
The point is the state of the computer. For my program testing process, when I use this command ./rxe_cfg start , any new ssh connection will cause the computer to crash(the reboot command does not work, you must restart it manually). But as long as the computer does not create a new ssh connection, there will be no problems.
In addition, when I use this rmmod module or this ./rxe_cfg stop after, the new ssh connection does not cause the computer to crash.
Keepmoving-ZXY
commented
3 years ago I encounter the same error when run in a non root linux user.
Keepmoving-ZXY
commented
3 years ago I encounter the same error when run in a non root linux user.
And no error occurs when run code in root user.
Setup: I have a host and a VirtualBox System, both with installed SoftRoCE on a current Ubuntu. I am implementing a client/server model where the client connects to the server, they exchange basic information like size and a memory region and then the client transfers a file via RDMA Write to the server. After the transfer finished the server disconnects from the client. For clarification I attached a screenshot from Wireshark capturing two such file transfers where
192.168.99.1is the "client" and192.168.99.102is the "server". However, when I tried some stress testing, at some point the server just did not respond to a ConnectionRequest at all. The strsss test looked like following: A script executed the program described above consecutively with different files and file sizes. This works pretty good for about the first 1000 files. However, after that the client tries to connect to the server (in wireshark I see the ConnectRequests) but the server doesn't respond at all and eventually the client gives up (after defined 16 cm retries). WithdmesgI got the following kernel error:Trying to restart rxe with
rxe_cfg stop && rxe_cfg startdoesn't work becauserxe_cfg stoptimes out. When rebooting the server (shutting down times out, so I unplug the power and reboot), however, everything works as expected again.Does anyone have an idea what the problem is or how I could fix it? If needed I surely can supply further system information on monday.