retest OpenMPI, jucx, infinileap and disni when SoftiWarp or SoftRoCe module is loaded

[subes]

https://github.com/zrlio/softiwarp

software based infiniband (similar to TCP/SCTP)

[subes]

also try again to run RMA put/get tests in jucx

[subes]

https://www.reflectionsofthevoid.com/2020/07/software-rdma-revisited-setting-up.html

Reflections Of The Void Software RDMA revisited setting up SoftiWARP on Ubuntu 20.04.pdf

Requires the use of an actually connected network interface:

modprobe siw
ifconfig #to find a connected ethernet or wifi module, "lo" did not work
sudo rdma link add siw0 type siw netdev wlp112s0
rdma link #should list the new device
ifconfig #to find the ip address of wlp112s0
rping -s -a 192.168.0.20 -v #server
rping -c -a 192.168.0.20 -v #client
sudo rdma link delete siw0 #call this during a test to verify that the interface is used, test should crash

[subes]

ucx does not support iWarp as it seems: https://github.com/openucx/ucx/issues/2507

They have some commits for it but say it is untested since 2017? At least I can not get it to work with SoftiWarp.

Also seems as if the code does not support iWarp because it checks for only Infiniband? https://github.com/openucx/ucx/blob/eadd74f9fe5b0edc081ba1ce589fb850d6809934/src/uct/ib/base/ib_md.c

[subes]

Alternative is rdma_rxe (similar to UDP, though seems to keep packet order?): https://enterprise-support.nvidia.com/s/article/howto-configure-soft-roce (though outdated https://github.com/linux-rdma/rdma-core/commit/0d2ff0e1502ebc63346bc9ffd37deb3c4fd0dbc9)

modprobe rdma_rxe
ifconfig #to find a connected ethernet or wifi module, "lo" did not work
sudo rdma link add rxe0 type rxe netdev wlp112s0
rdma link #should list the new device
ifconfig #to find the ip address of wlp112s0
rping -s -a 192.168.0.20 -v #server
rping -c -a 192.168.0.20 -v #client
sudo rdma link delete rxe0 #call this during a test to verify that the interface is used, test should crash

Hadronio works with Soft-RoCe, our Jucx integration requires that the listener does not get closed (which is now the default).

This here suggests Soft-RoCe can improve performance of normal networks cards as well: https://www.reflectionsofthevoid.com/2011/08/soft-roce-alternative-to-soft-iwarp.html https://www.lanl.gov/projects/national-security-education-center/information-science-technology/_assets/docs/2010-si-docs/Team_CYAN_Implementation_and_Comparison_of_RDMA_Over_Ethernet_Presentation.pdf

[subes]

Soft-RoCe Checklist:

• [x] BlockingHadroNIO (does not automatically pick up RoCe)
• [x] HadroNIO
• [x] NettyHadroNIO (hangs without Bidi after ~500k records)
• [x] JUCX (hangs with PeerErrorHandling enabled after ~5 million records)
• [x] OpenMPI (does not automatically pick up RoCe, don't know the required params)
• [x] Infinileap (impossible due to frequent JVM crashes)
• [x] Disni (seems to be the fastest library that works both on iWarp and RoCE)
• [x] Neutrino (can't get it to work, it says bad argumen when exchanging connection details internally)

RoCe hangs might be due to unreliability of the protocol: https://github.com/zrlio/disni/issues/37#issuecomment-458055469

SoftiWarp Checklist:

• [x] BlockingHadroNIO (ucx does not support iWarp)
• [x] HadroNIO (ucx does not support iWarp)
• [x] NettyHadroNIO (ucx does not support iWarp)
• [x] JUCX (ucx does not support iWarp)
• [x] OpenMPI (does not automatically pick up iWarp, don't know the required params)
• [x] Infinileap (impossible due to frequent JVM crashes)
• [x] Disni (seems to be the fastest library that works both on iWarp and RoCE)
• [x] Neutrino (can't get it to work, it says bad argumen when exchanging connection details internally)

[subes]

finished