search

cms-patatrack / cmssw

CMSSW fork of the Patatrack project
https://patatrack.web.cern.ch/patatrack/index.html
Apache License 2.0
2 stars 5 forks source link

Remote offload: try RDMAoE via OpenMPI 4.0 or UCX #226

Open felicepantaleo opened 5 years ago

fwyzard commented 5 years ago

According to Wikipedia, the two widely used RDMAoE protocols are RoCE v1/v2 and iWARP:

Intel and Chelsio recommend and exclusively support iWARP. Mellanox, Xilinx, and Broadcom recommend and exclusively support RoCE/RoCEv2.

From https://www.open-mpi.org/faq/?category=openfabrics#run-ucx:

51. Does Open MPI support UCX?

Yes.

UCX is an open-source optimized communication library which supports multiple networks, including RoCE, InfiniBand, uGNI, TCP, shared memory, and others. UCX mixes-and-matches transports and protocols which are available on the system to provide optimal performance. It also has built-in support for GPU transports (with CUDA and RoCM providers) which lets RDMA-capable transports access the GPU memory directly.

From Open MPI: Major User-Noticeable Changes Across Major Release Series:

v4.0.x series

In the Open MPI v4.0.x series, the OpenFabrics "openib" BTL will still be used — by default — for RoCE and iWARP networks (although UCX works fine with these networks, too). Users can force the use of UCX for RoCE and iWARP networks, if desired (see this FAQ item).