Closed maddyscientist closed 11 years ago
Having spoken to Rolf vanderVaart (OpenMPI maintainer) about this, I found out that these are essential if we want to be able to use GPUDirect and actually reduce latency. We could conceivably leave a fall-back option if this is needed.
This issue was closed with 50eef47ae631cb6339af587cc7066ae53278e010.
do you guys think that now it is a good moment to plug APEnet P2P+RDMA support ?
Almost. That's why I have working on cleaning up the comms.
I wonder if the best way is to provide an APEnet comm backend (like for the qmp, mpi and single backends that already exist) or whether to a qmp-APEnet interface? See include/comm_quda.h.
This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by
il like that comm_declare_receive_relative() and its twin. those are easily ported to a RDMA GPU-aware API. If the frame communication were implemented in terms of those two functions, it'd perfect.
Let's got with this the, Davide, can you provide a comm_apenet backend then that implements this (and the other functions in comm_quda.h). You only need to implement the functions that comm_qmp and comm_single implement, eventually all of the additional ones that comm_mpi implements will be deprecated and the code that uses them will be converted to use persistent message handlers like QMP does.
Currently, MPI communicators are created on demand. This will not play nicely with peer-2-peer MPI between devices since the cost to set up the handshaking is significant. Thus all MPI communications should be done persistent communicators, that are set up when the FaceBuffer is created. This also aligns better with the QMP model.