The mapping of MPI processes to the physical hardware is crucial for the performance of MPI communication.
Achieving a good mapping is currently left mainly to users via MPI library dependent mpirun arguments. The alternative virtual topology interface provides some potential to pass information about communication paths to the MPI library - and allow reordering of processes. However, the existing interfaces seem to lack at the moment. MPI_Cartcreate and the MPI*Graph interface target this but due to lack of information, e.g., as there might be other communication paths being used or it is unclear how provided edge weights have to be interpreted - implementations did not take action in many cases.
Proposal
As a first step, introduce an info key for MPI_DIstgraph_create which asserts that only P2P communication will be performed only over the specified edges.
Problem
The mapping of MPI processes to the physical hardware is crucial for the performance of MPI communication. Achieving a good mapping is currently left mainly to users via MPI library dependent mpirun arguments. The alternative virtual topology interface provides some potential to pass information about communication paths to the MPI library - and allow reordering of processes. However, the existing interfaces seem to lack at the moment. MPI_Cartcreate and the MPI*Graph interface target this but due to lack of information, e.g., as there might be other communication paths being used or it is unclear how provided edge weights have to be interpreted - implementations did not take action in many cases.
Proposal
As a first step, introduce an info key for MPI_DIstgraph_create which asserts that only P2P communication will be performed only over the specified edges.
Changes to the Text
Add info key "assert_p2p_on_edges_only"
Effect on implementations
TBD
Effect on users
TBD