It turns out that fi_inject and friends can cause problems for the GNI provider for certain communication patterns. The osu_barrier test run with 4 endpoints (MPI processes) illustrates this well. The barrier timing part works fine, the problem is with the MPI_Reduce to get the timing statistics. With an unmodified GNI provider which has an inject size of 64 bytes, the test hangs. With an inject size of 0, the test passes. The problem has to do with the fact that the communication pattern at this job size differs between the barrier and reduce patterns. Note the reduce is a one-way operation for some ranks, they just send to a parent. In the case of the Open MPI OFI MTL, fi_tinject is used. If the VC connection from the sender to the receiver endpoint is not yet set up, the sender ranks end up in MPI_Finalize in an out-of-band barrier. The VC never got moved to completed state.
If the gni provider is modified to advertise an inject size of 0, the test passes.
There may be other osu collectives that have a similar behavior pattern.
It turns out that
fi_inject
and friends can cause problems for the GNI provider for certain communication patterns. The osu_barrier test run with 4 endpoints (MPI processes) illustrates this well. The barrier timing part works fine, the problem is with theMPI_Reduce
to get the timing statistics. With an unmodified GNI provider which has an inject size of 64 bytes, the test hangs. With an inject size of 0, the test passes. The problem has to do with the fact that the communication pattern at this job size differs between the barrier and reduce patterns. Note the reduce is a one-way operation for some ranks, they just send to a parent. In the case of the Open MPI OFI MTL,fi_tinject
is used. If the VC connection from the sender to the receiver endpoint is not yet set up, the sender ranks end up inMPI_Finalize
in an out-of-band barrier. The VC never got moved to completed state.If the gni provider is modified to advertise an inject size of 0, the test passes.
There may be other osu collectives that have a similar behavior pattern.