Some thoughts about semantics

[devreal]

Some thoughts I had while browsing through the code:

• How do these operations interact with the execution space? For example, is the user required to fence before calling any consuming operation (like isend)? Non-packing isend is not interacting with the execution space (https://github.com/cwpearson/kokkos-comm/blob/develop/src/impl/KokkosComm_isend.hpp#L54)
• Do you define that the semantics are the exact same as prescribed by the MPI standard or do you allow deviation? One such deviation to consider could be the use of mpi_assert_allow_overtaking option in MPI 4.0 (disabling ordering of same-tag messages). Either way, the semantics should be well-defined.
• Do you require the MPI implementation to be device-aware? This should be documented since there are implementations (or configurations of implementations) that may not support the underlying device hardware.

[cedricchevalier19]

• We have to think more about asynchronous communication handling. We do not want for the user to fence everywhere.

• I understand your concerns about the semantics and the MPI standard and it was one of the reasons to move from the KokkosMPI name as I am not sure we can strictly follow MPI.

• I think we do not want to require a device-aware MPI implementation. So the code will be less straightforward, with explicit deep_copy and temporary buffers etc.