worleyph
commented
7 years ago Note that I have not observed any problems with using MPI_RSEND or MPI_IRSEND in the MCT code (though this has only been in there very recently). Even in PIO, the only issues have been with the MPAS Ocean history writes, and I do not know why it is only with these particular history writes. However, the PIO issue is resolved by either mapping MPI_RSEND to MPI_SEND or by using an older version of the cray-mpich library (e.g. 7.2.5). I'd prefer to apply a blanket bandaid, to MCT, PIO, and CAM, to avoid repeating the months-long-now debugging exercise we have been going through (assuming that the NERSC issues have the same cause).
The above has been "tested" in ACME cases, and works fine as far as I can tell. Given the nature of the change, an error would likely have shown up in the compile or immediately in the run.
rljacob
MPI_RSEND and MPI_IRSEND are not implemented correctly in recent versions of the MPI library provided by Cray. This modification maps these to MPI_SEND and MPI_ISEND, respectively, except when the CPP token _USE_MPI_RSEND is defined during compilation. Note that the default is now NOT to use MPI_RSEND. The use of MPI_RSEND and MPI_IRSEND should be a performance enhancements, and has been in the past and with other compilers, so prefer to retain this as an option for future evaluation.