cb-geo / mpm

CB-Geo High-Performance Material Point Method
https://www.cb-geo.com/research/mpm
Other
243 stars 83 forks source link

MPI_TAG overflows for large number of domain shared nodes with Halo exchange #620

Closed kks32 closed 4 years ago

kks32 commented 4 years ago

Describe the bug Running a large problem where the number of shared nodes exceeds the capacity of MPI_TAG a 16-bit signed integer, we have an overflow problem in the MPI_TAG.

Abort(671723268) on node 3 (rank 3 in comm 0): Fatal error in PMPI_Isend: Invalid tag, error stack:
PMPI_Isend(148): MPI_Isend(buf=0x7fff7328ea78, count=1, MPI_DOUBLE, dest=5, tag=524547, MPI_COMM_WORLD, request=0xc54a70fc) failed
PMPI_Isend(95).: Invalid tag, value is 524547

To Reproduce Run a problem where the number of domain shared nodes exceed 32,768.

Expected behavior Pass MPI_TAG without an overflow

Runtime environment (please complete the following information):

Additional context How can we alter this prototype code to avoid MPI_TAG overflows? At the moment, the node id or particle id is used as the MPI_TAG.