MPI_Type, MPI_Alltoallw, mpp_global_field update

[aidanheerdegen]

This is the work of @marshallward

git cherry-pick d216cfd4506be1f369c8a265cb40e9f9b34321c2

This patch contains three new features for FMS: Support for MPI datatypes, an MPI_Alltoallw interface, and modifications to mpp_global_field to use these changes for select operations.

These changes were primarily made to improve stability of large (>4000 rank) MPI jobs under OpenMPI at NCI.

There are differences in the performance of mpp_global_field, occasionally even very large differences, but there is no consistency across various MPI libraries. One method will be faster in one library, and slower in another, even across MPI versions. Generally, the MPI_Alltoallw method showed improved performance on our system, but this is not a universal result. We therefore introduce a flag to control this feature.

The inclusion of MPI_Type support may also be seen as an opportunity to introduce other new MPI features for other operations, e.g. halo exchange.

Detailed changes are summarised below.

• MPI data transfer type ("MPI_Type") support has been added to FMS. This is done with the following features:

  • A mpp_type derived type has been added, which manages the type details and hides the MPI internals from the model developer. Types are managed inside of an internal linked list, datatypes.

  Note: The name mpp_type is very similar to the preprocessor variable MPP_TYPE_ and should possibly be renamed to something else, e.g. mpp_datatype.*

  • mpp_type_create and mpp_type_free are used to create and release these types within the MPI library. These append and remove mpp_types from the internal linked list, and include reference counters to manage duplicates.

  • A mpp_byte type is created as a module-level variable for default operations.

  NOTE: As the first element of the list, it also inadvertently provides access to the rest of datatypes, which is private, but there is probably some ways to address this.*

• A MPI_Alltoallw wrapper, using MPI_Types, has been added to the mpp_alltoall interface.

• An implementation of mpp_global_field using MPI_Alltoallw and mpp_types has been added. In addition to replacing the point-to-point operations with a collective, it also eliminates the need to use the internal MPP stack.

  Since MPI_Alltoallw requires that the input field by contiguous, it is only enabled for data domains (i.e. compute + halo). This limitation can be overcome, either by copying or more careful attention to layout, but it can be addressed in a future patch.

  This method is enabled in the mpp_domains_nml namelist group, by setting the use_alltoallw flag to True.

Provisional interfaces to SHMEM and serial ("nocomm") builds have been added, although they are as yet untested and primarily meant as placeholders for now.

This patch also includes the following changes to support this work.

• In get_peset, the method used to generate MPI subcommunicators has been changed; specifically MPI_Comm_create has been replaced with MPI_Comm_create_group. The former is blocking over all ranks, while the latter is only blocking over ranks in the subgroup.

  This was done to accommodate IO domains of a single rank, usually due to masking, which would result in no communication and cause a model hang.

  It seems that more recent changes in FMS related to handling single-rank communicators were made to avoid this particular scenario from happening, but I still think that it's more correct to use MPI_Comm_create_group and have left the change.

  This is an MPI 3.0 feature, so this might be an issue for older MPI libraries.

• Logical interfaces added to mpp_alltoall and mpp_alltoallv

• Single-rank PE checks in mpp_alltoall were removed to prevent model hangs with the subcommunicators.

• NULL_PE checks have been added to the original point-to-point implementation of mpp_global_field, although these may not be required anymore due to changes in the subcommunicator implementation.

  This work was by Nic Hannah, and may actually be part of an existing pull request. (TODO: Check this!)

• Timer events have been added to mpp_type_create and mpp_type_free, although they are not yet initialized anywhere.

• The diagnostic field count was increased from 150 to 250, to support the current needs of researchers.

[aidanheerdegen]

Hey @marshallward I am just merging this work you did into our FMS fork. Can you just confirm that this is ok, I assume the code is fine, it is what was accepted into the upstream FMS but I guess we had to branch before it was merged into master.

[marshallward]

Not really in a good position to test it out, but it looks OK to me. If it's not breaking your runs then I suspect it's fine to merge.