Replace MPI calls with ESMF calls

ESCOMP / MOSART

Model for Scale Adaptive River Transport, Mosart, part of the Community Earth System Model

http://www.cesm.ucar.edu/

Other

8 stars 27 forks source link

Replace MPI calls with ESMF calls #100

Open mvertens opened 4 weeks ago

mvertens commented 4 weeks ago

In MOSART - the calls using shr_mpi_sum, shr_mpi_max, shr_mpi_min, mpi_barrier, mpi_bcast could be replaced by calls to ESMF_VMAllReduce and ESMF_VMBroadcast. The following must be taken into account, however:

ESMF_VMAllReduce only accepts one-dimensional arrays. tmp_in and tmp_glob are 2d-arrays. So copies will need to be made and 6 calls will need to be done and then copies need to be made.
Using ESMF_VMBroadcast -only accepts 1d arrays. That means that an array of length 1 needs to be created and then a copy of finidat needs to be made to that and then broadcast and then copied back. I think this will make the code more complicated and harder to read.

ESMF currently only has interfaces to a small subset of MPI communications and only supports arrays in those interfaces. So this needs to be taken into consideration in the decision to replace the MPI calls with ESMF calls.

ekluzek commented 4 weeks ago

@mvertens thanks for the analysis there. It sounds like at least for MOSART this wouldn't be a good thing to do right now anyway. @billsacks as our ESMF rep. do you think this would be good to do in the future?

@jedwards4b made the following point about doing this here:

https://github.com/ESCOMP/MOSART/pull/94#pullrequestreview-2101937327

Since, ESMF VM objects abstract the distributed memory communication package it's a general interface where MPI could be switched to another protocol, but all under the covers without touching CESM code. That's a worthwhile thing to do even if we don't do it immediately.

billsacks commented 4 weeks ago

I'd like to understand this better before weighing in on it myself. It sounds like it may complicate the code somewhat, but buys us a potential for more abstraction / long-term maintainability.

Things I'd like to understand - and this probably warrants a discussion with others on the ESMF core team - are:

What is the likelihood of needing to move to something other than mpi? I assume, that, in the near future, this would be for GPUs???
Is it reasonable for ESMF to support whatever alternative(s) CESM might want? i.e., will ESMF_VMAllReduce and ESMF_VMBroadcast actually support the potentially-desired underlying calls?
How much work would it be for ESMF to extend its wrappers to keep the CESM code simpler? If feasible, it seems like that would be better than adding complexity in the CESM code due to limitations in the ESMF wrappers.

jedwards4b commented 4 weeks ago

Exactly the discussion you should have. We may need to move away from mpi or to some combination of mpi with something else - ideally we would keep that complexity isolated. I think that ESMF_VM* will need to be extended to work with scalars and multidimensional arrays. I don't think it would be too complicated and I think this embraces a vision of what the esmf interface should be capable of.

billsacks commented 4 weeks ago

Sounds great, @jedwards4b . I'll add this to the agenda for an upcoming ESMF core team call. The team is stretched thin right now, but we can discuss whether it makes sense to put this on our roadmap.

billsacks commented 3 weeks ago

@jedwards4b - one thing that could help for this discussion is: Do you have any examples of alternative calls? e.g., for mpi_bcast, what would be a specific alternative that could be wrapped by the ESMF calls?

jedwards4b commented 3 weeks ago

The CMEPS mediator is an example, there are no calls to MPI there, only ESMF is used.

billsacks commented 3 weeks ago

Sorry, let me clarify: what I meant is: what would be the underlying library call that takes the place of mpi_bcast that the ESMF_VMBroadcast might wrap in the future?

jedwards4b commented 3 weeks ago

I don't have any examples - perhaps Gerhard does.