Enable MPI_Alltoallv collective calls with sub-communicators in Rearranger

[amametjanov]

Enable MPI_Alltoallv collective calls with local communicators in Rearranger. Add an element to Router called pe_list_loc which contains the MPI task ids for the local communicator the Router is initialized with. Then when calling Rearranger AND using mpialltoallv, use the the pe_list_loc to figure out the communication AND call mpialltoallv with the local communicator as the argument.

Fixes MCSclimate/MCT#54

[BFB]

[amametjanov]

Checked with ./cime/scripts/create_test --machine anvil SMS.ne30_ne30.FC5AV1C-H01A and

$ tail -n 2 user_nl_cpl 
!------------------------------------------------------------------------
mct_usealltoall = .true.

for default pelayout on 77 nodes (and CPL+ATM on joint nodes)

$ ./pelayout 
Comp  NTASKS  NTHRDS  ROOTPE
CPL :   2700/     1;      0
ATM :   2700/     1;      0
LND :     72/     1;   2700
ICE :     72/     1;   2628
OCN :     72/     1;   2700
ROF :     72/     1;   2700
GLC :      1/     1;      0
WAV :      1/     1;      0
ESP :      1/     1;      0

and problem layout on 21 nodes (and CPL+ATM on disjoint nodes)

$ ./pelayout
Comp  NTASKS  NTHRDS  ROOTPE
CPL :     72/     1;    684
ATM :    675/     1;      0
LND :    684/     1;      0
ICE :     72/     1;    684
OCN :     72/     1;    684
ROF :     72/     1;    684
GLC :      1/     1;      0
WAV :      1/     1;      0
ESP :      1/     1;      0

[worleyph]

@amametjanov , thanks for doing this. I've been traveling, and dealing with some other distractions now that I am back, but I will look at this by Friday (I hope).

[rljacob]

No change to answers in MCT system test.

Note that the MatVecMul routine is hardcoded to use alltoall and swapm when called with a SparseMatrixPlus (which is how all the interpolations in the cpl7 coupler are done).

     call Rearrange(xAV, xPrimeAV, sMatPlus%XToXPrime,  &
                    tag=sMatPlus%Tag, Vector=usevector, &
                    alltoall=.true., handshake=.true.   )

[rljacob]

Also BFB in an F-case.