Improve MPI Module

[ben-albrecht]

Next steps for the MPI module.

Features

• [ ] Extend functionality
  • Extend to MPI-2 and 3 routines
    • We currently support most of MPI-1, and a random subset of MPI-2 and MPI-3.
    • We should support a more rational subset of these routines (we may not want all of them)
• [ ] Support --spmd with CHPL_COMM=none for all launchers
  • Can we make this only available when using MPI module or selected configurations?
  • Should this wait for a longer-term launcher overhaul?
• [ ] Extend support to Intel compilers
  • May need a new target compiler analogous to mpi-gnu
• [ ] Support for CHPL_TARGET_ARCH in mpi-gnu.
  • Currently set to unknown which produces a warning with --specialize (from --fast)
• [ ] Add a writeln-equivalent for MPI SPMD mode that always prints on rank 0
• [x] Support qthreads (#6192)
  • [x] Tackle known deadlock test case: #4193
  • [x] Drop-in replacements for blocking MPI calls #6314
• [X] Fix destructor order to resolve dereference nil bug for _initMPI object (#5573)

Testing

• [ ] More feature testing (bcast, send, recv, ....)

  • [easy]

• [ ] Support/testing for MPI_THREAD_SERIALIZED

  • We currently recommend and test with MPI_THREAD_MULTIPLE, which allows multiple threads to make concurrent MPI calls
  • It would be good to test and see if MPI_THREAD_SERIALIZED works in the case where only one thread is making an MPI call at any given time

• [ ] Check support for latest OpenMPI (related: #6516)

  • OpenMPI 1 unsupported
  • OpenMPI 2+ suspected to work, but needs confirmation and testing

• [x] Add PREDIFF to MPI initialization tests (#6213)

• [x] Test MPI module over ugni or gasnet with a more performant backend (#5924)

• [x] Utilize recursive <directory>.skipif instead of SKIPIF per directory (#6213)

• Nightly testing

  • [x] linux64 SPMD
  • [x] linux64 multilocale - gasnet+mpi (#6214)
  • [ ] linux64 multilocale - gasnet+ibv
  • [ ] cray SPMD / multilocale - ugni+aries

[ben-albrecht]

Credit to @npadmana for several of the above ideas (and the MPI module itself!)

[ben-albrecht]

Marked [easy] for the additional testing bullet.

[npadmana]

@ben-albrecht -- I think we can cross off the qthreads stuff here, plus close #6192. Whee.

[ben-albrecht]

Removing [good first issue] label here and will apply [good first issue] to sub-issues where are appropriate.