MPI needs a standard ABI

[jeffhammond]

Problem

This issue needs to exist so I can submit a pull request to solve it, because of our voting procedures.

The problem is described in some detail in https://arxiv.org/abs/2308.11214. I do not want to repeat it here.

Proposal

Define a standard ABI for MPI. This includes:

• describe how calling conventions work
• header, module and shared library names
• status object
• integer types
• handle types
• integer constants
• handle constants
• callback constants

We will need a way to detect the existence and versions of the ABI.

The primary impact is on C but there are Fortran aspects too.

Changes to the Text

Write a completely new chapter to define all of the above. Define relevant terms. State the constraints.

Add all the constants to tables.

Impact on Implementations

The implementation of this in an MPI library is not trivial but not profound, either.

There is a prototype in MPICH already: https://github.com/pmodels/mpich/pull/6390.

Impact on Users

Some users are desperate for this, because they are tired of compiling all their MPI software two or more times.

Use cases include:

• Python, Julia and Rust bindings.
• Users of containers.

References and Pull Requests

Pull request:

https://github.com/mpi-forum/mpi-standard/pull/875

Related issues:

https://github.com/mpi-forum/mpi-issues/issues/744 https://github.com/mpi-forum/mpi-issues/issues/743 https://github.com/mpi-forum/mpi-issues/issues/735 https://github.com/mpi-forum/mpi-issues/issues/709 https://github.com/mpi-forum/mpi-issues/issues/704 https://github.com/mpi-forum/mpi-issues/issues/702 https://github.com/mpi-forum/mpi-issues/issues/654 https://github.com/mpi-forum/mpi-issues/issues/642 https://github.com/mpi-forum/mpi-issues/issues/159 https://github.com/mpi-forum/mpi-issues/issues/107

[Wee-Free-Scot]

@jeffhammond -- Feedback from Intel (@alexander-sannikov, @ddurnov, @garzaran):

1. We're looking at how extensible the ABI might be -- a specific example would be "what if there were a C integer type different to MPI_Int, MPI_Long, etc. that needed to be added to the ABI? There is no room in the 8:3 allocation for C integer types" -- I know provision has been made in some categories, but not all. How will awkward extensions to the standard ABI be handled? We are thinking about new categories in the 12:9 part assigned to "extension_01" etc. which gives us an extra 8:3 space for new values.

2. Separately, we are expecting that non-standard experimental features can be added into the ABI header file prior to standardisation without breaking compatibility, as long as the extension does not re-use a constant value already assigned to something else. For example, MPIX_ERR_INVALID_NOTIFICATION = 891 for the notified-RMA proposal. Of course, the specific values are subject to change during the subsequent standardisation effort.

3. We would like to add an advice to users along the lines of "The MPI handles might not be valid pointers. The user must never attempt to dereference any MPI handle, even when it is defined/declared using an incomplete pointer typedef." -- I think this is okay and doesn't change anything, it just clarifies what should be common sense.

Ultimately, the extensibility questions push in the direction of asking "how useful is the Huffman code going forward?"

[jeffhammond]

1. I would add it in the # other C/C++ types branch of the Huffman code here:

   568 001000111000 MPI_C_BOOL
   569 001000111001 MPI_CXX_BOOL
   570 001000111010 reserved datatype
   571 001000111011 reserved datatype
   572 001000111100 MPI_WCHAR
   573 001000111101 reserved datatype
   574 001000111110 reserved datatype
   575 001000111111 reserved datatype

   There is also plenty of space after the # 32 byte Fortran category.

We can also add another branch for datatypes but before we do that, we ought to think whether we should instead use the unnamed predefined datatype route, which is what we discussed in Boston (unfortunately, it was offline) for solving the AI float types problem. We have these for F90 kind but it can be generalized. I will write it up in detail if necessary.

[jeffhammond]

2. Can you make the MPIX error codes larger than MPI_ERR_LASTCODE? That seems like the easy solution.

[jeffhammond]

3. We can add that, although I must say, I don't think anyone has ever dereferenced an Open MPI handle before, so I doubt it's necessary.

[jeffhammond]

"how useful is the Huffman code going forward?"

I think it's useful and I worked really hard to make sure it's future-proof. We are using less than 1/3 of the 1024 bits I intended to use, and we can go all the way up to 4095 if necessary. I think if we are considering adding a lot more predefined handles, we ought to evaluate whether that is the right design in the first place.

[Wee-Free-Scot]

1. Thanks for the reminder about the "unnamed predefined datatype" idea. That would likely solve a lot of the datatype extensibility issues.
2. Hmm, maybe a new error code wasn't the best specific example. Unless you're advocating that ALL experimental constants should have values above the 4096 reserved ones until they are standardised and a value in [0,4095] is assigned by the MPI Forum? That might work as a general answer.
3. The advice should be common sense, but we're dealing with users here :)

Your work on this proposal is valuable and we're all grateful you've taken on another topic like this (large count, etc.). Apologies if my phraseology gave the wrong impression there. Perhaps we won't know the answer to this until we've all implemented the existing scheme and then extended it a bunch of times in the future.

[jeffhammond]

2. For handles, as long as an implementation can guarantee that it will not create a user handle that conflicts with the predefined one, it can use any value. Since Intel MPI does not create user handles with malloc anyways, reserving the zero page is not relevant.

However, if you want, we can reserve the last 1024 bits of the predefined space for experimentation. I think that's more than enough.

This is valuable feedback and I appreciate the care taken in making the proposal better.

[Wee-Free-Scot]

I think a specific reservation for experimentation (and differentiation) would be very useful. Thanks @jeffhammond. The advice to users is belt-and-braces, but it heads-off a potential misinterpretation problem in future and does no harm. Both changes are no-no-vote size adjustments, IMHO.