delete deprecated features of the C API as part of ABI standardization

[jeffhammond]

Problem

If we support deprecated features like MPI_HOST, MPI_User_function and MPI_Attr_set in the ABI, we have to keep them forever or break ABI backwards-compatibility to delete them.

Most of these features are obsolete and have been deprecated for a while. They should be deleted now.

Proposal

Delete all deprecated features from the C API.

If people don't want to delete the _x functions for large-count, then we can specify they are #define to the _c equivalents now, so the ABI doesn't have to have them.

Changes to the Text

TODO

Impact on Implementations

See https://github.com/mpiwg-abi/header_and_stub_library/pull/32/files for a list of changes that will occur and how implementations can continue to support deleted APIs if their users want them.

Impact on Users

Users of deprecated features will not be able to compile against MPI 4.2.

References and Pull Requests

https://github.com/mpi-forum/mpi-issues/issues/751

[eschnett]

An ABI for MPI 4.2 needs to support all features of MPI 4.2, including deprecated ones. Switching an application from using a vendor-specific ABI to a standard ABI should not require any source changes.

It would be fine if the respective features could be implemented via #define, but I don't think the MPI standard allows this.

[jeffhammond]

The tradeoff is between breaking API for deprecated features now or forcing us to break ABI later when we delete one of these APIs.

Deprecated features should not be used. They are already eligible for deletion.

I prefer to just delete them now but they may be a hard sell for the Forum for functions that were only deprecated in 4.1. However, I'm still going to argue for removing them either from the ABI or altogether, because I am not willing to have an ABI break every time we decide to delete something we already deprecated.

There is precedent for this. Once features are deprecated, they are no longer maintained by the MPI Forum. For example, once the C++ bindings were deprecated, there were no C++ bindings defined for new APIs. This created a gross situation where one would have to use both the C and C++ headers and APIs to use the full MPI API. Anyone using the deprecated C++ header would have found that new features were missing, which is not that different from the situation here.

[eschnett]

It seems you want to keep the ABI stable even if the MPI standard makes incompatible changes. That will be difficult – the ABI chapter will then have to define the semantics of functions that are not mentioned elsewhere. It would be better if there are different ABIs for different MPI standards. The problem that the MPI standard evolves and deprecates/removes functions will not go away in the future.

I strongly suggest that an ABI that claims to support MPI 4.2 should do exactly that. If MPI 5.0 will be incompatible so be it.

My motivation is that this ABI isn't just useful for mpi4py or MPI.jl. This ABI will be used by many applications and libraries written in C/C++/Fortran/Rust/Haskell which want to interoperate with Python or Julia. Asking all of them to not use deprecated features is not practical.

[jeffhammond]

if we delete stuff, we'll rename 4.2 -> 5.0 and 5.0 -> ???. version names will be what they need to be.

[eschnett]

My goal is to have an MPI ABI that is easy to get started with. I want to be able to convince people easily that a standard ABI is a good thing for everyone, and not a scapegoat for forcing people to update their applications.

Several MPI implementations exist, with their own ABIs. The goal is to standardize these ABIs, not to force people to clean up their codes. A standard ABI could be introduced quite easily and without breaking anything.

At this point I am starting to consider proposing a separate "extended MPI ABI" that will support the full MPI standard, supporting existing applications out of the box.

[jeffhammond]

A standard ABI could be introduced quite easily and without breaking anything.

Sure, and the price to pay is that we can't delete stuff that has been deprecated without breaking ABI. Do you believe that your goals outweigh the MPI Forum's ability to delete deprecated features?

At this point I am starting to consider proposing a separate "extended MPI ABI" that will support the full MPI standard, supporting existing applications out of the box.

What exactly do you want to put in the "extended MPI ABI" besides eprecated features?

If you try to standardize Fortran module name-mangling, I fear you are in for a bad time, because you'll break the implementations in both MPICH and Open-MPI that will require them to implement Fortran twice, once for the ABI and once for backwards-compatibility with their own ABI.

[jprotze]

During the discussion someone mentioned that the deprecated info functions could not be replaced with macros. What is wrong with these definitions?

#define MPI_Info_get_valuelen(i, k, l, f) do{int len=0; MPI_Info_get_string(i, k, &len, NULL, f); l=len;} while(0)
#define MPI_Info_get(i, k, l, v, f) do{int len=l; MPI_Info_get_string(i, k, &len, v, f);} while(0)

With a simple test, I get the same result with and without the macro:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char **argv) {
  MPI_Init(&argc, &argv);

  char value[10] = "123456789";
  MPI_Info testinfo = MPI_INFO_NULL;
  MPI_Info_create(&testinfo);
  MPI_Info_set(testinfo, "key", "value");
  int vallen, flag;

  MPI_Info_get_valuelen(testinfo, "key", &vallen, &flag);
  printf("MPI_Info_get_valuelen(valuelen=%i, flag=%i)\n", vallen, flag);
  MPI_Info_get(testinfo, "key", vallen, value, &flag);
  printf("MPI_Info_get(value=\"%s\", flag=%i)\n", value, flag);

#define MPI_Info_get_valuelen(i, k, l, f)                                      \
  do {                                                                         \
    int len = 0;                                                               \
    MPI_Info_get_string(i, k, &len, NULL, f);                                  \
    *(l) = len - 1;                                                            \
  } while (0)
#define MPI_Info_get(i, k, l, v, f)                                            \
  do {                                                                         \
    int len = l + 1;                                                           \
    MPI_Info_get_string(i, k, &len, v, f);                                     \
  } while (0)
  MPI_Info_get_valuelen(testinfo, "key", &vallen, &flag);
  printf("MPI_Info_get_valuelen(valuelen=%i, flag=%i)\n", vallen, flag);
  MPI_Info_get(testinfo, "key", vallen, value, &flag);
  printf("MPI_Info_get(value=\"%s\", flag=%i)\n", value, flag);

  MPI_Finalize();
}

[jeffhammond]

What is wrong with these definitions?

Probably nothing but the question is whether these macro workarounds belong in the MPI standard.

[dalcinl]

For reference, take a look here. https://github.com/mpiwg-abi/header_and_stub_library/pull/32/files

I've added two files, one for deprecated stuff and another for removed stuff, just to make the point that everything from deprecated to removed can be easily supported at the ABI level with ease, without applications ever noticing at compile time. Before @jeffhammond hits hard at me for even considering the removed stuff, I would just mention that codes caring exclusively about MPICH may still be using these removed APIs.

The only possible hard point of failure I can envision would be the a try-link configure step that explicitly looks for the presence of a deprecated/removed symbol in the MPI library, which for the ABI library would be declared missing.

Regarding whether these "tricks" belong to the MPI standard, come on, guys... practicality beats purity. On the one side there is an intention of defining MPI in a language-neutral way. On the other side, we three damn different standard ways to use MPI from Fortran (mpif.h, use mpi, and use mpi_f08). Why a bunch of C #define's added for backward compatibility cannot belong to the MPI standard, but three different ways of doing Fortran can?

[jeffhammond]

I said it in the meeting and I will say it again here: if people try to force me to put deleted features in the ABI, I will quit the MPI Forum forever. We must be allowed to remove things that are bad. This is a fundamental aspect of standardization work.

#define workarounds are a hack to accommodate laziness, which deserve the same standard treatment as the compiler wrapper scripts. Implementations can provide them. The MPI Forum should not standardize them.