Binding to the C instead of Fortran interface

[poulson]

Given that Julia is heavily tied to Clang and supports ccall, and there is no Clang Fortran compiler, it would clearly be beneficial to switch to binding MPI.jl to MPI's C interface instead of its Fortran interface. The main hurdle is that different MPI implementations define the MPI types in different manners, though one can achieve a high degree of portability with that assumption that {MPI_Comm,MPI_Group,MPI_Datatype,etc.} are either int or void*, so that one can simply perform checks such as sizeof(MPI_Comm) == sizeof(int) to decide on the appropriate datatype.

I would be happy to help with this port if it would be of interest; I'm running into substantial problems with MPI.jl on Yosemite.

[poulson]

Also, it might be a better idea to rely on regular expressions on mpi.h to determine values of the various constants than to rely on execute_process, as the latter would not make sense in a cross-compilation environment.

[lcw]

Hi Jack,

As you point out, it would be great to remove the dependence of MPI.jl on Fortran. Even better it would be great if we could remove any compilation needed at install time and move abi determination to run time.

This makes using different mpi implementations on large clusters just a module load away. I have some time on stampede (and possibly titan) that I can use to test any patches. I think if we can get something portable across these systems that would go a long way for portability.

Any patches you have towards this goal would be welcome.

[ViralBShah]

This does sound like overall the right direction.

[JaredCrean2]

Moving to the C interface would also be helpful for the Petsc wrappers I am working on. They work with mpich but not with openmpi because mpich uses an int for the MPI communicator while openmpi uses a void * in C but a Fortran integer, which have different sizes. Moving to the C API sounds like it would be pretty straightforward because translating make_f_const.c into C is quite doable. Going the ABI route might be more complicated, especially because the MPI standard does not define the type of MPI_Comm and the shared object file might not contain the information (I know the Petsc .so does not contain data type information).

[JaredCrean2]

Another option is to just convert the Fortran communicator to a C communicator (assuming you know what datatype it is): http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node361.htm . Having both would be useful for different use cases.

cc: @stevengj

[stevengj]

@poulson, it is actually possible to detect the sizeof a type reliably in a cross-compiling environment using only the compiler. This is one way to do it using switch, though the current version of autoconf instead does a binary search on test programs using a static-array declaration trick.

[jakebolewski]

Do we have to be this fancy? Most MPICH derived MPI implementations have committed to using a common ABI. Wouldn't that mean you would just have to special case OpenMPI? Of course this wouldn't work on legacy systems.

[stevengj]

I think it's reasonable to only support relatively modern MPI implementations, in which case we can just tabulate the sizes. Is there a reasonable way to detect what MPI implementation is being used at runtime? Does the string returned by MPI_Get_library_version suffice?

[jakebolewski]

MPI4Py compiles a small test file. The Cmake findmpi configuration does the same thing. I think this is the most robust way to determine the MPI configuration.

[eschnett]

If you compile C MPI programs, then you may as well output sizeof() and offsetof() all interesting data structures and their fields.

[stevengj]

One issue is that constants like MPI_COMM_SELF may be harder to define using the C interface; e.g. in OpenMPI they are pointers, and we may need to define them after MPI_Initialize has been called, which will be awkward.

[tbole]

I don't know, if there's still much effort going into the development of the MPI interface, since this thread is already pretty old. I think it's probably less important, whether the Fortran or C interface is used. It's mainly relevant for the stuff "under the hood". In most cases people will interact with the wrapper and eventually pass communicators, info handles, etc to external libs. If those are written in Fortran, the Fortran interface will be more convenient, in case of C libs the one in C. Extending the idea of @JaredCrean2 I think it may be a good idea to just have both at the same time. Instead of having structs like

mutable struct Comm
    val::Cint
end

one could simply by default create C types at the same time

mutable struct Comm
    fval::Cint
    cval::CComm
end

It's actually not much to do, it's only for the interfaces provided here (comm,op,info,..) If other libs return C stuff, we could just extend the constructors to accept both C and Fortran.

I think this could be done with relatively limited overhead. As a further advantage, if one decides to shift the MPI.jl over to the C interface, there will be no outward facing changes, since everyone will have their preferred (even mixed) interface directly available.

[stevengj]

The MPI.jl package is definitely still being maintained. I don't think there has been any effort yet to change to the C API, however, because it is a lot of work for not much (if any?) benefit.

Compatibility with C libraries that need a C MPI_Comm was fixed by #89.

[lcw]

For anyone thinking about changing to the C API, here [1, 2] are a couple of nice blog posts by @jsquyres about some of the issues with using MPI outside of C and Fortran.

[andreasnoack]

I don't think there has been any effort yet to change to the C API

I prepared https://github.com/JuliaParallel/MPI.jl/pull/169 a while ago but I bundled it with a change to hardcode the header definitions instead of relying on CMake. However, I'm not sure if that is the best solution anymore.

[jsquyres]

Wow, it looks like some of the HTML formatting in those blog entires has bit rotted over time -- apologies for that. But the content is all still correct. 😄

FWIW, your future selves will thank you if you don't try to do tricks to figure out what the underlying types / sizes of MPI_Comm (etc.) are, and try to map to underlying implementation details. Just write C code and use MPI_Comm -- whatever it is. That is how MPI is intended to be used.

A humble suggestion: wrap MPI handles in some Julia-known container format that is both versatile enough to handle whatever the underlying type of various MPI handles are, and also can then be mapped precisely into Julia. This will both isolate you from the differences between the Open MPI and MPICH families of MPI implementations, and also isolate you from changes to underlying MPI implementations over time.

[lcw]

An extreme example of what I think @jsquyres is talking about is MorphMPI, which allows for switching out an MPI implementation of a C based application without recompiling the whole application.

[tbole]

I wasn't really suggesting to change the interfaces. @jsquyres I think this is, what MPI.jl actually tries to achieve using the Fortran interface. Given the quirks with OpenMPI and MPICH, it is a quite good alternative.

I see two problems here: the first which can be simply solved, is that most other libs rely on the C version, which we can wrap. The other problem, is that MPI is often only an optional dependency for many people/libs and Julia is rather bad at handling optional package dependencies. I have for example just extended HDF5.jl to mpio (pull request incoming), since it's one of the most common parallel IO solutions on clusters. Coming from the HPC world it was rather surprising, that there was no support for that yet. Since I didn't want to introduce an additional dependency on MPI.jl, I have done it "the wrong and ugly way" and created two dummy handles (32 bit, 64 bit) for the MPICH and the OpenMPI and dispatched to the MPI specific functions on those.

Bad thing is that people actually need to know about the interfaces, that Julia uses the Fortran one (which only affects OpenMPI), and that they have to get the C handles. I was merely suggesting to include both C and Fortran handles by default, which could be created on the fly.

[JaredCrean2]

don't try to do tricks to figure out what the underlying types / sizes of MPI_Comm (etc.) are

The situation appears similar to what the Rust commenter described below the second article. Julia doesn't have access to the C preprocessor, but calling C requires specifying the sizes of all the arguments. Having to do some kind of trick to get the information is undesirable, but I'm not sure there is an alternative.

mpio (pull request incoming)

Awesome.

[jsquyres]

Forgive me, I know very little (i.e., nothing) about Julia, so my suggestions may be meaningless.

Given that Julia doesn't have access to the C pre-processor, it may well be that a Morph-MPI-like idea (cited above) could be a good one here. I.e., have a fairly thin middle layer (in C) that provides a constant-sized MPI handle to Julia (e.g., 64 bits), and then Julia can use that. The middle C layer can translate between Open MPI / MPICH as relevant. Then the size of the underlying MPI implementation's handles don't matter, etc.

That being said, if you have already have everything working with the Fortran bindings, then so be it. It isn't optimal, but it sounds like it isn't broken, either. Chopping out the Fortran bindings would:

• Eeek back a tiny little bit of performance (i.e., remove 1 or 2 extra fairly-thin function calls when calling MPI functions)
• Avoid some Fortran compiler / linker issues
• Avoid a Fortran dependency
• Potentially make some of the callback semantics a bit easier (this is an iffy benefit -- you may or may not care much about MPI's callbacks)

These benefits may or may not be worth it.

Just curious -- which MPI Fortran bindings are you using: mpif.h, use mpi, or use mpi_f08?

[stevengj]

@jsquyres, the MPI_Comm issue was already solved in #89. It indeed uses a C program when MPI.jl is built to correctly define our MPI.CComm type, and uses the documented MPI_Comm_f2c/c2f API to convert to/from Fortran communicators as needed.

I agree that we ultimately want to look at the header file when determining how to call the C API, even if we build in pre-detected interfaces for a couple of popular MPI implementations. I don't think we want or need to write our own C glue layer, just a build script that uses a C program to detect the types and constants in mpi.h.

This isn't urgent because we so far haven't encountered any linking problems (I think?) with using the Fortran API (we only use the Fortran compiler at build time, not at runtime), the performance difference is almost certainly negligible (I am willing to be persuaded otherwise, but I challenge you to measure a difference in any nontrivial call, e.g. MPI_Send, between the Fortran and C APIs … you can time them directly in Julia via ccall), and it's not clear that we have any use for MPI's callbacks (and even if we did we could call the C API for that portion of MPI.jl, since we can convert the communicators as noted above).

[simonbyrne]

An alternative to MorphMPI is https://github.com/cea-hpc/wi4mpi, but it uses a funny (non-OSI-approved) license.