Open jeffhammond opened 8 years ago
Fujitsu MPI, which is based on Open MPI, also provides multiple compiler support unofficially (not general product).
The issue is not specific to Fortran (and removed C++). Even for C, OpenMP support is not ABI compatible. So Fujitsu MPI provides mpifcc
/mpiFCC
/mpifrt
for Fujitsu compiler, mpigcc
/mpig++
/mpigfortran
for GCC, and mpiicc
/mpicpc
/mpiifort
for Intel compiler.
The implementation is not same as the one which @jeffhammond suggested. But if someone need this, I can contribute it to Open MPI.
All MPI wrapper compilers are installed in the same directory (/opt/FJSVpclang/bin
). Users don't need to switch the PATH
environment variable but need to switch command names.
All shared libraries have distinct names (e.g. libmpi_gnu.so
) and installed in the same directory (/opt/FJSVpclang/lib64
). Include files and module files are installed separately with the same file names (e.g. /opt/FJSVpclang/include/mpi/gnu/mpi.h
). All are hidden by the wrapper compilers and users don't need to know the file names or directory paths.
mpiexec
and other files are shared. When user executes a MPI program using mpiexec
, appropriate libraries are linked automatically.
Developers need to run configure
and make all install
for each compiler product. On each configure
run, --enable-extra-named-lib=COMPILER_NAME
option must be supplied and CC
/CXX
/FC
must be set appropriately. The --prefix
option must be same.
Though this does not save sysadmin's time, but saves filesystem metadata.
@kawashima-fj It sounds like Fujitsu MPI does what users are doing already, meaning recompiling the full library for every compiler toolchain. Did I misunderstand you?
I was hoping for an implementation that would only compile the Fortran interface faces for every Fortran compiler, but reuse the C object codes for all Fortran compilers, for a single C compiler.
OpenMP might break C ABI compatibility when used to build object files but this should not affect the compilation of C source files in Open-MPI because it does not use OpenMP constructs, nor does it include omp.h
in any source file. Thus, one can build the vast majority of the Open-MPI object files with the system C compiler and end up with a C library that can be used with any compiler.
@jeffhammond You understood correctly. The mechanism I described is not a solution for you. It's slightly off topic. The merit of the mechanism is mainly for end users, not sysadmins.
My OpenMP description was not related to this topic, sorry. Please ignore it.
@jeffhammond i think we should decouple end user experience and sysadmin experience.
sysadmins can script openmpi build/install, and run several builds in parallel. nowadays SSD can store multi TB of data, so as far as i am concerned, i'd rather focus on enduser experience.
an other nice feature to have is the use of a single fortran wrapper that picks the "right" modules and libs.
for example
mpifort foo.f90
picks the "standard" modules/libs
whereas the infamous
mpifort -i8 -d8 foo.f90
would rather pick the modules/libs compiled with 8 bytes integer and 8 bytes real
also, an option could switch between the libs that support MPI_THREAD__MULTIPLE
and the libs that do not.
iirc, IntelMPI wrappers do a great job at that
about saving disk space, i am afraid you assume all fortran compilers work just fine with the GNU C compiler. though that is very likely true for the Intel Fortran compilers, PGI Fortran compilers do not work out of the box with GNU C compiler (and you can consider this is an Open MPI bug) i am also skeptical about Intel C and PGI Fortran (and no, i do not have any plan to do this in the real life)
last but not least, an other nice to have feature is the ability to generate wrappers for both the native compiler and the (several ?) cross compilers
@ggouaillardet It is a user experience issue if the sysadmin users experience is such that the latter does not build every case. Some sysadmins believe in validating libraries before exposing them to users. Because we have both observed that C correctness can depend on Fortran, at least when PGI is involved (e.g. http://trac.mpich.org/projects/mpich/ticket/1869), sysadmins need to validate the entire build, so building the entire library for N Fortran compilers forces the sysadmin to run N times as many tests. At least in the case where the C library correctness is not dependent on the Fortran compiler, my proposal reduces sysadmin effort.
You are right that switching between 32b and 64b INTEGER
is a great feature for Fortran users, even though that feature of Fortran is itself horrible beyond imagination.
I don't see any reason for the wrapper to need to know about MPI thread support. That should be selectable at runtime the same as choosing a BTL/MTL.
It is great that all of your filesystems are infinitely large. I use both laptops and supercomputers where this is not the case.
if you might need MPI_THREAD_MULTIPLE
, then you have to configure with --enable-mpi-thread-multiple
, otherwise Open MPI is configured with default --disable-mpi-thread-multiple
.
the latter library is a bit faster because some thread related stuff is #ifdef'ed out.
/* and there is some work to make the former library nearly as fast as the latter library */
iirc, Intel MPI does the same thing under the hood (well, if you use Open MP, then the MPI_THREAD_MULTIPLE
library is selected, even if your app does not require MPI_THREAD_MULTIPLE
)
we previously had a discussion of how Open MPI (or any MPI implementaion) should behave when configure'd with --disable-mpi-fortran
with respect to Fortran predefined datatypes.
iirc, we kind of agreed that was a (dark) grey area since some assumptions had to be made
(e.g. do we simply assume MPI_INT
is equivalent to MPI_INTEGER
?,
and as we all know, bad things happen when we start assuming things).
having said that, i do not think it is realistic to build the C libs with --disable-mpi-fortran
and then build one or several Fortran lib(s) on top of that.
my point is if we build with one C compiler and several Fortran compilers, then we would have to check all Fortran compilers have the same "characteristics" from the Open MPI C library point of view.
an obvious mistake would be to run gfortran
as is and ifort -i8 -d8
, but there might be less obvious source of errors in the real life.
below are two additional though weaker reasons why i'd rather have multiple Open MPI install dir
the install dir of the latest master, built with -g
and without --disable-dlopen
(needless to say this is far from optimal from a disk usage point of view) takes 48 MB, which is several orders of magnitude lower than infinitely large or even a laptop SSD.
if i assume a sysadmin has already scripted the installation and validation process of Open MPI, then running several instances (one for each C&Fortran compiler) in parallel might require a shorter elapsed time than running one instance with all the Fortran compilers.
i made a proof-of-concept at open-mpi/ompi#2070
here is how it can be used
./configure --with-mpi-fortran-dir=/intel --disable-oshmem CC=gcc CXX=g++ FC=ifort
make
make install
mv /usr/local/share/openmpi/mpifort-wrapper-data.txt /usr/local/share/openmpi/mpiifort-wrapper-data.txt
ln -s opal_wrapper /usr/local/bin/mpiifort
./configure --with-mpi-fortran-dir=/intel --disable-oshmem CC=gcc CXX=g++ FC=gfortran
make
make install
and then you can use mpifort
if you want to use the GNU compiler, and mpiifort
if you want to use the Intel compiler
note
--disable-oshmem
because there is a single oshmem lib (and not two for C and Fortran)`configure
and make
can run in parallel if you use VPATH
with two directories, justmake install
mv /usr/local/share/openmpi/mpifort-wrapper-data.txt /usr/local/share/openmpi/mpiifort-wrapper-data.txt
ln -s opal_wrapper /usr/local/bin/mpiifort
the Intel built library before you
make install
the GNU build library
-i8 -d8
like optionUnder the hood, when configured with --with-mpi-fortran-dir=/intel
Fortran headers will be installed in /usr/local/include/intel
and libs and modules will be installed in /usr/local/lib/intel
@jsquyres @gpaulsen would you mind raising this issue during today's telcon ? Ideally, I'd like some more folks think about it and provide some directions on how to move forward.
As a note, IBM's Spectrum MPI provides GNU, PGI, and XL Fortran modules that work with the same core MPI library:
Summary
Please support the ability to build Open-MPI Fortran bindings for multiple Fortran compilers in the same build. This will dramatically improve user experience.
I recognize this is not a small thing to implement, but at least two vendors do this already, and since some vendors use Open-MPI as the basis for their MPI product, it might be the sort of thing that makes sense for the project to support.
Alternative / Workaround
The current solution for multiple Fortran compilers is to build the MPI library once for every toolchain, e.g. one has installation directories
open-mpi-intel15
,open-mpi-intel16
,open-mpi-gcc5
,open-mpi-gcc6
, etc. This naturally leads tompi.mod
andmpi_f08.mod
for every Fortran compiler.While simple, the current solution is a waste of sysadmin time, processor cycles, filesystem metadata, etc. given that one could instead build Open-MPI once with the preferred C compiler once and then support O(n) Fortran compiler wrappers and modules.
Future Workaround
In theory, there could be an implementation-agnostic implementation of the MPI Fortran 2008 bindings that could be compiled independent of the MPI C library, but such a project does not yet exist and in any case, very few users have adopted
use mpi_f08
.Background
The Fortran bindings are largely if not completely independent of the internals of MPI, which are compiled with a C compiler, and C allows for a standard ABI. On the other hand, Fortran does not have a standard ABI and modules must be generated for every compiler vendor, and possibly also multiple (major) versions of the compiler from a single vendor.
Suggested implementation
FCn
for n=1,2,... as an alternative toFC
. The build system would then generate MPI Fortran scripts namedmpi$FC1
,mpi$FC2
, etc., assumingFCn
do not have the full paths, otherwise they will be stripped.-enable-fortran-compilers=$FC1,$FC2
. This is just different syntax for option 1.Ideally, the user can do something like
$PATH_TO_OMPI/configure FC1=ifort FC2=gfortran
and get$OMPI_PREFIX/bin/mpiifort
,$OMPI_PREFIX/bin/mpigfortran
,$OMPI_PREFIX/include/ifort/mpi.mod
,$OMPI_PREFIX/include/gfortran/mpi.mod
,$OMPI_PREFIX/include/ifort/mpi_f08.mod
,$OMPI_PREFIX/include/gfortran/mpi_f08.mod
, and any companion files.Prior Engineering / Proof-of-Principle
Intel MPI
Intel MPI packages the wrappers and modules for Intel Fortran and GNU Fortran.
IBM Blue Gene/Q
IBM supported this on Blue Gene/Q. I do not know the implementation details but can demonstrate the results if anyone needs to see it.