search

ecmwf / fckit

A Fortran toolkit for interoperating Fortran with C/C++
https://confluence.ecmwf.int/display/fckit
Apache License 2.0
29 stars 13 forks source link

f_comm%size and f_comm%rank always return 1 and 0 #13

Closed sebastienbarthelemy closed 3 years ago

sebastienbarthelemy commented 3 years ago

Hi,

I use fckit within a fortran code: bump-standalone. I noticed lately (while investigating another bug I have with bump) that the variables f_comm%size and f_comm%rank are always equal to 1 and 0 respectively while in this configuration f_comm%size should be equal to 6 and f_comm%rank should range from 0 to 5. I have no idea where the problem is coming from, can you provide some help ?

wdeconinck commented 3 years ago

Could it be that eckit was not compiled with MPI support? You can double check with executable fckit

fckit --info

CC @benjaminmenetrier

sebastienbarthelemy commented 3 years ago

"fckit --info" returns:

fckit version (0.7.0), git-sha1 8502aaeec5b3012ab337b8b3f696733037d4072b

Build: build type : Debug timestamp : 20200728163322 op. system : Linux-3.10.0-1062.18.1.el7.x86_64 (linux.64) processor : x86_64 sources : /cluster/projects/nn9039k/NorCPM/Code/code_bump/bump-standalone/fckit c++ compiler : GNU 9.3.0 flags : -pipe -O0 -g fortran compiler: GNU 9.3.0 flags : -O0 -g -fcheck=bounds -fbacktrace -finit-real=snan

Features: MPI : ON final : ON eckit : ON

Dependencies: eckit version (1.11.6), git-sha1 cb9a4d1cebbc9

wdeconinck commented 3 years ago

That looks OK to me. If you can create a branch with a failing unit-test within fckit, that would surely help to understand / diagnose the problem.

sebastienbarthelemy commented 3 years ago

Hi, the minimal failing test I can provide is the following, it's equivalent to the test "test_default_comm" of the file test_mpi.F90 file provided with fckit:

program debug
use fckit_module
use fckit_mpi_module, only: fckit_mpi_comm

type(fckit_mpi_comm) :: f_comm

call fckit_main%init()

f_comm = fckit_mpi_comm()
write(*,*) 'f_comm%size()', f_comm%size()
write(*,*) 'f_comm%rank', f_comm%rank()
end program debug

Both this code and the test "test_default_comm" provide the same results when running the command "mpirun -n 4 debug":

 f_comm%size()           1
 f_comm%rank           0
 f_comm%size()           1
 f_comm%rank           0
 f_comm%size()           1
 f_comm%rank           0
 f_comm%size()           1
 f_comm%rank           0
wdeconinck commented 3 years ago

That actually works OK for me. There are heuristics in eckit to determine if MPI is actually being used ( with mpirun, aprun, srun, ... ). If not it will take a "serial" eckit MPI implementation. See https://github.com/ecmwf/eckit/blob/develop/src/eckit/mpi/Comm.cc#L38-L54

As you can see, it is possible to force the "parallel" backend by setting

export ECKIT_MPI_FORCE=parallel

Try if that works. If so, we need to improve the eckit heuristics to better detect you are running with mpirun

sebastienbarthelemy commented 3 years ago

Hello. I am sorry, I have to say that the export of the variable ECKIT_MPI_FORCE did not bring any improvement and the outputs of the test remain the same.

wdeconinck commented 3 years ago

I have built the bump-standalone project, with the JCSDA release-stable branch of ecbuild. I have succesfully run ctest -R fckit_test_mpi -VV:

test 118
    Start 118: fckit_test_mpi

118: Test command: /usr/local/apps/openmpi/4.0.3/GNU/7.3.0/bin/mpiexec "-n" "4" "/tmp/nawd/src/bump/bump-standalone/build/fckit/src/tests/fckit_test_mpi"
118: Environment variables: 
118:  OMP_NUM_THREADS=1
118: Test timeout computed to be: 1500
118:  test_default_comm
118:  default size:           4
118:  default rank:           2
118:  test_comm
118:  default size:           4
118:  default rank:           2
118:  world size:           4
118:  default world:           2
118:  test_default_comm
118:  test_set_comm_default
118:  test_uninitialised
118:  test_allreduce
118:  default size:           4
118:  default rank:           0
118:  test_comm
118:  default size:           4
118:  default rank:           0
118:  test_default_comm
118:  world size:           4
118:  default world:           0
118:  default size:           4
118:  default rank:           3
118:  test_comm
118:  default size:           4
118:  default rank:           3
118:  world size:           4
118:  default world:           3
118:  test_set_comm_default
118:  test_set_comm_default
118:  test_uninitialised
118:  test_allreduce
118:  test_uninitialised
118:  test_allreduce
118:  test_default_comm
118:  default size:           4
118:  default rank:           1
118:  test_comm
118:  default size:           4
118:  default rank:           1
118:  world size:           4
118:  default world:           1
118:  test_set_comm_default
118:  test_uninitialised
118:  test_allreduce
118:  test_allreduce_inplace
118:  test_allreduce_inplace
118:  test_allreduce_inplace
118:  test_allreduce_inplace
118:  test_allgather
118:  test_allgather
118:  test_allgather
118:  test_allgather
118:  test_broadcast
118:  test_broadcast
118:  test_broadcast
118:  test_broadcast
118:  test_nonblocking_send_receive
118:  test_blocking_send_receive
118:  test_blocking_send_receive_rank1
118:  test_blocking_send_receive_int32_rank1
118:  test_nonblocking_send_receive
118:  test_nonblocking_send_receive
118:  test_nonblocking_send_receive
118:  test_blocking_send_receive
118:  test_blocking_send_receive_rank1
118:  test_blocking_send_receive_int32_rank1
118:  test_blocking_send_receive_int64_rank1
118:  test_blocking_send_receive_int64_rank1
118:  receive-request:           2
118:  test_blocking_send_receive
118:  send-request:           1
118:  test_blocking_send_receive
118:  test_blocking_send_receive_rank1
118:  test_blocking_send_receive_int32_rank1
118:  test_blocking_send_receive_rank1
118:  test_blocking_send_receive_int64_rank1
118:  test_blocking_send_receive_int32_rank1
118:  test_blocking_send_receive_int64_rank1
1/1 Test #118: fckit_test_mpi ...................   Passed    1.71 sec

My bet is that eckit is not picking up MPI somehow during configuration. You can help it detect MPI by setting the environment variable MPI_HOME to the root of the MPI installation. Can you check that libeckit_mpi.so actually links with MPI?

ldd lib/libeckit_mpi.so
sebastienbarthelemy commented 3 years ago

Hi Willem, thank you for taking the time to investigate that bug. Following the comment of Gilles Gouillardet on that page on stackoverflow I could fix the bug. When running a mpi application with the mpiexec command you have to make sure that the mpiexec is from the same libraries with those linked to your executable.

What happened in my case is that in order to be able to compile the whole package, saber, fckit... I had to load some modules. And the loading of these modules led to the loading of another module "OpenMPI/4.0.3-GCC-9.3.0" . So, in the end, libfckit.so was linked to the following mpi libraries:

libeckit_mpi.so => /cluster/projects/nn9039k/NorCPM/Code/code_bump/build/bump-standalone_debug/lib/libeckit_mpi.so (0x00002b2b10ea4000)
libmpicxx.so.12 => /cluster/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpicxx.so.12 (0x00002b2b124f6000)
libmpifort.so.12 => /cluster/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpifort.so.12 (0x00002b2b12716000)
libmpi.so.12 => /cluster/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/lib64/libmpi.so.12 (0x00002b2b12abf000)

while the command to run the test with bump was (file test/CTestTestfile.cmake):

"/cluster/software/OpenMPI/4.0.3-GCC-9.3.0/bin/mpiexec" "-n" "6" "/cluster/projects/nn9039k/NorCPM/Code/code_bump/build/bump-standalone_debug/bin/saber_bump.x" "testinput/bump_norcpm.yaml" "testoutput"

Now, I have corrected this command to avoid the conflict in libraries and it works fine. The correct command reads then:

"/cluster/software/impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28/bin64/mpiexec" "-n" "6" "/cluster/projects/nn9039k/NorCPM/Code/code_bump/build/bump-standalone_debug/bin/saber_bump.x" "testinput/bump_norcpm.yaml" "testoutput"