search

scivision / mumps

MUMPS via CMake
http://mumps-solver.org
MIT License
115 stars 46 forks source link

Observing Mumps Execution errors (Windows/IntelOneAPI Compiler/Ninja Build) #10

Closed vchunchu-amd closed 2 years ago

vchunchu-amd commented 3 years ago

Mumps Version (5.3.5)

When I try to run the executable for the sample c code (simple.c) and enabling only 'd' - double precision, I observe the following errors: Kindly help. Attached build log and simple.c used. build.txt simple.c.txt

Error Log: **// C:\Users\winuser2\vishy\repos\mumps\mumps_fixes\ninja_build_dir\tests>Csimple.exe No of Processes = 1, My Rank = 0 Start Mumps Init Done Mumps Init Start Mumps Analyse/Fact/Solve [PROC 0] Error Return in Test application, id.infog[0] = -16 [PROC 0] Error Return in Test application, id.infog[1] = 0 [PROC 0] id.infog[2] = 0

C:\Users\winuser2\vishy\repos\mumps\mumps_fixes\ninja_build_dir\tests>mpiexec -n 2 Csimple.exe No of Processes = 2, My Rank = 0 Start Mumps Init No of Processes = 2, My Rank = 1 Start Mumps Init ERROR RETURN FROM DMUMPS INFO(1)= -3 INFO(2)= -1 ERROR RETURN FROM DMUMPS INFO(1)= -3 INFO(2)= -1 [PROC 0] Error Return in Test application, id.infog[0] = -3 [PROC 0] Error Return in Test application, id.infog[1] = -1 [PROC 1] Error Return in Test application, id.infog[0] = -3 [PROC 1] Error Return in Test application, id.infog[1] = -1 [PROC 1] id.infog[2] = 0 [PROC 0] id.infog[2] = 0 //**

vchunchu-amd commented 3 years ago

The problem seems to be in enabling intsize64 option and linking ILP64 enabled numeric libraries. LP64 (intsize64=off) works fine provided I link LP64 numeric libraries.

Any specific code changes needed to work with ILP64(intsize64) option?

scivision commented 3 years ago

I don't know offhand, I haven't used the intsize64 myself.

scivision commented 3 years ago

I looked at this for v5.4.1.5. I don't know how to make oneAPI work with intsize64. GCC itself does work without MKL

vchunchu-amd commented 3 years ago

On further debugging, Using the MPI API MPI_Type_size( ) to get the size of mpi datatype MPI_INTEGER reveals some important findings.

  1. with Mumps intsize64=on and calling MPI_Type_size(MPI_INTEGER, sz, ierrsiz) in dmumps_driver.F prints size "4"

  2. A standalone fortran test code compiled using mpiifort and -i8 flag, prints the size as "8". attached test source and build log.

So, MPI_INTEGER seems to be of size 4 bytes even when "intsize64=on" in mumps which seems to cause the problem. Kindly check. build_test_fortransource.txt test.f90.txt

scivision commented 3 years ago

Maybe I've got the options wrong. If you do a fresh CMake build using Intel compiler starting with

 cmake -Dintsize64=on -DCMAKE_Fortran_FLAGS=-i8

do the tests work?

vchunchu-amd commented 3 years ago

I tried enabling "CMAKE_Fortran_FLAGS" to "-i8" as suggested. The tests still fail to execute. Also the compilers.cmake has FORTRAN_FLAG_INT64 set to "i8" flags for fortran compilation as part of mumps build system. The flag gets used for compilation which I made sure from build log.

I tried to reproduce this error in a standalone fortran test code using 2 scenarios,

  1. using a single line command to compile an executable directly mpiifort -o test.exe -i8 test.f90
  2. using a CmakeLists file to compile the test with necessary Fortran 64 bit "-i8" option and find_package(MPI). I made sure MPI libraries are picked (impi.lib, libmpi_ilp64.lib) and MPI_Fortran_Compiler points to mpiifort

I use a test of determining the size of MPI_INTEGER using an API MPI_Type_size( ) to check if 64 bit environment is enabled for Fortran/MPI setup. call MPI_Type_size(MPI_INTEGER, sz, ierrsiz) print *, 'sizeof(MPI_INTEGER) ', sz Scenario 1 prints correct size of 8 bytes (64 bit) Scenario 2 prints incorrect size of 4 bytes ( 32 bit) I do use "-i8" option in Cmake build system to enable 64 bit environment. But MPI still seems to be 32 bit. Attached these sample files for reference. CMakeLists.txt fortran_test_build.txt test.f90.txt test_cmakelists_example.f90.txt

vchunchu-amd commented 2 years ago

Adding the below line in tests/CMakelists.txt to include Intel MPI Libraries explicitly helped solve the ILP64 issue for me. A specific Intel's ILP64 MPI library needs to be linked first ("libmpi_ilp64.lib")

Kindly check ... target_link_libraries(Csimple PRIVATE ${IMPI_LIB_ILP64} ${MPI_C_LIBRARIES} MUMPS::MUMPS ${NUMERIC_LIBS}) ...

where the new variables are as follows:

  1. find_library(IMPI_LIB_ILP64 NAMES libmpi_ilp64 HINTS ${USER_PROVIDED_IMPILIB_ILP64_PATH})

       -----USER_PROVIDED_IMPILIB_ILP64_PATH="C:/Program Files (x86)/Intel/oneAPI/mpi/2021.3.0/lib"

  2. MPI_C_LIBRARIES is populated by "find_package(MPI COMPONENTS C Fortran REQUIRED)"

scivision commented 2 years ago

Thank you, I also observe that the IntelMPI "mpi_ilp64" library is the vital missing piece to being able to run int64 MUMPS