VictorEijkhout commented 4 years ago

This page:

https://trilinos.org/packages/amesos/amesos-installation/amesos-mumps-install/

doesn't seem to be current, considering the exhortation at te top.

Also: it doesn't mention the cmake options I should use to include Mumps in the installation.

Is there current documentation? Feel free to just dictate me the relevant option(s).

mhoemmen commented 4 years ago

@trilinos/amesos @trilinos/amesos2

srajama1 commented 4 years ago

@searhein You are the only MUMPS user I know from Amesos2 as of now. Can you help ?

VictorEijkhout commented 4 years ago

Immediate question: your old instructions talk about downloading Mumps 4.6. I have 5.1. Is that compatible?

Next question: what are the cmake options? (Is there a place where all your cmake options are documented?)

Victor.

On Nov 25, 2019, at 12:33 AM, Siva Rajamanickam notifications@github.com<mailto:notifications@github.com> wrote:

@searheinhttps://github.com/searhein You are the only MUMPS user I know from Amesos2 as of now. Can you help ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/trilinos/Trilinos/issues/6339?email_source=notifications&email_token=AAFHBAQLNFWCBCWMCMQFDDDQVNWTZA5CNFSM4JRAWZB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFBJOKY#issuecomment-558012203, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFHBATMQJLVDT4IZ4UN34DQVNWTZANCNFSM4JRAWZBQ.

This message is from an external sender. Learn more about why this matters.https://ut.service-now.com/sp?id=kb_article&number=KB0011401

searhein commented 4 years ago

Actually, I am still using MUMPS 4.7, which works fine for me. @chochmuth, do you use a more recent version of MUMPS?

mhoemmen commented 4 years ago

@VictorEijkhout you're talking about Amesos, not Amesos2, right?

VictorEijkhout commented 4 years ago

On Nov 25, 2019, at 10:06 AM, Mark Hoemmen notifications@github.com<mailto:notifications@github.com> wrote:

you're talking about Amesos, not Amesos2, right?

I think so. User requested the following:

%%%%%%% I work with and develop a code that uses the trilinos library, including the MUMPS dependency. I've loaded the module trilinos/12.10.1 on stampede2. My code needs to include Amesos_Mumps.h, and I do not find this available, %%%%%%%

Victor.

chochmuth commented 4 years ago

I am currently using MUMPS 5.1.1. Lately, I have only used Tpetra and therefore the Amesos2 interface. I might add, that I made some modifications to the Amesos2_Mumps interface, since the current version does not support the use of all MUMPS parameters. I want to commit these changes with PR in the near future.

The Amesos_Mumps interface should be available with the following cmake defines:

-D TPL_ENABLE_MUMPS:BOOL=ON \
-D MUMPS_LIBRARY_DIRS:PATH=$MUMPS/lib \
-D MUMPS_INCLUDE_DIRS:PATH=$MUMPS/include \
-D MUMPS_LIBRARY_NAMES:STRING="dmumps;mumps_common;pord" \

with MUMPS=/your/mumps/dir.

Furthermore, the name of the MUMPS integer type has changed with version 4.9. Therefore, the Amesos_Mumps interface uses the following distinction:

#ifdef MUMPS_4_9
    MDS.comm_fortran = (MUMPS_INT) MPI_Comm_c2f( MUMPSComm_);
#else

#ifndef HAVE_AMESOS_OLD_MUMPS
    MDS.comm_fortran = (DMUMPS_INT) MPI_Comm_c2f( MUMPSComm_);
#else
    MDS.comm_fortran = (F_INT) MPI_Comm_c2f( MUMPSComm_);
#endif
#endif

I am not sure wether the macro#define MUMPS_4_9 is set per default now. The README-MUMPS says:

September 23, 2015 CMake has been updated to now use mumps 5.0.0+. This is the defualt.
The MACRO for -DMUMPS_4_9 or -DMUMPS_5_0 is needed. This is be made default in future.

Therefore, you might want to add the following line to your configure script: -D CMAKE_CXX_FLAGS:STRING="-DMUMPS_4_9" \

VictorEijkhout commented 4 years ago

Thanks, that’s very clear.

Do I need an option

-D TPL_ENABLE_MUMPS:BOOL=ON

to activate the mumps interface?

Victor.

On Nov 25, 2019, at 1:49 PM, Christian Hochmuth notifications@github.com<mailto:notifications@github.com> wrote:

I am currently using MUMPS 5.1.1. Lately, I have only used Tpetra and therefore the Amesos2 interface. I might add, that I made some modifications to the Amesos2_Mumps interface, since the current version does not support the use of all MUMPS parameters. I want to commit these changes with PR in the near future.

The Amesos_Mumps interface should be available with the following cmake defines:

-D MUMPS_LIBRARY_DIRS:PATH=$MUMPS/lib \ -D MUMPS_INCLUDE_DIRS:PATH=$MUMPS/include \ -D MUMPS_LIBRARY_NAMES:STRING="dmumps;mumps_common;pord" \

with MUMPS=/your/mumps/dir.

Furthermore, the name of the MUMPS integer type has changed with version 4.9. Therefore, the Amesos_Mumps interface uses the following distinction:

ifdef MUMPS_4_9

MDS.comm_fortran = (MUMPS_INT) MPI_Comm_c2f( MUMPSComm_);

else

ifndef HAVE_AMESOS_OLD_MUMPS

MDS.comm_fortran = (DMUMPS_INT) MPI_Comm_c2f( MUMPSComm_);

else

MDS.comm_fortran = (F_INT) MPI_Comm_c2f( MUMPSComm_);

endif

I am not sure wether the macro#define MUMPS_4_9 is set per default now. The README-MUMPS says:

September 23, 2015 CMake has been updated to now use mumps 5.0.0+. This is the defualt. The MACRO for -DMUMPS_4_9 or -DMUMPS_5_0 is needed. This is be made default in future.

Therefore, you might want to add the following line to your configure script: -D CMAKE_CXX_FLAGS:STRING="-DMUMPS_4_9" \

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/trilinos/Trilinos/issues/6339?email_source=notifications&email_token=AAFHBASVI4TAFQ32EZ2EGDTQVQT5VA5CNFSM4JRAWZB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFDS3TQ#issuecomment-558312910, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFHBAW7NHCFKMO2NR2L7X3QVQT5VANCNFSM4JRAWZBQ.

This message is from an external sender. Learn more about why this matters.https://ut.service-now.com/sp?id=kb_article&number=KB0011401

srajama1 commented 4 years ago

@chochmuth It would be very nice if you can submit a PR for these MUMPS changes. @searhein can show you the process to do this.

chochmuth commented 4 years ago

@VictorEijkhout yes, you are correct. You need to enable MUMPS. I edited my above comment. Somehow the line -D TPL_ENABLE_MUMPS:BOOL=ON \ was not displayed.

VictorEijkhout commented 4 years ago

I’m getting a bunch of undefined references.

../../../../amesos/src/libamesos.so.12.18.1: undefined reference to descinit_' ../../../../amesos/src/libamesos.so.12.18.1: undefined reference toSCOTCHdgraphInit' ../../../../amesos/src/libamesos.so.12.18.1: undefined reference to `scotchfdgraphordergather' ../../../../amesos/src/libamesos.so.12.18.1: undefined reference to scotchfdgraphcorderexit_' ../../../../amesos/src/libamesos.so.12.18.1: undefined reference tometisnodend' ../../../../amesos/src/libamesos.so.12.18.1: undefined reference to pdpotrs_' ../../../../amesos/src/libamesos.so.12.18.1: undefined reference toscotchfdgraphorderinit_' ../../../../amesos/src/libamesos.so.12.18.1: undefined reference to `esmumps' make[2]: *** [packages/intrepid/test/Discretization/FunctionSpaceTools/Intrepid_test_Discretization_FunctionSpaceTools_Test_06.exe] Error 1

I thought I had supplied the Scalapack library in the LAPACK_LIBRARY_DIRS/NAMES. Does the cmake test for a functioning scalapack?
Does it need Scotch? I may have that somewhere.
ParMetis & ordinary Metis?

Victor.

On Nov 25, 2019, at 3:05 PM, Christian Hochmuth notifications@github.com<mailto:notifications@github.com> wrote:

@VictorEijkhouthttps://github.com/VictorEijkhout yes, you are correct. You need the enable MUMPS. I edited my above comment. Somehow the line -D TPL_ENABLE_MUMPS:BOOL=ON \ was not displayed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/trilinos/Trilinos/issues/6339?email_source=notifications&email_token=AAFHBAVTNEVJ37IEZSUDMT3QVQ42VA5CNFSM4JRAWZB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFDZMWI#issuecomment-558339673, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFHBAXPSTGPWXQK5H5HKA3QVQ42VANCNFSM4JRAWZBQ.

This message is from an external sender. Learn more about why this matters.https://ut.service-now.com/sp?id=kb_article&number=KB0011401

VictorEijkhout commented 4 years ago

PS

cmake -VV -D BUILD_SHARED_LIBS:BOOL=ON -D Trilinos_VERBOSE_CONFIGURE=OFF -D CMAKE_VERBOSE_MAKEFILE=ON -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -D Trilinos_ASSERT_MISSING_PACKAGES=OFF -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -D Trilinos_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_EXAMPLES:BOOL=ON -D Trilinos_ENABLE_Export_Makefiles:BOOL=ON -D Trilinos_ENABLE_Fortran:BOOL=ON -D CMAKE_INSTALL_PREFIX:PATH=/home1/apps/intel18/impi18_0/trilinos/12.18.1 -D CMAKE_BUILD_TYPE:STRING=RELEASE -D 'CMAKE_C_FLAGS:STRING=-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl' -D 'CMAKE_CXX_FLAGS:STRING=-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl -DMPICH_SKIP_MPICXX' -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON -D Tpetra_INST_INT_LONG_LONG:BOOL=ON -D Tpetra_INST_INT_INT:BOOL=ON -D Tpetra_INST_DOUBLE:BOOL=ON -D Tpetra_INST_FLOAT:BOOL=OFF -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF -D Tpetra_INST_INT_LONG:BOOL=OFF -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF -D BLAS_INCLUDE_DIRS:PATH=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/include -D BLAS_LIBRARY_DIRS:PATH=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -D 'BLAS_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64' -D LAPACK_INCLUDE_DIRS:PATH=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/include -D LAPACK_LIBRARY_DIRS:PATH=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -D 'LAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64' -D TPL_ENABLE_MPI:BOOL=ON -D MPI_EXEC:FILEPATH=/opt/apps/xalt/0.6/bin/ibrun -D TPL_MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include/.. -D TPL_MPI_INCLUDE_DIRS=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include -D TPL_ENABLE_GLM=OFF -D TPL_ENABLE_Matio=OFF -D TPL_ENABLE_MUMPS:BOOL=ON -D MUMPS_INCLUDE_DIRS=/home1/apps/intel18/impi18_0/petsc/3.12/skylake/include -D MUMPS_LIBRARY_DIRS:PATH=/home1/apps/intel18/impi18_0/petsc/3.12/skylake/lib -D 'MUMPS_LIBRARY_NAMES:STRING=libdmumps.a;libmumps_common.a;libpord.a' -D 'TPL_ParMETIS_LIBRARIES=/home1/apps/intel18/impi18_0/petsc/3.12/skylake/lib/libptscotchparmetis.a;/home1/apps/intel18/impi18_0/petsc/3.12/skylake/lib/libparmetis.so' -D TPL_ParMETIS_INCLUDE_DIRS=/home1/apps/intel18/impi18_0/petsc/3.12/skylake/include -D TPL_ENABLE_Boost:BOOL=ON -D Boost_INCLUDE_DIRS:PATH=/opt/apps/intel18/boost/1.68/include -D Boost_LIBRARY_DIRS:PATH=/opt/apps/intel18/boost/1.68/lib -D TPL_ENABLE_BoostLib:BOOL=ON -D BoostLib_INCLUDE_DIRS:PATH=/opt/apps/intel18/boost/1.68/include -D BoostLib_LIBRARY_DIRS:PATH=/opt/apps/intel18/boost/1.68/lib -D TPL_ENABLE_HDF5:BOOL=ON -D HDF5_INCLUDE_DIRS:PATH=/opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64/include -D HDF5_LIBRARY_DIRS:PATH=/opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64/lib -D TPL_ENABLE_Netcdf:BOOL=ON -D Netcdf_INCLUDE_DIRS:PATH=/opt/apps/intel18/impi18_0/parallel-netcdf/4.6.2/x86_64/include -D Netcdf_LIBRARY_DIRS:PATH=/opt/apps/intel18/impi18_0/parallel-netcdf/4.6.2/x86_64/lib -D VLE_SUPERLU_IS_FIXED_AFTER_12.12:BOOL=ON -D TPL_ENABLE_SuperLU:BOOL=OFF -D SuperLU_INCLUDE_DIRS:PATH=/home/foo/include -D SuperLU_LIBRARY_DIRS:PATH=/home/foo -D SuperLU_LIBRARY_NAMES:STRING=superlu -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON -D Trilinos_ENABLE_Amesos:BOOL=ON -D Trilinos_ENABLE_Amesos2:BOOL=ON -D Amesos2_ENABLE_KLU2:BOOL=ON -D Amesos2_ENABLE_Basker:BOOL=ON -D Trilinos_ENABLE_Anasazi:BOOL=ON -D Trilinos_ENABLE_AztecOO:Bool=ON -D Trilinos_ENABLE_Belos:BOOL=ON -D Trilinos_ENABLE_Epetra:Bool=ON -D Trilinos_ENABLE_EpetraExt:Bool=ON -D Epetra_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_FEI:Bool=ON -D Trilinos_ENABLE_Ifpack:Bool=ON -D Trilinos_ENABLE_Ifpack2:BOOL=ON -D Trilinos_ENABLE_Intrepid:BOOL=ON -D Trilinos_ENABLE_Intrepid2:BOOL=ON -D Intrepid_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_Issoropia:BOOL=ON -D Trilinos_ENABLE_ML:BOOL=ON -D ML_TAKES_SUPERLU_LESS_THAN_5=TRUE -D ML_ENABLE_SuperLU:BOOL=OFF -D Trilinos_ENABLE_MOOCHO:BOOL=ON -D Trilinos_ENABLE_MueLu:BOOL=ON -D MueLu_ENABLE_Tutorial:BOOL=OFF -D MueLu_ENABLE_EXAMPLES:BOOL=OFF -D Trilinos_ENABLE_NOX=ON -D NOX_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Pamgen:Bool=ON -D Pamgen_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Panzer:Bool=ON -D Trilinos_ENABLE_Phalanx:BOOL=ON -D Phalanx_EXPLICIT_TEMPLATE_INSTANTIATION=ON -D Phalanx_ENABLE_EXAMPLES=OFF -D Trilinos_ENABLE_Piro:BOOL=ON -D Trilinos_ENABLE_Rythmos:BOOL=ON -D Trilinos_ENABLE_Sacado:Bool=ON -D Trilinos_ENABLE_SEACAS:BOOL=ON -D Trilinos_ENABLE_SEACASIoss:BOOL=ON -D Trilinos_ENABLE_SEACASBlot:BOOL=ON -D Trilinos_ENABLE_SEACASExodus:BOOL=ON -D Trilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON -D Trilinos_ENABLE_Shards:BOOL=ON -D Trilinos_ENABLE_ShyLU:BOOL=ON -D Trilinos_ENABLE_Stokhos:BOOL=ON -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_Teko:BOOL=ON -D Trilinos_ENABLE_Teuchos:BOOL=ON -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON -D Trilinos_ENABLE_Thyra:BOOL=ON -D Trilinos_ENABLE_Tpetra:BOOL=ON -D Trilinos_ENABLE_TriKota:BOOL=ON -D Trilinos_ENABLE_Zoltan:BOOL=ON -D Trilinos_ENABLE_Zoltan2:BOOL=ON -D Trilinos_ENABLE_Kokkos:BOOL=ON -D Trilinos_ENABLE_KokkosCore:BOOL=ON -D Phalanx_KOKKOS_DEVICE_TYPE:STRING=SERIAL -D Phalanx_INDEX_SIZE_TYPE:STRING=INT -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF -D Kokkos_ENABLE_Serial:BOOL=ON -D Kokkos_ENABLE_OpenMP:BOOL=OFF -D Kokkos_ENABLE_Pthread:BOOL=OFF -D Trilinos_ENABLE_STK:BOOL=ON -D Trilinos_ENABLE_STKIO:BOOL=ON -D Trilinos_ENABLE_STKMesh:BOOL=ON -D Trilinos_ENABLE_PyTrilinos:Bool=OFF -D CMAKE_PYTHON_INCLUDE_DIR:PATH=NOPYTHON -D CMAKE_PYTHON_LIBRARIES:STRING=NOPYTHON -D 'CMAKE_PYTHON_EXECUTABLE:FILEPATH=/python%{python_major_version}' -D PyTrilinos_DOCSTRINGS:BOOL=OFF -D PyTrilinos_ENABLE_Tpetra:BOOL=OFF -D SWIG_EXECUTABLE:FILEPATH=/home1/apps/intel18/swig/3.0.12/bin/swig /admin/build/admin/rpms/stampede2/BUILD//trilinos-12.18.1

On Nov 25, 2019, at 3:05 PM, Christian Hochmuth notifications@github.com<mailto:notifications@github.com> wrote:

@VictorEijkhouthttps://github.com/VictorEijkhout yes, you are correct. You need the enable MUMPS. I edited my above comment. Somehow the line -D TPL_ENABLE_MUMPS:BOOL=ON \ was not displayed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/trilinos/Trilinos/issues/6339?email_source=notifications&email_token=AAFHBAVTNEVJ37IEZSUDMT3QVQ42VA5CNFSM4JRAWZB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFDZMWI#issuecomment-558339673, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFHBAXPSTGPWXQK5H5HKA3QVQ42VANCNFSM4JRAWZBQ.

This message is from an external sender. Learn more about why this matters.https://ut.service-now.com/sp?id=kb_article&number=KB0011401

chochmuth commented 4 years ago

@VictorEijkhout to 1: On an intel machine, I am adding, among other, the scalapack and blacs libs to the MUMPS libaries string. -D TPL_MUMPS_LIBRARIES:STRING="-L$MUMPS/lib -ldmumps -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lmkl_blacs_intelmpi_lp64 -lpthread -lifcore" \ The above define replaces the previous two lines

-D MUMPS_LIBRARY_DIRS:PATH=$MUMPS/lib \
-D MUMPS_LIBRARY_NAMES:STRING="dmumps;mumps_common;pord" \

I can't remember why I have added them there, but I probably ran into a similar issue like you did, when adding them to the BLAS_LIBRARY_NAMES.

2: You don't need SCOTCH. You can enable it optionally when compiling MUMPS. 3: Same for ParMETIS and METIS.

However, I would recommend to use one of the two. I only have experience using METIS/ParMETIS for the ordering which works fine.

VictorEijkhout commented 4 years ago

I'm having a compilation problem, and I can't find the error message. Full and total log file attached. trilinoslog.zip

dangars commented 4 years ago

I use MUMPS 5.2.1 and I use the makefile for Debian.

In order to build MUMPS for Trilinos I had to activate the option -fPIC in MUMPS. I took a look to the compile options of the bundled MUMPS of PETSc. And if I add the following options to MUMPS I'm able to compile Trilinos with MUMPS:

SCALAP = -lscalapack-openmpi -lblacs-openmpi
OPTF    =  -fPIC  -ffree-line-length-0 -Wno-unused-dummy-argument -g 
OPTL    = -O -I.
OPTC    = -fPIC -fstack-protector -g3

chochmuth commented 4 years ago

@VictorEijkhout Your compilation problem does not seem to be related to the Amesos_Mumps interface. Maybe you should post this problem somewhere else and someone from the Core-Trilinos team can have a look.

VictorEijkhout commented 4 years ago

Actually, when I ran the install single-threaded I saw the error message:

[ 19%] Linking CXX executable Amesos_example_AmesosFactory.exe
cd /tmp/trilinos-build/packages/amesos/example && /opt/apps/cmake/3.10.2/bin/cmake -E cmake_link_script CMakeFiles/Amesos_example_AmesosFactory.dir/link.txt --verbose=1
/opt/apps/intel18/impi/18.0.2/bin/mpicxx  -g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl -DMPICH_SKIP_MPICXX -std=c++11 -O3 -DNDEBUG   CMakeFiles/Amesos_example_AmesosFactory.dir/example_AmesosFactory.cpp.o  -o Amesos_example_AmesosFactory.exe -Wl,-rpath,/tmp/trilinos-build/packages/amesos/src:/tmp/trilinos-build/packages/galeri/src-xpetra:/tmp/trilinos-build/packages/galeri/src-epetra:/tmp/trilinos-build/packages/common/auxiliarySoftware/SuiteSparse/src:/tmp/trilinos-build/packages/xpetra/sup:/tmp/trilinos-build/packages/xpetra/src:/tmp/trilinos-build/packages/thyra/adapters/tpetra/src:/tmp/trilinos-build/packages/thyra/adapters/epetraext/src:/tmp/trilinos-build/packages/epetraext/src:/tmp/trilinos-build/packages/triutils/src:/opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64/lib:/tmp/trilinos-build/packages/thyra/adapters/epetra/src:/tmp/trilinos-build/packages/thyra/core/src:/tmp/trilinos-build/packages/rtop/src:/tmp/trilinos-build/packages/tpetra/core/ext:/tmp/trilinos-build/packages/tpetra/core/inout:/tmp/trilinos-build/packages/tpetra/core/src:/tmp/trilinos-build/packages/epetra/src:/tmp/trilinos-build/packages/tpetra/tsqr/src:/tmp/trilinos-build/packages/kokkos-kernels/src:/tmp/trilinos-build/packages/kokkos/algorithms/src:/tmp/trilinos-build/packages/kokkos/containers/src:/tmp/trilinos-build/packages/tpetra/classic/LinAlg:/tmp/trilinos-build/packages/tpetra/classic/NodeAPI:/tmp/trilinos-build/packages/tpetra/classic/src:/tmp/trilinos-build/packages/teuchos/kokkoscomm/src:/tmp/trilinos-build/packages/teuchos/kokkoscompat/src:/tmp/trilinos-build/packages/teuchos/remainder/src:/tmp/trilinos-build/packages/teuchos/numerics/src:/tmp/trilinos-build/packages/teuchos/comm/src:/tmp/trilinos-build/packages/teuchos/parameterlist/src:/tmp/trilinos-build/packages/teuchos/parser/src:/tmp/trilinos-build/packages/teuchos/core/src:/tmp/trilinos-build/packages/kokkos/core/src ../src/libamesos.so.12.18.1 ../../galeri/src-xpetra/libgaleri-xpetra.so.12.18.1 ../../galeri/src-epetra/libgaleri-epetra.so.12.18.1 ../../common/auxiliarySoftware/SuiteSparse/src/libtrilinosss.so.12.18.1 -lm -L/home1/apps/intel18/impi18_0/petsc/3.12/skylake/lib ../../xpetra/sup/libxpetra-sup.so.12.18.1 ../../xpetra/src/libxpetra.so.12.18.1 ../../thyra/adapters/tpetra/src/libthyratpetra.so.12.18.1 ../../thyra/adapters/epetraext/src/libthyraepetraext.so.12.18.1 ../../epetraext/src/libepetraext.so.12.18.1 ../../triutils/src/libtriutils.so.12.18.1 /opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64/lib/libhdf5.so /usr/lib64/libz.so /opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64/lib/libhdf5_hl.so ../../thyra/adapters/epetra/src/libthyraepetra.so.12.18.1 ../../thyra/core/src/libthyracore.so.12.18.1 ../../rtop/src/librtop.so.12.18.1 ../../tpetra/core/ext/libtpetraext.so.12.18.1 ../../tpetra/core/inout/libtpetrainout.so.12.18.1 ../../tpetra/core/src/libtpetra.so.12.18.1 ../../epetra/src/libepetra.so.12.18.1 ../../tpetra/tsqr/src/libkokkostsqr.so.12.18.1 ../../kokkos-kernels/src/libkokkoskernels.so.12.18.1 ../../kokkos/algorithms/src/libkokkosalgorithms.so.12.18.1 ../../kokkos/containers/src/libkokkoscontainers.so.12.18.1 ../../tpetra/classic/LinAlg/libtpetraclassiclinalg.so.12.18.1 ../../tpetra/classic/NodeAPI/libtpetraclassicnodeapi.so.12.18.1 ../../tpetra/classic/src/libtpetraclassic.so.12.18.1 ../../teuchos/kokkoscomm/src/libteuchoskokkoscomm.so.12.18.1 ../../teuchos/kokkoscompat/src/libteuchoskokkoscompat.so.12.18.1 ../../teuchos/remainder/src/libteuchosremainder.so.12.18.1 ../../teuchos/numerics/src/libteuchosnumerics.so.12.18.1 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread ../../teuchos/comm/src/libteuchoscomm.so.12.18.1 ../../teuchos/parameterlist/src/libteuchosparameterlist.so.12.18.1 ../../teuchos/parser/src/libteuchosparser.so.12.18.1 ../../teuchos/core/src/libteuchoscore.so.12.18.1 ../../kokkos/core/src/libkokkoscore.so.12.18.1 /usr/lib64/libdl.so
../src/libamesos.so.12.18.1: undefined reference to `dmumps_c'

That unresolved reference is because none of the mumps libraries are being linked.

And I definitely specify them:

+ cmake -VV -D BUILD_SHARED_LIBS:BOOL=ON -D Trilinos_VERBOSE_CONFIGURE=OFF -D CMAKE_VERBOSE_MAKEFILE=ON -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -D Trilinos_ASSERT_MISSING_PACKAGES=OFF -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -D Trilinos_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_EXAMPLES:BOOL=ON -D Trilinos_ENABLE_Export_Makefiles:BOOL=ON -D Trilinos_ENABLE_Fortran:BOOL=ON -D CMAKE_INSTALL_PREFIX:PATH=/home1/apps/intel18/impi18_0/trilinos/12.18.1 -D CMAKE_BUILD_TYPE:STRING=RELEASE -D 'CMAKE_C_FLAGS:STRING=-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl' -D 'CMAKE_CXX_FLAGS:STRING=-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl -DMPICH_SKIP_MPICXX' -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON -D Tpetra_INST_INT_LONG_LONG:BOOL=ON -D Tpetra_INST_INT_INT:BOOL=ON -D Tpetra_INST_DOUBLE:BOOL=ON -D Tpetra_INST_FLOAT:BOOL=OFF -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF -D Tpetra_INST_INT_LONG:BOOL=OFF -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF -D BLAS_INCLUDE_DIRS:PATH=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/include -D BLAS_LIBRARY_DIRS:PATH=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -D 'BLAS_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64' -D LAPACK_INCLUDE_DIRS:PATH=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/include -D LAPACK_LIBRARY_DIRS:PATH=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -D 'LAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64' -D TPL_ENABLE_MPI:BOOL=ON -D MPI_EXEC:FILEPATH=/opt/apps/xalt/0.6/bin/ibrun -D TPL_MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include/.. -D TPL_MPI_INCLUDE_DIRS=/opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/include -D TPL_ENABLE_GLM=OFF -D TPL_ENABLE_Matio=OFF -D TPL_ENABLE_MUMPS:BOOL=ON -D MUMPS_INCLUDE_DIRS=/home1/apps/intel18/impi18_0/petsc/3.12/skylake/include -D TPL_MUMPS_LIBRARIES:STRING=-L/home1/apps/intel18/impi18_0/petsc/3.12/skylake/lib -ldmumps -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lmkl_blacs_intelmpi_lp64 -lpthread -lifcore -D 'TPL_ParMETIS_LIBRARIES=/libptscotchparmetis.a;/libparmetis.so' -D TPL_ParMETIS_INCLUDE_DIRS= -D TPL_ENABLE_Boost:BOOL=ON -D Boost_INCLUDE_DIRS:PATH=/opt/apps/intel18/boost/1.68/include -D Boost_LIBRARY_DIRS:PATH=/opt/apps/intel18/boost/1.68/lib -D TPL_ENABLE_BoostLib:BOOL=ON -D BoostLib_INCLUDE_DIRS:PATH=/opt/apps/intel18/boost/1.68/include -D BoostLib_LIBRARY_DIRS:PATH=/opt/apps/intel18/boost/1.68/lib -D TPL_ENABLE_HDF5:BOOL=ON -D HDF5_INCLUDE_DIRS:PATH=/opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64/include -D HDF5_LIBRARY_DIRS:PATH=/opt/apps/intel18/impi18_0/phdf5/1.10.4/x86_64/lib -D TPL_ENABLE_Netcdf:BOOL=ON -D Netcdf_INCLUDE_DIRS:PATH=/opt/apps/intel18/impi18_0/parallel-netcdf/4.6.2/x86_64/include -D Netcdf_LIBRARY_DIRS:PATH=/opt/apps/intel18/impi18_0/parallel-netcdf/4.6.2/x86_64/lib -D VLE_SUPERLU_IS_FIXED_AFTER_12.12:BOOL=ON -D TPL_ENABLE_SuperLU:BOOL=OFF -D SuperLU_INCLUDE_DIRS:PATH=/home/foo/include -D SuperLU_LIBRARY_DIRS:PATH=/home/foo -D SuperLU_LIBRARY_NAMES:STRING=superlu -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON -D Trilinos_ENABLE_Amesos:BOOL=ON -D Trilinos_ENABLE_Amesos2:BOOL=ON -D Amesos2_ENABLE_KLU2:BOOL=ON -D Amesos2_ENABLE_Basker:BOOL=ON -D Trilinos_ENABLE_Anasazi:BOOL=ON -D Trilinos_ENABLE_AztecOO:Bool=ON -D Trilinos_ENABLE_Belos:BOOL=ON -D Trilinos_ENABLE_Epetra:Bool=ON -D Trilinos_ENABLE_EpetraExt:Bool=ON -D Epetra_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_FEI:Bool=ON -D Trilinos_ENABLE_Ifpack:Bool=ON -D Trilinos_ENABLE_Ifpack2:BOOL=ON -D Trilinos_ENABLE_Intrepid:BOOL=ON -D Trilinos_ENABLE_Intrepid2:BOOL=ON -D Intrepid_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_Issoropia:BOOL=ON -D Trilinos_ENABLE_ML:BOOL=ON -D ML_TAKES_SUPERLU_LESS_THAN_5=TRUE -D ML_ENABLE_SuperLU:BOOL=OFF -D Trilinos_ENABLE_MOOCHO:BOOL=ON -D Trilinos_ENABLE_MueLu:BOOL=ON -D MueLu_ENABLE_Tutorial:BOOL=OFF -D MueLu_ENABLE_EXAMPLES:BOOL=OFF -D Trilinos_ENABLE_NOX=ON -D NOX_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Pamgen:Bool=ON -D Pamgen_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Panzer:Bool=ON -D Trilinos_ENABLE_Phalanx:BOOL=ON -D Phalanx_EXPLICIT_TEMPLATE_INSTANTIATION=ON -D Phalanx_ENABLE_EXAMPLES=OFF -D Trilinos_ENABLE_Piro:BOOL=ON -D Trilinos_ENABLE_Rythmos:BOOL=ON -D Trilinos_ENABLE_Sacado:Bool=ON -D Trilinos_ENABLE_SEACAS:BOOL=ON -D Trilinos_ENABLE_SEACASIoss:BOOL=ON -D Trilinos_ENABLE_SEACASBlot:BOOL=ON -D Trilinos_ENABLE_SEACASExodus:BOOL=ON -D Trilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON -D Trilinos_ENABLE_Shards:BOOL=ON -D Trilinos_ENABLE_ShyLU:BOOL=ON -D Trilinos_ENABLE_Stokhos:BOOL=ON -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_Teko:BOOL=ON -D Trilinos_ENABLE_Teuchos:BOOL=ON -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON -D Trilinos_ENABLE_Thyra:BOOL=ON -D Trilinos_ENABLE_Tpetra:BOOL=ON -D Trilinos_ENABLE_TriKota:BOOL=ON -D Trilinos_ENABLE_Zoltan:BOOL=ON -D Trilinos_ENABLE_Zoltan2:BOOL=ON -D Trilinos_ENABLE_Kokkos:BOOL=ON -D Trilinos_ENABLE_KokkosCore:BOOL=ON -D Phalanx_KOKKOS_DEVICE_TYPE:STRING=SERIAL -D Phalanx_INDEX_SIZE_TYPE:STRING=INT -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF -D Kokkos_ENABLE_Serial:BOOL=ON -D Kokkos_ENABLE_OpenMP:BOOL=OFF -D Kokkos_ENABLE_Pthread:BOOL=OFF -D Trilinos_ENABLE_STK:BOOL=ON -D Trilinos_ENABLE_STKIO:BOOL=ON -D Trilinos_ENABLE_STKMesh:BOOL=ON -D Trilinos_ENABLE_PyTrilinos:Bool=OFF -D CMAKE_PYTHON_INCLUDE_DIR:PATH=NOPYTHON -D CMAKE_PYTHON_LIBRARIES:STRING=NOPYTHON -D 'CMAKE_PYTHON_EXECUTABLE:FILEPATH=/python%{python_major_version}' -D PyTrilinos_DOCSTRINGS:BOOL=OFF -D PyTrilinos_ENABLE_Tpetra:BOOL=OFF -D SWIG_EXECUTABLE:FILEPATH=/home1/apps/intel18/swig/3.0.12/bin/swig /admin/build/admin/rpms/stampede2/BUILD//trilinos-12.18.1

and your cmake does not complain about them so they should be fine....

-- MUMPS_LIBRARY_NAMES='dmumps;libmumps_common;pord'
-- TPL_MUMPS_LIBRARIES='-L/home1/apps/intel18/impi18_0/petsc/3.12/skylake/lib'
-- Searching for headers in MUMPS_INCLUDE_DIRS='/home1/apps/intel18/impi18_0/petsc/3.12/skylake/include'
-- Searching for a header file in the set "dmumps_c.h":
--   Searching for header 'dmumps_c.h' ...
--     Found header '/home1/apps/intel18/impi18_0/petsc/3.12/skylake/include/dmumps_c.h'
-- Found TPL 'MUMPS' include dirs '/home1/apps/intel18/impi18_0/petsc/3.12/skylake/include'
-- TPL_MUMPS_INCLUDE_DIRS='/home1/apps/intel18/impi18_0/petsc/3.12/skylake/include'

So why are they missing in that link line?

V.

chochmuth commented 4 years ago

Ok, so cmake finds the header but it doesn't even look for the libraries. On my local machine the configure script does the following:

-- MUMPS_LIBRARY_NAMES='dmumps;mumps_common;pord'
-- Searching for libs in MUMPS_LIBRARY_DIRS='/Users/chris/programs/installed/MUMPS_5.1.1/lib'
-- Searching for a lib in the set "dmumps":
--   Searching for lib 'dmumps' ...
--     Found lib '/Users/chris/programs/installed/MUMPS_5.1.1/lib/libdmumps.a'
-- Searching for a lib in the set "mumps_common":
--   Searching for lib 'mumps_common' ...
--     Found lib '/Users/chris/programs/installed/MUMPS_5.1.1/lib/libmumps_common.a'
-- Searching for a lib in the set "pord":
--   Searching for lib 'pord' ...
--     Found lib '/Users/chris/programs/installed/MUMPS_5.1.1/lib/libpord.a'

Can you run the configure script with MUMPS_LIBRARY_DIRS=... instead of TPL_MUMPS_LIBRARIES=... to specify the path of MUMPS libs.

dangars commented 4 years ago

@chochmuth I think that the problems is related to Amesos_MUMPS and the trilinos CMake configuration. It seems that it does not link to the required libraries.

I tried the configuration that you suggested with MUMPS_LIBRARY_DIRS and I have the same problem. Trilinos/Amesos compiles with MUMPS but when I try to run an application with Amesos/MUMPS then I see unresolved references (the same as @VictorEijkhout)

packages/amesos/test/Test_Basic/CMakeFiles/amesos_test.dir/build.make:268: recipe for target 'packages/amesos/test/Test_Basic/amesos_test.exe' failed
make[2]: *** [packages/amesos/test/Test_Basic/amesos_test.exe] Error 1
CMakeFiles/Makefile2:48369: recipe for target 'packages/amesos/test/Test_Basic/CMakeFiles/amesos_test.dir/all' failed
make[1]: *** [packages/amesos/test/Test_Basic/CMakeFiles/amesos_test.dir/all] Error 2
[ 89%] Linking CXX executable Amesos_TestOptions.exe
../../src/libamesos.so.12.14.1: undefined reference to `blacs_gridinfo_'
../../src/libamesos.so.12.14.1: undefined reference to `scotchfgraphpart_'
../../src/libamesos.so.12.14.1: undefined reference to `METIS_SetDefaultOptions'
../../src/libamesos.so.12.14.1: undefined reference to `scotchfgraphbuild_'
../../src/libamesos.so.12.14.1: undefined reference to `scotchfgraphexit_'
../../src/libamesos.so.12.14.1: undefined reference to `blacs_gridinit_'
../../src/libamesos.so.12.14.1: undefined reference to `descinit_'
../../src/libamesos.so.12.14.1: undefined reference to `pdpotrf_'
../../src/libamesos.so.12.14.1: undefined reference to `pdgetrs_'
../../src/libamesos.so.12.14.1: undefined reference to `metis_nodend_'
../../src/libamesos.so.12.14.1: undefined reference to `blacs_gridexit_'
../../src/libamesos.so.12.14.1: undefined reference to `METIS_PartGraphKway'

VictorEijkhout commented 4 years ago

Guys, I'd like this to be solved. Help?

chochmuth commented 4 years ago

@VictorEijkhout Sorry that I can't help you more. Yes, it seems that some libaries, e.g., BLACS/SCALAPACK, are not linked correctly, as it is the case for the last error message by @dangars. @VictorEijkhout your last error message is really confusing, since now even the MUMPS libraries are not linked correctly. Sadly, I am no expert on the CMake structure of the Amesos package. Maybe someone from the @trilinos/amesos team can help us out.

@dangars none of the references, which are missing, are for MUMPS. However, I presented some workarounds for that in one of my comments above. You can add BLACS/SCALAPACK (and it seems you also need Metis) to the MUMPS library string. Here is an example if you are on an intel machine with mkl:

TPL_MUMPS_LIBRARIES:STRING="-L$MUMPS/lib -ldmumps -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lmkl_blacs_intelmpi_lp64 -lpthread -lifcore" \

github-actions[bot] commented 2 years ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

github-actions[bot] commented 2 years ago

This issue was closed due to inactivity for 395 days.

trilinos / Trilinos

Is there a new version of the Amesos Mumps instructions? #6339

ifdef MUMPS_4_9

else

ifndef HAVE_AMESOS_OLD_MUMPS

else

endif

endif