MueLu + Amesos2 + Mumps

[mathsen]

Hello,

I try to use MueLu with Mumps as direct solver through Amesos2 on trilinos master. According to the MueLu release notes, this should work:

• Mumps can now be used as a coarse grid solver through Amesos2.

The situation in my case is now as follows: in order to use Mumps within Amesos2, according to here one needs to enable Tpetra_INST_INT_INT. If I do this in combination with MueLu, so:

    ...
    -D MueLu_ENABLE_TESTS:STRING=ON \
    -D MueLu_ENABLE_EXAMPLES:STRING=ON \
    -D MueLu_ENABLE_Kokkos_Refactor:STRING=ON \
    -D Tpetra_INST_INT_INT:BOOL=ON \
    ....

MueLu will fail linking:

ld: ../../../../tpetra/core/src/libtpetra.a(Tpetra_MultiVector_DOUBLE_INT_INT_SERIAL.cpp.o): in function Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos: :HostSpace> >::update(double const&, Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, double const&)': Tpetra_MultiVector_DOUBLE_INT_INT_SERIAL.cpp:(.text._ZN6Tpetra11MultiVectorIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos6SerialENS3_9HostSpaceEEEE6updateERKdRKS7_S9_[_ZN6Tpetra11MultiVectorIdiiNS_12Kokko sCompat23KokkosDeviceWrapperNodeIN6Kokkos6SerialENS3_9HostSpaceEEEE6updateERKdRKS7_S9_]+0xe4c): undefined reference toKokkosBlas::Impl::Axpby<Kokkos::Serial, double, Kokkos::View<double const, Kokkos::LayoutLe ft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >, double, Kokkos::View<double, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> >, 2, false, true>::axpby(Kokkos::Serial const&, double const&, Kokkos::View<double const, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> > const&, double const&, Ko kkos::View<double, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1u> > const&)' ld: ../../../../teuchos/parameterlist/src/libteuchosparameterlist.a(Teuchos_StandardDependencyXMLConverters.cpp.o): in function Teuchos::ConditionVisualDependencyXMLConverter::convertSpecialVisualAttributes(Teuch os::RCP<Teuchos::VisualDependency const>, Teuchos::XMLObject&, std::map<Teuchos::RCP<Teuchos::ParameterEntry const>, unsigned int, Teuchos::RCPConstComp, std::allocator<std::pair<Teuchos::RCP<Teuchos::ParameterEnt ry const> const, unsigned int> > > const&) const': Teuchos_StandardDependencyXMLConverters.cpp:(.text+0x4798): undefined reference toTeuchos::ConditionXMLConverterDB::convertCondition(Teuchos::RCP, std::map<Teuchos::RCP<Teuchos::Paramet erEntry const>, unsigned int, Teuchos::RCPConstComp, std::allocator<std::pair<Teuchos::RCP const, unsigned int> > > const&)' ld: ../../../../teuchos/parameterlist/src/libteuchosparameterlist.a(Teuchos_StandardDependencyXMLConverters.cpp.o): in function Teuchos::ConditionVisualDependencyXMLConverter::convertSpecialVisualAttributes(Teuch os::XMLObject const&, std::set<Teuchos::RCP<Teuchos::ParameterEntry const>, Teuchos::RCPConstComp, std::allocator<Teuchos::RCP<Teuchos::ParameterEntry const> > >, std::set<Teuchos::RCP<Teuchos::ParameterEntry>, Te uchos::RCPComp, std::allocator<Teuchos::RCP<Teuchos::ParameterEntry> > >, bool, std::map<unsigned int, Teuchos::RCP<Teuchos::ParameterEntry>, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, T euchos::RCP<Teuchos::ParameterEntry> > > > const&) const': Teuchos_StandardDependencyXMLConverters.cpp:(.text+0x4cd4): undefined reference toTeuchos::ConditionXMLConverterDB::convertXML(Teuchos::XMLObject const&, std::map<unsigned int, Teuchos::RCP<Teuchos::ParameterEnt ry>, std::less, std::allocator<std::pair<unsigned int const, Teuchos::RCP > > > const&)'

A workaround is to disable MueLu Tests and examples:

    ...
    -D Trilinos_ENABLE_MueLu:BOOL=ON \
    -D MueLu_ENABLE_TESTS:STRING=OFF \
    -D MueLu_ENABLE_EXAMPLES:STRING=OFF \
    -D MueLu_ENABLE_Kokkos_Refactor:STRING=ON \
    -D Tpetra_INST_INT_INT:BOOL=ON \
    ....

Then Trilinos compiles - but later when I try to compile my own code I get the exactly same ld problems. I assume it has something to do with only enabling the global ordinal type int. However, if I enable more ordinal typers (especially long long):

    ...
    -D Trilinos_ENABLE_MueLu:BOOL=ON \
    -D MueLu_ENABLE_TESTS:STRING=OFF \
    -D MueLu_ENABLE_EXAMPLES:STRING=OFF \
    -D MueLu_ENABLE_Kokkos_Refactor:STRING=ON \
    -D Tpetra_INST_INT_INT:BOOL=ON \
    -D Tpetra_INST_INT_LONG:BOOL=ON \
    -D Tpetra_INST_INT_LONG_LONG:BOOL=ON \
    ...

I get the configure error:

Tpetra requires only one global ordinal, but more than one are set: {int;long;long long}.

Can anybody help me here, to make it possible to use this combination of MueLu, Amesos2 and Mumps?

Thanks and many greetings mathse

[github-actions[bot]]

Automatic mention of the @trilinos/muelu team

[cgcgcg]

Yes, disabling tests and examples only delays the problem. Enabling int as the single global ordinal is correct. I don't have MUMPS on my system, but maybe I can reproduce without it given the error doesn't seem to be coming out of Amesos2.

Side note: you don't need MueLu_ENABLE_Kokkos_Refactor anymore, and this probably gives an unused variable warning at configure time.

[cgcgcg]

@mathsen Hm, I'm not seeing the issue. Could you post your CMake line and the resulting terminal output?

[mathsen]

@cgcgcg thanks for looking into this! I made a new attempt without MueLu_ENABLE_Kokkos_Refactor and this finally worked! For completeness, my final CMake line is:

    -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_C_COMPILER=${CC} \
    -D CMAKE_CXX_COMPILER=${CXX} \
    -D CMAKE_Fortran_COMPILER=${FC} \
    -D CMAKE_CXX_FLAGS:STRING='-D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -O2 -DMUMPS_5_0' \
    -D CMAKE_C_FLAGS:STRING='-fPIC -O2' \
    -D CMAKE_FORTRAN_FLAGS:STRING='-O2' \
    -D BUILD_SHARED_LIBS:BOOL=OFF \
    -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
    -D Trilinos_ENABLE_TESTS:BOOL=ON \
    -D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \
    -D Trilinos_ENABLE_MueLu:BOOL=ON \
    -D MueLu_ENABLE_TESTS:STRING=ON \
    -D MueLu_ENABLE_EXAMPLES:STRING=ON \
    -D Tpetra_INST_INT_INT:BOOL=ON \
    -D TPL_ENABLE_BLAS:BOOL=ON \
    -D Trilinos_ENABLE_PyTrilinos:BOOL=OFF \
    -D Trilinos_ENABLE_Gtest:BOOL=OFF \
    -D Trilinos_ENABLE_TESTS:BOOL=OFF \
    -D Trilinos_ENABLE_TrilinosFrameworkTests:BOOL=OFF \
    -D Trilinos_ENABLE_TrilinosATDMConfigTests:BOOL=OFF \
    -D TPL_ENABLE_gtest:BOOL=OFF \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D TPL_ENABLE_HDF5:BOOL=OFF \
    -D TPL_ENABLE_ParMETIS:BOOL=ON \
    -D TPL_ParMETIS_LIBRARIES:FILEPATH='${PARMETIS_DIR}/lib/libparmetis.a;${PARMETIS_DIR}/lib/libmetis.a' \
    -D TPL_ParMETIS_INCLUDE_DIRS:PATH=${PARMETIS_DIR}/include \
    -D TPL_ENABLE_MUMPS=ON \
    -D MUMPS_INCLUDE_DIRS:FILEPATH='${MUMPS_DIR}/include' \
    -D MUMPS_LIBRARY_DIRS:FILEPATH='${MUMPS_DIR}/lib' \
    -D BLAS_LIBRARY_DIRS:STRING='/home/intel_compiler/2023.1.0/thirdparty/stable/package/mkl/2023.1.0/lib/intel64' \
    -D LAPACK_LIBRARY_DIRS:STRING='/home/intel_compiler/2023.1.0/thirdparty/stable/package/mkl/2023.1.0/lib/intel64' \
    -D BLAS_LIBRARY_NAMES='mkl_intel_lp64;mkl_sequential;mkl_core' \
    -D LAPACK_LIBRARY_NAMES='' \

I need to use an intel compiler, so $CC points wo mpiicc, similar $CXX to mpiicpc and $FC to mpiifort.

I just have another run running with MueLu_ENABLE_Kokkos_Refactor enabled, just to be sure that this was really the issue. Many greetings mathse

[mathsen]

OK, and also this version compiled now fine - and I indeed only got the warning about the deprecated option. So currently I can't reproduce my initial problem anymore. Therefore I will close this issue now. Thanks for the help and many greetings mathse