MueLu fails to build with Tpetra_INST_INT_INT:BOOL=OFF

[mhoemmen]

MueLu fails to build with Tpetra_INST_INT_INT:BOOL=OFF. Until this is fixed, Tpetra developers cannot improve build time and library sizes by removing support for all GlobalOrdinal types but int64_t.

Here is an example of the link errors I get:

../../adapters/libmuelu-adapters.so.12.13: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::~MultiVector()'
../../adapters/libmuelu-adapters.so.12.13: undefined reference to `Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::MultiVector(Tpetra::MultiVector<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&)'
../../adapters/libmuelu-adapters.so.12.13: undefined reference to `Ifpack2::Hiptmair<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >::Hiptmair(Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const> const&, Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const> const&, Teuchos::RCP<Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const> const&)'

@trilinos/muelu

Related Issues

• Blocks #2548

[mhoemmen]

I added a static_assert to Maxwell, and found that MueLu was attempting to build Maxwell for for GO=int, even though I had set Tpetra_INST_INT_INT:BOOL=OFF.

[ 98%] Building CXX object packages/muelu/adapters/CMakeFiles/muelu-adapters.dir/ExplicitInstantiation/MueLu_RefMaxwell.cpp.o
In file included from .../packages/muelu/adapters/ExplicitInstantiation/MueLu_RefMaxwell.cpp:52:0:
.../Trilinos/packages/muelu/adapters/xpetra/MueLu_RefMaxwell_def.hpp: In instantiation of ‘void MueLu::RefMaxwell<Scalar, LocalOrdinal, GlobalOrdinal, Node>::compute() [with Scalar = double; LocalOrdinal = int; GlobalOrdinal = int; Node = Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP>]’:
.../Trilinos/packages/muelu/src/Interface/../Utils/MueLu_ETI_4arg.hpp:35:3:   required from here
.../Trilinos/packages/muelu/adapters/xpetra/MueLu_RefMaxwell_def.hpp:400:9: error: static assertion failed: Uh oh, GO is int for Hiptmair
         static_assert (! std::is_same<GO, int>::value, "Uh oh, GO is int for Hiptmair");
         ^

[mhoemmen]

It looks like the problem is that MueLu might enable GO=int for its Epetra back-end. If both the Epetra and Tpetra back-ends are enabled, then RefMaxwell will try to instantiate Tpetra objects for GO=int, even though this is not enabled.

[mhoemmen]

@micahahoward Right now SPARC can't build Intel debug because libmuelu.a is 4.2G. I'm hoping to cut this down by setting Tpetra_INST_INT_INT:BOOL=OFF and only enabling GO=long long.

[cgcgcg]

@mhoemmen I'll have a look at this.

[cgcgcg]

So we should essentially augment the pragma https://github.com/trilinos/Trilinos/blob/69f8850c25bdead51b38866ae34d4fa40947b67f/packages/muelu/adapters/xpetra/MueLu_RefMaxwell_def.hpp#L1173 to read #if defined(HAVE_MUELU_IFPACK2) && (!defined(HAVE_MUELU_EPETRA) || (defined(HAVE_MUELU_EPETRA) && defined(HAVE_MUELU_TPETRA_INST_INT_INT)))

[mhoemmen]

@cgcgcg Thanks for working on this! I'm a little more worried about the bigger issue, namely that it looks like a bunch of stuff in MueLu won't work if GO=int is disabled.

[mhoemmen]

On the other hand, if Tpetra purges the GlobalOrdinal template parameter, then there's no need for Xpetra or MueLu to have that template parameter, either. So perhaps I'm worrying for no reason.

[cgcgcg]

@mhoemmen I set Tpetra_INST_INT_INT:BOOL=OFF in my build, but somehow the compilation still goes though. I do get a bunch of build errors from panzer, but all of MueLu seems to compile. Epetra is enabled, and when I try to run the Maxwell example, it tells me that it won't work since I don't have the correct Tpetra instantiations. Could you share your configure script?

[mhoemmen]

-D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-D BUILD_SHARED_LIBS:BOOL=ON
-D Trilinos_ENABLE_OpenMP:BOOL=ON
-D Kokkos_ENABLE_OpenMP:BOOL=ON
-D Tpetra_INST_OPENMP:BOOL=ON
-D Trilinos_SHOW_DEPRECATED_WARNINGS:BOOL=ON
-D Trilinos_ENABLE_Fortran:BOOL=ON
-D CMAKE_CXX_FLAGS:STRING="-Wall --pedantic"
-D TPL_ENABLE_MKL:BOOL=OFF
-D TPL_ENABLE_Matio:BOOL=OFF
-D TPL_ENABLE_SuperLU:BOOL=ON
  -D SuperLU_INCLUDE_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/superlu/4.3/gcc/4.9.3/base/include
  -D SuperLU_LIBRARY_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/superlu/4.3/gcc/4.9.3/base/lib
-D TPL_ENABLE_Zlib:BOOL=ON
  -D Zlib_INCLUDE_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/zlib/1.2.8/gcc/4.9.3/base/include
  -D Zlib_LIBRARY_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/zlib/1.2.8/gcc/4.9.3/base/lib
-D TPL_ENABLE_Netcdf:BOOL=ON
  -D Netcdf_INCLUDE_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/netcdf/4.4.1/gcc/4.9.3/openmpi/1.10.1/include
  -D Netcdf_LIBRARY_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/netcdf/4.4.1/gcc/4.9.3/openmpi/1.10.1/lib
-D TPL_ENABLE_HDF5:BOOL=ON
  -D HDF5_INCLUDE_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/hdf5/1.8.12/gcc/4.9.3/openmpi/1.10.1/include
  -D HDF5_LIBRARY_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/hdf5/1.8.12/gcc/4.9.3/openmpi/1.10.1/lib
-D TPL_ENABLE_ParMETIS:BOOL=ON
  -D ParMETIS_INCLUDE_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/parmetis/4.0.3/gcc/4.9.3/openmpi/1.10.1/include
  -D ParMETIS_LIBRARY_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/parmetis/4.0.3/gcc/4.9.3/openmpi/1.10.1/lib
-D TPL_ENABLE_Boost:BOOL=ON
  -D Boost_INCLUDE_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/boost/1.59.0/gcc/4.9.3/base/include
  -D Boost_LIBRARY_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/boost/1.59.0/gcc/4.9.3/base/lib
-D TPL_ENABLE_BoostLib:BOOL=ON
  -D BoostLib_INCLUDE_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/boost/1.59.0/gcc/4.9.3/base/include
  -D BoostLib_LIBRARY_DIRS=/projects/sems/install/rhel6-x86_64/sems/tpl/boost/1.59.0/gcc/4.9.3/base/lib
-D TPL_ENABLE_yaml-cpp:BOOL=OFF
-D TPL_ENABLE_MPI:BOOL=ON
-D CMAKE_BUILD_TYPE:STRING=DEBUG
-D Kokkos_ENABLE_Pthread:BOOL=OFF
-D Tpetra_INST_SERIAL:BOOL=OFF
-D Kokkos_ENABLE_DEBUG:BOOL=ON
-D Teuchos_ENABLE_DEBUG:BOOL=ON

[mhoemmen]

Thanks @cgcgcg ! I got SPARC's configuration of Trilinos to build when I set Tpetra_INST_INT_INT=OFF and Amesos2_ENABLE_Epetra=OFF. I didn't try enabling tests or examples, but at least I could build MueLu. I'll close this issue.

[cgcgcg]

@mhoemmen I don't think my changes got merged yet ;-) Maybe disabling tests and examples did the trick for you?

[mhoemmen]

@cgcgcg oops, good point :(

[cgcgcg]

@mhoemmen OK, merged the PR. I hope this fixed the issue, otherwise let me know and I'll investigate.

[mhoemmen]

Thanks @cgcgcg ! I got it to work with Trilinos:develop, so I'll close this issue. I was able to build MueLu with the following CMake options:

• Tpetra_INST_INT_INT:BOOL=OFF
• Amesos2_ENABLE_Epetra:BOOL=OFF
• Tpetra_INST_Serial:BOOL=OFF (only the OpenMP Node enabled)

with GCC 4.9.3 and OpenMPI 1.10.1, and the following package enables and disables:

Final set of enabled packages:  Gtest Kokkos Teuchos KokkosKernels RTOp Sacado MiniTensor Epetra Zoltan GlobiPack Triutils Tpetra TrilinosSS EpetraExt Thyra Xpetra OptiPack Isorropia AztecOO Galeri Amesos Pamgen Zoltan2 Ifpack ML Belos ShyLU_Node Amesos2 SEACAS Anasazi Ifpack2 Stratimikos FEI Teko STK NOX MueLu ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro TrilinosCouplings Pike 46

Final set of enabled SE packages:  Gtest KokkosCore KokkosContainers KokkosAlgorithms Kokkos TeuchosCore TeuchosParser TeuchosParameterList TeuchosComm TeuchosNumerics TeuchosRemainder TeuchosKokkosCompat TeuchosKokkosComm Teuchos KokkosKernels RTOp Sacado MiniTensor Epetra Zoltan GlobiPack Triutils TpetraClassic TpetraTSQR TpetraCore Tpetra TrilinosSS EpetraExt ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Thyra Xpetra OptiPack Isorropia AztecOO Galeri Amesos Pamgen Zoltan2 Ifpack ML Belos ShyLU_NodeHTS ShyLU_NodeTacho ShyLU_Node Amesos2 SEACASAprepro_lib SEACAS Anasazi Ifpack2 Stratimikos FEI Teko STKUtil STKSimd STKTopology STKExprEval STK NOX MueLu ShyLU_DDFROSch ShyLU_DDCommon ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro TrilinosCouplings PikeBlackBox PikeImplicit Pike 75

Final set of non-enabled packages:  ThreadPool Shards Domi Pliris Claps Trios Komplex TriKota Intrepid Intrepid2 Phalanx Moertel Panzer PyTrilinos NewPackage MeshingGenie 16

Final set of non-enabled SE packages:  ThreadPool KokkosExample Shards Domi Pliris Claps ShyLU_NodeBasker ShyLU_NodeFastILU SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASNemesis SEACASIoss SEACASChaco SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASPLT SEACASAlgebra SEACASAprepro SEACASBlot SEACASConjoin SEACASEjoin SEACASEpu SEACASExo2mat SEACASExodiff SEACASExomatlab SEACASExotxt SEACASExo_format SEACASExplore SEACASEx1ex2v2 SEACASFastq SEACASGjoin SEACASGen3D SEACASGenshell SEACASGrepos SEACASMapvarlib SEACASMapvar SEACASMapvar-kd SEACASMat2exo SEACASNemslice SEACASNemspread SEACASNumbers SEACASTxtexo SEACASEx2ex1v2 Trioscommsplitter Triossupport Triosnnti Triosnssi Triosprograms Triosexamples Triostests Triosnetcdf-service Trios Komplex TriKota Intrepid Intrepid2 STKClassic STKMesh STKIO STKUnit_test_utils STKSearch STKSearchUtil STKTransfer STKTools STKUnit_tests STKDoc_tests STKExp Phalanx Moertel ShyLU_DDBDDC ShyLU_DDCore PanzerCore PanzerDofMgr PanzerDiscFE PanzerAdaptersSTK PanzerAdaptersIOSS PanzerMiniEM PanzerExprEval Panzer PyTrilinos NewPackage MeshingGenie 86

Final set of enabled TPLs:  Pthread MPI BLAS LAPACK Boost ParMETIS Zlib HDF5 Netcdf SuperLU BoostLib DLlib 12

Final set of non-enabled TPLs:  MKL yaml-cpp Peano CUDA CUSPARSE Thrust Cusp TBB HWLOC QTHREAD BinUtils ARPREC QD Scotch OVIS gpcd METIS MTMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF CGNS Pnetcdf y12m SuperLUDist SuperLUMT Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Clp GLPK qpOASES Matio PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX CGAL CGALCore VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField Valgrind QUO ViennaCL Avatar pebbl 96

If I have trouble with a different, I'll open a new issue. Thanks!

[mhoemmen]

@micahahoward @tcfisher @prwolfe @rrdrake FYI