search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.21k stars 563 forks source link

teuchos: Teuchos_ExplicitInstantiationHelpers.hpp incorrectly instantiates for complex<float> if Trilinos_ENABLE_FLOAT and Trilinos_ENABLE_COMPLEX_DOUBLE are set to ON, but TRILINOS_ENABLE_COMPLEX_FLOAT is off. #10635

Closed vbrunini closed 2 years ago

vbrunini commented 2 years ago

Bug Report

@trilinos/teuchos

Description

Teuchos_ExplicitInstantiationHelpers.hpp checks #if defined(HAVE_TEUCHOS_COMPLEX) && defined(HAVE_TEUCHOS_FLOAT) to determine whether or not to instantiate for complex. This incorrectly enables those instantiations when the user wants to only instantiate for float, double, and complex.

This is causing linking errors when trying to build Sierra against Trilinos/develop. I think this issue has been latent in Teuchos for a long time, but is now impacting us after commit e08b53677de4ac72b9e060001dd5e63b6c2b167f which started using these Teuchos macros in stratimikos.

jhux2 commented 2 years ago

@trilinos/teuchos @cgcgcg @bartlettroscoe

bartlettroscoe commented 2 years ago

So you want double, float and std::complex<double> but not std::complex<float>?

What are your exact Trilinos configure options so that I can reproduce?

vbrunini commented 2 years ago

Your comment got a bit mangled, but I think you're correct. We want double, float, std::complex<double> but not std::complex<float>.

Here's my set of configure options from a recent Cuda build that hit the problem, but I also saw it in a clang build with just the Kokkos Serial backend that would probably be faster to reproduce. Note that Trilinos itself compiled and installed, it was only when trying to link against Trilinos that I saw the errors (though maybe the Trilinos tests would show the problem as well?)


-DBUILD_SHARED_LIBS:BOOL=OFF
-DCMAKE_BUILD_TYPE:STRING=release
-DCMAKE_C_COMPILER=/usr/tce/packages/spectrum-mpi/spectrum-mpi-2019.06.24-gcc-7.2.1-redhat/bin/mpicc
-DCMAKE_CXX_COMPILER=/usr/tce/packages/spectrum-mpi/spectrum-mpi-2019.06.24-gcc-7.2.1-redhat/bin/mpicxx
-DCMAKE_Fortran_COMPILER=/usr/tce/packages/spectrum-mpi/spectrum-mpi-2019.06.24-gcc-7.2.1-redhat/bin/mpif90
-DCMAKE_CXX_STANDARD=14
-DCMAKE_C_FLAGS_RELEASE_OVERRIDE:STRING -DNDEBUG
-DCMAKE_CXX_FLAGS_RELEASE_OVERRIDE:STRING -DNDEBUG
-DCMAKE_Fortran_FLAGS_RELEASE_OVERRIDE:STRING -DNDEBUG
-DCMAKE_C_FLAGS:STRING=-O2 -w
-DCMAKE_CXX_FLAGS:STRING=-O2 -w
-DCMAKE_Fortran_FLAGS:STRING=-O2 -w -fno-underscoring
-DCMAKE_EXE_LINKER_FLAGS:STRING=
-DHAVE_GCC_ABI_DEMANGLE:BOOL=ON
-DHAVE_TEUCHOS_BLASFLOAT:BOOL=ON
-DHAVE_TEUCHOS_BLASFLOAT_DOUBLE_RETURN:BOOL=OFF
-DTrilinos_ASSERT_MISSING_PACKAGES:BOOL=OFF
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF
-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF
-DTrilinos_ENABLE_COMPLEX_DOUBLE:BOOL=ON
-DTrilinos_ENABLE_DEBUG:BOOL=OFF
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-DTrilinos_ENABLE_FLOAT:BOOL=ON
-DTrilinos_ENABLE_LINEAR_SOLVER_FACTORY_REGISTRATION:BOOL=ONTrilinos_SHOW_DEPRECATED_WARNINGS:BOOL=OFF
-DTrilinos_WARNINGS_AS_ERRORS_FLAGS:STRING=
-DTPL_ENABLE_BLAS:BOOL=ON
-DBLAS_LIBRARY_NAMES=blas
-DBLAS_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/netlib-lapack-3.9.0-x7kopcaztcrjx6vdf6posee26zp5ikwi/lib64
-DTPL_ENABLE_Boost:BOOL=ON
-DTPL_Boost_INCLUDE_DIRS:PATH=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/boost-1.77.0-lik62zm2hcn6eftuxlx2t6fpvtrow2ru/include
-DBoost_LIBRARY_DIRS:PATH=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/boost-1.77.0-lik62zm2hcn6eftuxlx2t6fpvtrow2ru/lib
-DTPL_ENABLE_CUDA:BOOL=ON
-DTPL_ENABLE_LAPACK:BOOL=ON
-DLAPACK_LIBRARY_NAMES=lapack
-DLAPACK_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/netlib-lapack-3.9.0-x7kopcaztcrjx6vdf6posee26zp5ikwi/lib64
-DTPL_ENABLE_METIS:BOOL=ON
-DMETIS_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/metis-5.1.0-ghfislukb3c2rsv4qncvl2vyoinul7ba/lib
-DMETIS_LIBRARY_NAMES=metis
-DMETIS_INCLUDE_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/metis-5.1.0-ghfislukb3c2rsv4qncvl2vyoinul7ba/include
-DTPL_ENABLE_MPI:BOOL=ON
-DMPI_BASE_DIR:PATH=/usr/tce/packages/spectrum-mpi/spectrum-mpi-2019.06.24-gcc-7.2.1-redhat
-DTPL_ENABLE_ParMETIS:BOOL=ON
-DParMETIS_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/parmetis-4.0.3-bygkh2pki56aakaqsjitfbq6rnmop26q/lib
-DParMETIS_LIBRARY_NAMES=parmetis;metis
-DParMETIS_INCLUDE_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/parmetis-4.0.3-bygkh2pki56aakaqsjitfbq6rnmop26q/include
-DTPL_ENABLE_Pthread:BOOL=OFF
-DSuperLU_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/superlu-5.2.1-mubeuixj7xzeqdjaeporkzmvhusykhud/lib
-DSuperLU_INCLUDE_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/superlu-5.2.1-mubeuixj7xzeqdjaeporkzmvhusykhud/include
-DTPL_ENABLE_yaml-cpp:BOOL=OFF
-DTPL_ENABLE_y12m:BOOL=ON
-Dy12m_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/y12m-1.0.0-4gmum67kdzctzpaylpjr4fs7lpw6xris/lib
-DTPL_ENABLE_UMFPACK:BOOL=ON
-DUMFPACK_LIBRARY_DIRS:PATH=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/umfpack-5.1.0-gmm3ukxqgonnwdxyckbrbnrrmrckivoj/lib
-DUMFPACK_INCLUDE_DIRS:PATH=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/umfpack-5.1.0-gmm3ukxqgonnwdxyckbrbnrrmrckivoj/include
-DUMFPACK_LIBRARY_NAMES=umfpack;amd
-DTPL_ENABLE_Scotch:BOOL=ON
-DScotch_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/scotch-6.0.4-d7v6yy5ptuv3g2ixrq6aihhauiswxcn4/lib
-DScotch_INCLUDE_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/scotch-6.0.4-d7v6yy5ptuv3g2ixrq6aihhauiswxcn4/include
-DTPL_ENABLE_CUSPARSE:BOOL=ON
-DKokkos_ENABLE_CUDA_UVM:BOOL=ON
-DKokkos_ENABLE_CUDA_LAMBDA:BOOL=ON
-DKokkos_ENABLE_CUDA_CONSTEXPR:BOOL=ON
-DTpetra_ASSUME_CUDA_AWARE_MPI:BOOL=OFF
-DKokkos_ARCH_VOLTA70:BOOL=ON
-DTrilinos_ENABLE_Amesos:BOOL=ON
-DTrilinos_ENABLE_Amesos2:BOOL=ON
-DTrilinos_ENABLE_Anasazi:BOOL=ON
-DTrilinos_ENABLE_AztecOO:BOOL=ON
-DTrilinos_ENABLE_Belos:BOOL=ON
-DTrilinos_ENABLE_Epetra:BOOL=ON
-DTrilinos_ENABLE_EpetraExt:BOOL=ON
-DTrilinos_ENABLE_FEI:BOOL=ON
-DTrilinos_ENABLE_Galeri:BOOL=OFF
-DTrilinos_ENABLE_Gtest:BOOL=OFF
-DTrilinos_ENABLE_Ifpack:BOOL=ON
-DTrilinos_ENABLE_Ifpack2:BOOL=ON
-DTrilinos_ENABLE_Intrepid:BOOL=ON
-DTrilinos_ENABLE_Intrepid2:BOOL=ON
-DTrilinos_ENABLE_Isorropia:BOOL=OFF
-DTrilinos_ENABLE_Kokkos:BOOL=ON
-DTrilinos_ENABLE_KokkosCore:BOOL=ON
-DTrilinos_ENABLE_KokkosContainers:BOOL=ON
-DTrilinos_ENABLE_MiniTensor:BOOL=ON
-DTrilinos_ENABLE_ML:BOOL=ON
-DTrilinos_ENABLE_MOOCHO:BOOL=OFF
-DTrilinos_ENABLE_MueLu:BOOL=ON
-DTrilinos_ENABLE_OpenMP:BOOL=OFF
-DTrilinos_ENABLE_Pamgen:BOOL=ON
-DTrilinos_ENABLE_ROL:BOOL=ON
-DTrilinos_ENABLE_RTOp:BOOL=ON
-DTrilinos_ENABLE_Rythmos:BOOL=OFF
-DTrilinos_ENABLE_Sacado:BOOL=ON
-DTrilinos_ENABLE_SEACAS:BOOL=OFF
-DTrilinos_ENABLE_Shards:BOOL=ON
-DTrilinos_ENABLE_ShyLU_NodeTacho:BOOL=ON
-DTrilinos_ENABLE_ShyLU_NodeHTS:BOOL=ON
-DTrilinos_ENABLE_STK:BOOL=OFF
-DTrilinos_ENABLE_Stokhos:BOOL=OFF
-DTrilinos_ENABLE_SuperLU5_API:BOOL=ON
-DTrilinos_ENABLE_Teuchos:BOOL=ON
-DTrilinos_ENABLE_Teko:BOOL=ON
-DTrilinos_ENABLE_ThreadPool:BOOL=ON
-DTrilinos_ENABLE_Thyra:BOOL=ON
-DTrilinos_ENABLE_ThyraEpetraExtAdapters:BOOL=ON
-DTrilinos_ENABLE_Tpetra:BOOL=ON
-DTrilinos_ENABLE_TrilinosCouplings:BOOL=OFF
-DTrilinos_ENABLE_Triutils:BOOL=ON
-DTrilinos_ENABLE_Xpetra:BOOL=ON
-DTrilinos_ENABLE_Zoltan:BOOL=ON
-DTrilinos_ENABLE_Zoltan2:BOOL=ON
-DTrilinos_ENABLE_ShyLU_NodeFastILU:BOOL=ON
-DTrilinos_ENABLE_PanzerExprEval:BOOL=ON
-DAmesos_ENABLE_SuperLU:BOOL=ON
-DAmesos2_ENABLE_ShyLU_NodeTacho:BOOL=ON
-DAmesos2_ENABLE_SuperLU:BOOL=ON
-DAztecOO_ENABLE_AZLU:BOOL=ON
-DBelos_ENABLE_Xpetra:BOOL=OFF
-DIfpack2_ENABLE_ThyraTpetraAdapters=ON
-DKokkos_ENABLE_CUDA:BOOL=ON
-DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE:BOOL=ON
-DKokkos_ENABLE_DEPRECATED_CODE:BOOL=OFF
-DKokkos_ENABLE_OPENMP:BOOL=OFF
-DKokkos_ENABLE_PROFILING:BOOL=ON
-DKokkosKernels_ENABLE_SUPERNODAL_SPTRSV=OFF
-DML_ENABLE_METIS:BOOL=OFF
-DML_ENABLE_ParMETIS:BOOL=OFF
-DPamgen_ENABLE_Boost:BOOL=OFF
-DPamgen_ENABLE_Extras:BOOL=OFF
-DROL_ENABLE_EXAMPLES:BOOL=OFF
-DROL_ENABLE_TESTS:BOOL=OFF
-DSacado_ENABLE_KokkosCore:BOOL=ON
-DTacho_ENABLE_INT_INT:BOOL=ON
-DTpetra_INST_INT_LONG:BOOL=ON
-DTpetra_INST_INT_LONG_LONG:BOOL=OFF
-DTpetra_INST_OPENMP:BOOL=OFF
-DTpetra_INST_SERIAL:BOOL=ON
-DZoltan_ENABLE_METIS:BOOL=OFF
-DZoltan_ENABLE_Scotch:BOOL=OFF
-DZoltan2_ENABLE_Experimental:BOOL=ON
-DIfpack2_ENABLE_ShyLU_NodeFastILU:BOOL=ON
-DAmesos2_ENABLE_UMFPACK:BOOL=ON
-DZoltan2_ENABLE_Scotch:BOOL=ON
bartlettroscoe commented 2 years ago

What is the exact link error you are seeing? What *.o file in Trilinos is this coming from? From what I am looking at, you should be getting std::complex<float> instantiations from these Teuchos macros.

For more detail about the status of this, see below ...

So it seems that the configure options:

exist at the global Trilinos level but only the configure options:

exist at the Teuchos level as seen at:

https://github.com/trilinos/Trilinos/blob/b9239cd636ef68fcb80fb659945c94cefc0400dc/packages/teuchos/CMakeLists.txt#L152-L160

It seems that no one ever duplicated those finer-grained Trilinos options at the Teuchos level with the todo shown at:

https://github.com/trilinos/Trilinos/blob/b9239cd636ef68fcb80fb659945c94cefc0400dc/packages/teuchos/CMakeLists.txt#L169-L171

which references https://github.com/trilinos/Trilinos/issues/362 (see the full discussion there). (I suspect I wrote that "ToDo" over 6 years aog.)

So at the Teuchos level, it assumes that if you want "float" and "complex", you will get float, std::complex<double> and std::complex<float>. And the Teuchos ETI macros seems to enforce that so you should not be seeing link errors (just extra std::complex<float> instantiations you may not want) as shown at:

https://github.com/trilinos/Trilinos/blob/b9239cd636ef68fcb80fb659945c94cefc0400dc/packages/teuchos/core/src/Teuchos_ExplicitInstantiationHelpers.hpp#L82-L90

We need to see the exact link errors to decide how to move forward.

vbrunini commented 2 years ago

The link errors are references to instantiations of Tpetra classes for std::complex<float> which Tpetra did not instantiate (as desired). For example:

/fgs/vebruni/src/code_clean/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_pat/linux-rhel7-x86_64/gcc-8.3.0/trilinos-2022.06.17.03-lq4lfl5k6staichygb35tieg4d2wg7h2/lib/libstratimikosbelos.a(Thyra_BelosLinearOpWithSolve.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<std::complex<float>, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeDenseHeader(std::ostream&, Tpetra::MultiVector<std::complex<float>, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&)':
tmpxft_0000f09d_00000000-6_Thyra_BelosLinearOpWithSolve.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixISt7complexIfEilN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIS4_ilSA_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESP_RKN7Teuchos3RCPINSQ_18basic_FancyOStreamIcSL_EEEESW_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixISt7complexIfEilN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIS4_ilSA_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESP_RKN7Teuchos3RCPINSQ_18basic_FancyOStreamIcSL_EEEESW_]+0x3bd): undefined reference to `Tpetra::MultiVector<std::complex<float>, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getGlobalLength() const'
tmpxft_0000f09d_00000000-6_Thyra_BelosLinearOpWithSolve.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixISt7complexIfEilN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIS4_ilSA_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESP_RKN7Teuchos3RCPINSQ_18basic_FancyOStreamIcSL_EEEESW_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixISt7complexIfEilN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIS4_ilSA_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESP_RKN7Teuchos3RCPINSQ_18basic_FancyOStreamIcSL_EEEESW_]+0x3e5): undefined reference to `Tpetra::MultiVector<std::complex<float>, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getNumVectors() const'

There are a bunch more, but all appear to be from files in Thyra missing references to things in Tpetra.

bartlettroscoe commented 2 years ago

There are a bunch more, but all appear to be from files in Thyra missing references to things in Tpetra.

Okay, that makes sense. That means that I can likely reproduce this locally just with the Trilinos test suite. (Which means that there are no PR builds that test this combination of configure options.)

The fix is a little more tricky if we want to maintain backward compatibility. But it just so happens that we are allowed to break backward compatibility until Trilinos 14.0 is released in the next 3 months or so.

This is the fix I propose:

1) Add the cache vars Teuchos_ENABLE_COMPLEX_DOUBLE and Teuchos_ENABLE_COMPLEX_FLOAT which map to #defines HAVE_TEUCHOS_COMPLEX_DOUBLE and HAVE_TEUCHOS_COMPLEX_FLOAT, respectively, in the configured Teuchos_config.h file.

2) Set the defaults for Teuchos_ENABLE_COMPLEX_DOUBLE and Teuchos_ENABLE_COMPLEX_FLOAT to match the same logic for the global options here.

3) Upgrade the macros in Teuchos_ExplicitInstantiationHelpers.hpp to key off of HAVE_TEUCHOS_COMPLEX_FLOAT and HAVE_TEUCHOS_COMPLEX_DOUBLE instead of HAVE_TEUCHOS_COMPLEX and Trilinos_ENABLE_COMPLEX.

The above has the following advantages:

  1. It should give consistent logic and ETI macros for Teuchos and Tpetra for all settings of the Trilinos-level configure options Trilinos_ENABLE_FLOAT, Trilinos_ENABLE_COMPLEX, Trilinos_ENABLE_COMPLEX_FLOAT and Trilinos_ENABLE_COMPLEX_DOUBLE.
  2. It maintains backward compatibility for customers setting Trilinos_ENABLE_FLOAT=ON and Trilinos_ENABLE_COMPLEX=ON in that it will enable std::complex<float> support (and ETI instantiations).
  3. Setting Trilinos_ENABLE_FLOAT=ON and Trilinos_ENABLE_COMPLEX_DOUBLE=ON will only give ETI instantiations for float, double and std::complex<double> but not std::complex<float>.
bartlettroscoe commented 2 years ago

@e10harvey, did you mention some issue with Teuchos related to the Trilinos 'develop' to 'master' promotion? Was it an ETI issue like this?

bartlettroscoe commented 2 years ago

I think this issue has been latent in Teuchos for a long time, but is now impacting us after commit https://github.com/trilinos/Trilinos/commit/e08b53677de4ac72b9e060001dd5e63b6c2b167f which started using these Teuchos macros in stratimikos.

I think the problem is not so much that this is a defect in Teuchos, it is that a package downstream from Tpetra used the Teuchos ETI macros to do the instantations. Perhaps the Stratimikos commit https://github.com/trilinos/Trilinos/commit/e08b53677de4ac72b9e060001dd5e63b6c2b167f should have used Tpetra ETI macros instead of the Teuchos ETI macros?

But in any case, Teuchos and Tpetra should have identical logic for what instantiations to produce based on the top-level options Trilinos_ENABLE_FLOAT, Trilinos_ENABLE_COMPLEX, Trilinos_ENABLE_COMPLEX_FLOAT and Trilinos_ENABLE_COMPLEX_DOUBLE so the change described above should be made in any case.

How urgent is the fix of this issues?

e10harvey commented 2 years ago

@bartlettroscoe: No - it was a link time problem with abs in intel-19. We should have filed a bug report for it but here is the comment showing the error: https://github.com/trilinos/Trilinos/pull/10486#issuecomment-1116083746.

bartlettroscoe commented 2 years ago

@bartlettroscoe: No - it was a link time problem with abs in intel-19. We should have filed a bug report for it but here is the comment showing the error: #10486 (comment).

Okay, that means that there is not even a 'develop' to 'master' build for this configuration Trilinos_ENABLE_FLOAT=ON and Trilinos_ENABLE_COMPLEX_DOUBLE=ON that would have caught this bug before it was synced/deployed to 'master'. Given that Sierra is an important Trilinos customer, I will put in a request to add/update and 'develop' to 'master' build to cover this.

vbrunini commented 2 years ago

I'd say fairly urgent. The issue will block integrating any version of Trilinos after the stratimikos change into Sierra, and we're going to need to do that within the next month or so for milestone work. If necessary we can apply a patch to our spack builds as a workaround but the shorter lived patches are the better, and not needing to would be ideal.

bartlettroscoe commented 2 years ago

I'd say fairly urgent.

Okay, I will try to reproduce and address this later this week.

jclause commented 2 years ago

Yes, I've been trying to clean up bugs so we can do another integration, and this bug is blocking that effort.

bartlettroscoe commented 2 years ago

Sorry for the delay. I will work on this tomorrow unless a (figurative) bomb goes off.

bartlettroscoe commented 2 years ago

FYI: I have reproduced the build error and am working on a solution along the lines above. However, there a lot of code (in 117 files) is keying off of HAVE_TEUCHOS_COMPLEX as shown in the following find-grep operation.

All Trilinos files that mention HAVE_TEUCHOS_COMPLEX (click to expand) ``` $ find packages -type f -exec grep -l HAVE_TEUCHOS_COMPLEX {} \; | wc -l 117 ``` and ``` $ find packages -type f -exec grep -l HAVE_TEUCHOS_COMPLEX {} \; packages/tpetra/core/test/MultiVector/MultiVector_UnitTests.cpp packages/tpetra/core/test/CrsMatrix/CrsMatrix_NonlocalSumInto.cpp packages/tpetra/core/test/CrsMatrix/CrsMatrix_NonlocalSumInto_Ignore.cpp packages/tpetra/core/test/BasicPerfTest/BasicPerfTest.cpp packages/tpetra/core/inout/Tpetra_Details_ReadTriples.hpp packages/amesos2/test/adapters/Tpetra_MultiVector_Adapter_UnitTests.cpp packages/amesos2/test/adapters/Tpetra_CrsMatrix_Adapter_UnitTests.cpp packages/amesos2/test/solvers/MUMPS_UnitTests.cpp packages/amesos2/test/solvers/Superlu_Solve_Tests.cpp packages/amesos2/test/solvers/Superlu_UnitTests.cpp packages/amesos2/test/solvers/Basker_UnitTests.cpp packages/amesos2/test/solvers/STRUMPACK_UnitTests.cpp packages/amesos2/test/solvers/KLU2_UnitTests.cpp packages/amesos2/test/solvers/Solver_Test.cpp packages/amesos2/example/NewSolver/Amesos2_NewSolver_TypeMap.hpp packages/amesos2/example/NewSolver/Amesos2_NewSolver_FunctionMap.hpp packages/amesos2/src/Amesos2_PardisoMKL_decl.hpp packages/amesos2/src/Amesos2_Superlu_TypeMap.hpp packages/amesos2/src/basker/basker_scalartraits.hpp packages/amesos2/src/KLU2/Include/klu2_scalartraits.h packages/amesos2/src/KLU2/Source/examples/klu2_simple.cpp packages/amesos2/src/Amesos2_ShyLUBasker_FunctionMap.hpp packages/amesos2/src/Amesos2_cuSOLVER_FunctionMap.hpp packages/amesos2/src/Amesos2_TpetraMultiVecAdapter_def.hpp packages/amesos2/src/Amesos2_Superludist_decl.hpp packages/amesos2/src/Amesos2_KLU2_decl.hpp packages/amesos2/src/Amesos2_Superlumt_decl.hpp packages/amesos2/src/Amesos2_STRUMPACK_decl.hpp packages/amesos2/src/Amesos2_Cholmod_decl.hpp packages/amesos2/src/Amesos2_ShyLUBasker_TypeMap.cpp packages/amesos2/src/Amesos2_Basker_decl.hpp packages/amesos2/src/Amesos2_MUMPS_decl.hpp packages/amesos2/src/Amesos2_Umfpack_decl.hpp packages/amesos2/src/Amesos2_Superlu_TypeMap.cpp packages/amesos2/src/Amesos2_cuSOLVER_decl.hpp packages/amesos2/src/Amesos2_Superludist_TypeMap.hpp packages/amesos2/src/Amesos2_KLU2_TypeMap.hpp packages/amesos2/src/Amesos2_MUMPS_FunctionMap.hpp packages/amesos2/src/Amesos2_Superlumt_TypeMap.hpp packages/amesos2/src/Amesos2_PardisoMKL_FunctionMap.hpp packages/amesos2/src/Amesos2_PardisoMKL_TypeMap.hpp packages/amesos2/src/Amesos2_ShyLUBasker_TypeMap.hpp packages/amesos2/src/Amesos2_Basker_FunctionMap.hpp packages/amesos2/src/Amesos2_Tacho_TypeMap.hpp packages/amesos2/src/Amesos2_Tacho_decl.hpp packages/amesos2/src/Amesos2_Cholmod_FunctionMap.hpp packages/amesos2/src/Amesos2_Superlu_decl.hpp packages/amesos2/src/Amesos2_cuSOLVER_TypeMap.hpp packages/amesos2/src/Amesos2_Superlumt_FunctionMap.hpp packages/amesos2/src/Amesos2_Basker_TypeMap.hpp packages/amesos2/src/Amesos2_MUMPS_TypeMap.cpp packages/amesos2/src/Amesos2_KLU2_FunctionMap.hpp packages/amesos2/src/Amesos2_Umfpack_TypeMap.hpp packages/amesos2/src/Amesos2_Lapack_decl.hpp packages/amesos2/src/Amesos2_Util.hpp packages/amesos2/src/Amesos2_Superludist_TypeMap.cpp packages/amesos2/src/Amesos2_Superludist_FunctionMap.hpp packages/amesos2/src/Amesos2_Superlu_FunctionMap.hpp packages/amesos2/src/Amesos2_Umfpack_FunctionMap.hpp packages/amesos2/src/Amesos2_Tacho_FunctionMap.hpp packages/amesos2/src/Amesos2_Basker_TypeMap.cpp packages/amesos2/src/Amesos2_Superlumt_TypeMap.cpp packages/amesos2/src/Amesos2_MUMPS_TypeMap.hpp packages/amesos2/src/Amesos2_ShyLUBasker_decl.hpp packages/amesos2/src/Amesos2_Cholmod_TypeMap.hpp packages/amesos2/src/Amesos2_ShyLUBasker_def.hpp packages/ifpack/src/Ifpack_Polynomial.cpp packages/teuchos/numerics/test/DenseMatrix/cxx_main_spd_solver.cpp packages/teuchos/numerics/test/DenseMatrix/cxx_main_solver.cpp packages/teuchos/numerics/test/DenseMatrix/cxx_main_band_solver.cpp packages/teuchos/numerics/test/DenseMatrix/cxx_main_qr_solver.cpp packages/teuchos/numerics/test/BLAS/cxx_tmpl_main_comp.cpp packages/teuchos/numerics/test/BLAS/cxx_tmpl_main.cpp packages/teuchos/numerics/src/Teuchos_MatrixMarket_assignScalar.hpp packages/teuchos/numerics/src/Teuchos_BLAS_wrappers.hpp packages/teuchos/numerics/src/Teuchos_ExpandScalarTypeMacros.hpp packages/teuchos/numerics/src/Teuchos_LAPACK.hpp packages/teuchos/numerics/src/Teuchos_MatrixMarket_CoordDataReader.hpp packages/teuchos/numerics/src/Teuchos_PromotionTraits.hpp packages/teuchos/numerics/src/Teuchos_LAPACK.cpp packages/teuchos/numerics/src/Teuchos_BLAS.cpp packages/teuchos/numerics/src/Teuchos_LAPACK_wrappers.hpp packages/teuchos/numerics/src/Teuchos_MatrixMarket_generic.hpp packages/teuchos/CMakeLists.txt packages/teuchos/core/cmake/Teuchos_config.h.in packages/teuchos/core/test/TypeConversions/TypeConversions_UnitTest.cpp packages/teuchos/core/test/ScalarTraits/ScalarTraits_test.cpp packages/teuchos/core/src/Teuchos_ExplicitInstantiationHelpers.hpp packages/teuchos/core/src/Teuchos_TypeNameTraits.hpp packages/teuchos/core/src/Teuchos_ScalarTraits.hpp packages/teuchos/core/src/Teuchos_as.hpp packages/teuchos/core/src/Teuchos_UnitTestHelpers.hpp packages/teuchos/comm/test/MpiTypeTraits/mpiTypeTraits.cpp packages/teuchos/comm/test/Comm/DefaultMpiComm_UnitTests.cpp packages/teuchos/comm/test/Comm/Comm_test.cpp packages/teuchos/comm/src/Teuchos_SerializationTraits.hpp packages/teuchos/comm/src/Teuchos_Details_MpiTypeTraits.hpp packages/nox/src-loca/src-lapack/LOCA_LAPACK_Group.C packages/nox/src-loca/src-lapack/LOCA_BorderedSolver_LAPACKDirectSolve.H packages/nox/src-loca/src-lapack/LOCA_LAPACK_Group.H packages/nox/src-loca/src-lapack/LOCA_BorderedSolver_LAPACKDirectSolve.C packages/rtop/test/base/PrimitiveTypeTraits_UnitTests.cpp packages/rtop/test/lapack/testLapackWrappers.cpp packages/rtop/src/interfaces/RTOpPack_Types.hpp packages/shylu/shylu_node/basker/src/shylubasker_scalar_traits.hpp packages/anasazi/cmake/Anasazi_config.h.in packages/anasazi/src/AnasaziHelperTraits.hpp packages/ifpack2/test/unit_tests/Ifpack2_UnitTestBlockCrsUtil.hpp packages/ifpack2/test/unit_tests/Ifpack2_UnitTestBlockTriDiContainer.cpp packages/ifpack2/src/Ifpack2_Details_LapackSupportsScalar.hpp packages/belos/tpetra/test/OrthoManager/belos_orthomanager_tpetra.cpp packages/belos/cmake/Belos_config.h.in packages/belos/src/Belos_Details_registerSolverFactory.cpp packages/belos/src/BelosBlockCGSolMgr.hpp packages/belos/src/BelosSolverFactory_Belos.hpp packages/belos/src/BelosSolverManager.hpp packages/belos/src/BelosGCRODRSolMgr.hpp ```

This is why no one pulled the trigger on this refactoring way back in #362.

With having to edit all 117 of those files and having to test all of that, this could take some time. (And if we keep HAVE_TEUCHOS_COMPLEX in addition to the new defines HAVE_TEUCHOS_COMPLEX_FLOAT and HAVE_TEUCHOS_COMPLEX_COMPLEX, then what does HAVE_TEUCHOS_COMPLEX mean at that point?)

bartlettroscoe commented 2 years ago

This is even harder than I thought. In addition to C++ files referencing the define HAVE_TEUCHOS_COMPLEX, there are also a total of 37 files spanning 11 packages mentioning Teuchos_ENABLE_COMPLEX as shown below.

All Trilinos files that mention Teuchos_ENABLE_COMPLEX
The total number of files mentioning `Teuchos_ENABLE_COMPLEX`: ``` $ find packages -type f -exec grep -l "Teuchos_ENABLE_COMPLEX" {} \; | wc -l 37 ``` and the full list of files mentioning `Teuchos_ENABLE_COMPLEX`: ``` $ find packages -type f -exec grep -l "Teuchos_ENABLE_COMPLEX" {} \; packages/tpetra/CMakeLists.txt packages/tpetra/tsqr/CMakeLists.txt packages/stratimikos/test/CMakeLists.txt packages/amesos2/src/KLU2/Include/klu2_scalartraits.h packages/thyra/CMakeLists.txt packages/panzer/maintenance/build_panzer_gcc_mac_s1002730.sh packages/muelu/CMakeLists.txt packages/teuchos/numerics/src/Teuchos_BLAS.hpp packages/teuchos/numerics/src/Teuchos_LAPACK.hpp packages/teuchos/CMakeLists.txt packages/teuchos/core/src/Teuchos_ScalarTraitsDecl.hpp packages/teuchos/doc/index.doc packages/nox/src-loca/src-lapack/LOCA_LAPACK_Group.C packages/nox/src-loca/src-lapack/LOCA_BorderedSolver_LAPACKDirectSolve.C packages/nox/examples/lapack/LOCA_Brusselator/CMakeLists.txt packages/nox/src/NOX_Description.H packages/shylu/shylu_node/hts/src/CMakeLists.txt packages/anasazi/CMakeLists.txt packages/anasazi/test/IRTR/CMakeLists.txt packages/anasazi/test/BlockDavidson/CMakeLists.txt packages/anasazi/test/BlockKrylovSchur/CMakeLists.txt packages/anasazi/test/LOBPCG/CMakeLists.txt packages/anasazi/test/MVOPTester/CMakeLists.txt packages/anasazi/doc/TSQR/test/Tsqr_TestCombine.txt packages/anasazi/doc/TSQR/test/Tsqr_TestSeqTsqr.txt packages/anasazi/doc/TSQR/test/Tsqr_TestDistTsqr.txt packages/framework/GenConfig/tests/supporting_files/test-config-specs-invalid-option-in-build-name-raises.ini packages/framework/ini-files/config-specs.ini packages/belos/test/BlockGmres/CMakeLists.txt packages/belos/test/RealSolverManager/CMakeLists.txt packages/belos/test/BlockCG/CMakeLists.txt packages/belos/test/TFQMR/CMakeLists.txt packages/belos/test/GCRODR/CMakeLists.txt packages/belos/test/MVOPTester/CMakeLists.txt packages/belos/test/MINRES/CMakeLists.txt packages/belos/src/BelosBlockCGSolMgr.hpp packages/belos/src/BelosGCRODRSolMgr.hpp ```

In total, there are 147 files mentioning HAVE_TEUCHOS_COMPLEX or Teuchos_ENABLE_COMPLEX as shown in:

$ find packages -type f -exec grep -l "\(Teuchos_ENABLE_COMPLEX\|HAVE_TEUCHOS_COMPLEX\)" {} \; | wc -l
147

The strategy of outlined above would take several days of detailed code inspection, code change, and testing. Also that change would undoubtedly break some existing Trilinos customers that may be keying off of at least the define HAVE_TEUCHOS_COMPLEX in the configured header file Teuchos_config.h.

Therefore, instead of refactoring Teuchos in the way described above, we should instead copy the file Teuchos_ExplicitInstantiationHelpers.hpp over to Tpetra_ExplicitInstantiationHelpers.hpp and update the macros to key off of the appropriate Tpetra CMake variables like Tpetra_INST_COMPLEX_FLOAT and Tpetra_INST_COMPLEX_DOUBLE. That way, we can fix the instantiation problem reported in this issue without requiring large code changes and lots of testing and debugging.

Basically, the current Teuchos logic assumes that if Teuchos_ENABLE_COMPLEX=TRUE and Teuchos_ENABLE_FLOAT=TRUE, then you get support for complex<float> enabled (in Teuchos) and you can't turn that off if those two vars are enabled. But since Teuchos support for complex needs to be a superset of what is required downstream, that should be fine.

bartlettroscoe commented 2 years ago

Giving the reproduction info I used to produced the same link errors reported above. Basically, you just configure Trilinos with:

-DTrilinos_ENABLE_TESTS=ON \
-D Trilinos_ENABLE_Stratimikos=ON \
-D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \
-D Trilinos_ENABLE_FLOAT=ON \

and you get the link errors.

Local reproduction of Stratimikos link errors (click to expand)
I was able to generate the build error for the Trilinos version: ``` $ ssh crf450 $ cd ~/Trilinos.base2/Trilinos/ $ git log-short --graph --name-status * a0654795305 "Merge pull request #10698 from hkthorn/develop" |\ Author: Heidi Thornquist | | Date: Tue Jul 5 09:27:38 2022 -0600 (24 hours ago) | | | * 7aab16eac78 "Removes unused Failure classes inheriting from Belos::Error" | | Author: Heidi Thornquist | | Date: Fri Jul 1 16:56:01 2022 -0600 (5 days ago) ``` with: ``` $ ssh crf450 $ cd ~/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/ $ cat load-env.sh . ~/Trilinos.base2/Trilinos/cmake/load_sems_dev_env.sh sems-archive-gcc/7.2.0 $ cat do-configure #!/bin/bash if [ -e CMakeCache.txt ] ; then echo "Removing CMake files" rm -r CMake* fi cmake \ -GNinja \ -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/sems/SEMSDevEnv.cmake \ -DBUILD_SHARED_LIBS=ON \ -DCMAKE_BUILD_TYPE=RELEASE \ -DTPL_ENABLE_MPI=ON \ -DTrilinos_ENABLE_TESTS=ON \ -DTrilinos_ENABLE_BUILD_STATS=ON \ "$@" \ ~/Trilinos.base2/Trilinos $ cat do-configure.stratimikos-complex-double-float #!/bin/bash ./do-configure \ -D Trilinos_ENABLE_Stratimikos=ON \ -D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \ -D Trilinos_ENABLE_FLOAT=ON \ "$@" $ . load-env.sh $ module list Currently Loaded Modules: 1) sems-archive-env 5) sems-archive-git/2.10.1 9) sems-archive-zlib/1.2.8/base 13) sems-archive-scotch/6.0.3/nopthread_64bit_parallel 2) sems-archive-python/2.7.9 6) sems-archive-gcc/7.2.0 10) sems-archive-hdf5/1.10.6/parallel 14) sems-archive-superlu/4.3/base 3) sems-archive-cmake/3.17.1 7) sems-archive-openmpi/1.10.1 11) sems-archive-netcdf/4.7.3/parallel 4) sems-archive-ninja_fortran/1.8.2 8) sems-archive-boost/1.63.0/base 12) sems-archive-parmetis/4.0.3/parallel $ rm CMake* *.out $ time ./do-configure.stratimikos-complex-double-float &> configure.out real 0m19.170s user 0m10.513s sys 0m4.943s $ time make -j16 &> make.out real 90m3.233s user 112m50.366s sys 67m5.553s ``` That produced the build error: ``` FAILED: packages/stratimikos/adapters/amesos2/test/Stratimikos_test_single_amesos2_tpetra_solver_driver.exe : && /ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/build_stat_cxx_wrapper.sh -pedantic -Wall -Wno-long-long -Wwrite-strings -O3 -DNDEBUG packages/stratimikos/adapters/amesos2/test/CMakeFiles/Stratimikos_test_single_amesos2_tpetra_solver_driver.dir/test_single_amesos2_tpetra_solver_driver.cpp.o packages/stratimikos/adapters/amesos2/test/CMakeFiles/Stratimikos_test_single_amesos2_tpetra_solver_driver.dir/test_single_amesos2_tpetra_solver.cpp.o -o packages/stratimikos/adapters/amesos2/test/Stratimikos_test_single_amesos2_tpetra_solver_driver.exe -Wl,-rpath,/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/stratimikos/adapters/amesos2/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/stratimikos/adapters/aztecoo/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/stratimikos/adapters/amesos/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/stratimikos/adapters/ml/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/stratimikos/adapters/ifpack/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/ifpack2/adapters:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/ifpack2/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/ml/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/galeri/src-xpetra:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/galeri/src-epetra:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/isorropia/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/ifpack/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/amesos/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/amesos2/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/shylu/shylu_node/tacho/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/common/auxiliarySoftware/SuiteSparse/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/belos/xpetra/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/belos/tpetra/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/belos/epetra/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/belos/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/aztecoo/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/shylu/shylu_node/hts/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/zoltan2/core/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/zoltan/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/xpetra/sup:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/xpetra/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/thyra/adapters/tpetra/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/thyra/adapters/epetraext/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/thyra/adapters/epetra/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/thyra/core/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/rtop/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/epetraext/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/triutils/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/tpetra/core/ext:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/tpetra/core/inout:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/tpetra/core/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/tpetra/core/compat:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/epetra/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/tpetra/tsqr/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/kokkos-kernels/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/kokkos/algorithms/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/kokkos/containers/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/teuchos/kokkoscomm/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/teuchos/kokkoscompat/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/teuchos/remainder/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/teuchos/numerics/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/teuchos/comm/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/teuchos/parameterlist/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/teuchos/parser/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/teuchos/core/src:/ascldap/users/rabartl/Trilinos.base2/BUILDS/GCC-7.2.0/gnu-shared-opt/packages/kokkos/core/src packages/stratimikos/adapters/amesos2/src/libstratimikosamesos2.so.13.5 packages/stratimikos/adapters/aztecoo/src/libstratimikosaztecoo.so.13.5 packages/stratimikos/adapters/amesos/src/libstratimikosamesos.so.13.5 packages/stratimikos/adapters/ml/src/libstratimikosml.so.13.5 packages/stratimikos/adapters/ifpack/src/libstratimikosifpack.so.13.5 packages/ifpack2/adapters/libifpack2-adapters.so.13.5 packages/ifpack2/src/libifpack2.so.13.5 liblast_lib.a packages/ml/src/libml.so.13.5 packages/galeri/src-xpetra/libgaleri-xpetra.so.13.5 packages/galeri/src-epetra/libgaleri-epetra.so.13.5 packages/isorropia/src/libisorropia.so.13.5 packages/ifpack/src/libifpack.so.13.5 packages/amesos/src/libamesos.so.13.5 packages/amesos2/src/libamesos2.so.13.5 packages/shylu/shylu_node/tacho/src/libtacho.so.13.5 packages/common/auxiliarySoftware/SuiteSparse/src/libtrilinosss.so.13.5 packages/belos/xpetra/src/libbelosxpetra.so.13.5 packages/belos/tpetra/src/libbelostpetra.so.13.5 packages/belos/epetra/src/libbelosepetra.so.13.5 packages/belos/src/libbelos.so.13.5 packages/aztecoo/src/libaztecoo.so.13.5 packages/shylu/shylu_node/hts/src/libshylu_nodehts.so.13.5 packages/zoltan2/core/src/libzoltan2.so.13.5 packages/zoltan/src/libzoltan.so.13.5 -lm packages/xpetra/sup/libxpetra-sup.so.13.5 packages/xpetra/src/libxpetra.so.13.5 packages/thyra/adapters/tpetra/src/libthyratpetra.so.13.5 packages/thyra/adapters/epetraext/src/libthyraepetraext.so.13.5 packages/thyra/adapters/epetra/src/libthyraepetra.so.13.5 packages/thyra/core/src/libthyracore.so.13.5 packages/rtop/src/librtop.so.13.5 packages/epetraext/src/libepetraext.so.13.5 packages/triutils/src/libtriutils.so.13.5 packages/tpetra/core/ext/libtpetraext.so.13.5 packages/tpetra/core/inout/libtpetrainout.so.13.5 packages/tpetra/core/src/libtpetra.so.13.5 packages/tpetra/core/compat/libtpetraclassic.so.13.5 packages/epetra/src/libepetra.so.13.5 packages/tpetra/tsqr/src/libkokkostsqr.so.13.5 packages/kokkos-kernels/src/libkokkoskernels.so.13.5 packages/kokkos/algorithms/src/libkokkosalgorithms.so.13.5 packages/kokkos/containers/src/libkokkoscontainers.so.13.5 packages/teuchos/kokkoscomm/src/libteuchoskokkoscomm.so.13.5 packages/teuchos/kokkoscompat/src/libteuchoskokkoscompat.so.13.5 packages/teuchos/remainder/src/libteuchosremainder.so.13.5 packages/teuchos/numerics/src/libteuchosnumerics.so.13.5 /usr/lib64/liblapack.so /usr/lib64/libblas.so packages/teuchos/comm/src/libteuchoscomm.so.13.5 packages/teuchos/parameterlist/src/libteuchosparameterlist.so.13.5 packages/teuchos/parser/src/libteuchosparser.so.13.5 packages/teuchos/core/src/libteuchoscore.so.13.5 packages/kokkos/core/src/libkokkoscore.so.13.5 /usr/lib64/libdl.so -lgomp -lgfortran -ldl && : /usr/bin/ld: warning: libgfortran.so.3, needed by /usr/lib64/liblapack.so, may conflict with libgfortran.so.4 packages/stratimikos/adapters/amesos2/src/libstratimikosamesos2.so.13.5: undefined reference to `Amesos2::TachoSolver, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode >, Tpetra::MultiVector, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode > >::TachoSolver(Teuchos::RCP, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode > const>, Teuchos::RCP, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode > >, Teuchos::RCP, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode > const>)' packages/stratimikos/adapters/amesos2/src/libstratimikosamesos2.so.13.5: undefined reference to `Amesos2::KLU2, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode >, Tpetra::MultiVector, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode > >::KLU2(Teuchos::RCP, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode > const>, Teuchos::RCP, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode > >, Teuchos::RCP, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode > const>)' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. make: *** [all] Error 1 ```
bartlettroscoe commented 2 years ago

FYI: PR #10720 contains the changes that should fix the build issue for Sierra. I am not ready to merge yet as I am fixing the Trilinos test suite for this use case.

NOTE: It appears there are not automated Trilinos builds that enable float! That is a huge hole in the testing coverage for Trilinos given that Sierra requires float. I will post an issue about that.

bartlettroscoe commented 2 years ago

FYI: I created the GitHub Issue #10742 and the internal issue:

for the gap in Trilinos testing that let this defect get on the 'develop' and the 'master' branches.

bartlettroscoe commented 2 years ago

@vbrunini and @jclause, with the merge of #10720, can we close this issue? Can you confirm that Trilinos 'develop' has this issue resolved?

vbrunini commented 2 years ago

Sorry for the delay, yes this is all set as far as I'm concerned. Thanks!