Closed vbrunini closed 2 years ago
@trilinos/teuchos @cgcgcg @bartlettroscoe
So you want double
, float
and std::complex<double>
but not std::complex<float>
?
What are your exact Trilinos configure options so that I can reproduce?
Your comment got a bit mangled, but I think you're correct. We want double
, float
, std::complex<double>
but not std::complex<float>
.
Here's my set of configure options from a recent Cuda build that hit the problem, but I also saw it in a clang build with just the Kokkos Serial backend that would probably be faster to reproduce. Note that Trilinos itself compiled and installed, it was only when trying to link against Trilinos that I saw the errors (though maybe the Trilinos tests would show the problem as well?)
-DBUILD_SHARED_LIBS:BOOL=OFF
-DCMAKE_BUILD_TYPE:STRING=release
-DCMAKE_C_COMPILER=/usr/tce/packages/spectrum-mpi/spectrum-mpi-2019.06.24-gcc-7.2.1-redhat/bin/mpicc
-DCMAKE_CXX_COMPILER=/usr/tce/packages/spectrum-mpi/spectrum-mpi-2019.06.24-gcc-7.2.1-redhat/bin/mpicxx
-DCMAKE_Fortran_COMPILER=/usr/tce/packages/spectrum-mpi/spectrum-mpi-2019.06.24-gcc-7.2.1-redhat/bin/mpif90
-DCMAKE_CXX_STANDARD=14
-DCMAKE_C_FLAGS_RELEASE_OVERRIDE:STRING -DNDEBUG
-DCMAKE_CXX_FLAGS_RELEASE_OVERRIDE:STRING -DNDEBUG
-DCMAKE_Fortran_FLAGS_RELEASE_OVERRIDE:STRING -DNDEBUG
-DCMAKE_C_FLAGS:STRING=-O2 -w
-DCMAKE_CXX_FLAGS:STRING=-O2 -w
-DCMAKE_Fortran_FLAGS:STRING=-O2 -w -fno-underscoring
-DCMAKE_EXE_LINKER_FLAGS:STRING=
-DHAVE_GCC_ABI_DEMANGLE:BOOL=ON
-DHAVE_TEUCHOS_BLASFLOAT:BOOL=ON
-DHAVE_TEUCHOS_BLASFLOAT_DOUBLE_RETURN:BOOL=OFF
-DTrilinos_ASSERT_MISSING_PACKAGES:BOOL=OFF
-DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF
-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF
-DTrilinos_ENABLE_COMPLEX_DOUBLE:BOOL=ON
-DTrilinos_ENABLE_DEBUG:BOOL=OFF
-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
-DTrilinos_ENABLE_FLOAT:BOOL=ON
-DTrilinos_ENABLE_LINEAR_SOLVER_FACTORY_REGISTRATION:BOOL=ONTrilinos_SHOW_DEPRECATED_WARNINGS:BOOL=OFF
-DTrilinos_WARNINGS_AS_ERRORS_FLAGS:STRING=
-DTPL_ENABLE_BLAS:BOOL=ON
-DBLAS_LIBRARY_NAMES=blas
-DBLAS_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/netlib-lapack-3.9.0-x7kopcaztcrjx6vdf6posee26zp5ikwi/lib64
-DTPL_ENABLE_Boost:BOOL=ON
-DTPL_Boost_INCLUDE_DIRS:PATH=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/boost-1.77.0-lik62zm2hcn6eftuxlx2t6fpvtrow2ru/include
-DBoost_LIBRARY_DIRS:PATH=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/boost-1.77.0-lik62zm2hcn6eftuxlx2t6fpvtrow2ru/lib
-DTPL_ENABLE_CUDA:BOOL=ON
-DTPL_ENABLE_LAPACK:BOOL=ON
-DLAPACK_LIBRARY_NAMES=lapack
-DLAPACK_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/netlib-lapack-3.9.0-x7kopcaztcrjx6vdf6posee26zp5ikwi/lib64
-DTPL_ENABLE_METIS:BOOL=ON
-DMETIS_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/metis-5.1.0-ghfislukb3c2rsv4qncvl2vyoinul7ba/lib
-DMETIS_LIBRARY_NAMES=metis
-DMETIS_INCLUDE_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/metis-5.1.0-ghfislukb3c2rsv4qncvl2vyoinul7ba/include
-DTPL_ENABLE_MPI:BOOL=ON
-DMPI_BASE_DIR:PATH=/usr/tce/packages/spectrum-mpi/spectrum-mpi-2019.06.24-gcc-7.2.1-redhat
-DTPL_ENABLE_ParMETIS:BOOL=ON
-DParMETIS_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/parmetis-4.0.3-bygkh2pki56aakaqsjitfbq6rnmop26q/lib
-DParMETIS_LIBRARY_NAMES=parmetis;metis
-DParMETIS_INCLUDE_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/parmetis-4.0.3-bygkh2pki56aakaqsjitfbq6rnmop26q/include
-DTPL_ENABLE_Pthread:BOOL=OFF
-DSuperLU_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/superlu-5.2.1-mubeuixj7xzeqdjaeporkzmvhusykhud/lib
-DSuperLU_INCLUDE_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/superlu-5.2.1-mubeuixj7xzeqdjaeporkzmvhusykhud/include
-DTPL_ENABLE_yaml-cpp:BOOL=OFF
-DTPL_ENABLE_y12m:BOOL=ON
-Dy12m_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/y12m-1.0.0-4gmum67kdzctzpaylpjr4fs7lpw6xris/lib
-DTPL_ENABLE_UMFPACK:BOOL=ON
-DUMFPACK_LIBRARY_DIRS:PATH=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/umfpack-5.1.0-gmm3ukxqgonnwdxyckbrbnrrmrckivoj/lib
-DUMFPACK_INCLUDE_DIRS:PATH=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/umfpack-5.1.0-gmm3ukxqgonnwdxyckbrbnrrmrckivoj/include
-DUMFPACK_LIBRARY_NAMES=umfpack;amd
-DTPL_ENABLE_Scotch:BOOL=ON
-DScotch_LIBRARY_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/scotch-6.0.4-d7v6yy5ptuv3g2ixrq6aihhauiswxcn4/lib
-DScotch_INCLUDE_DIRS=/vscratch1/vebruni/code/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_pl/linux-rhel7-ppc64le/gcc-7.2.1/scotch-6.0.4-d7v6yy5ptuv3g2ixrq6aihhauiswxcn4/include
-DTPL_ENABLE_CUSPARSE:BOOL=ON
-DKokkos_ENABLE_CUDA_UVM:BOOL=ON
-DKokkos_ENABLE_CUDA_LAMBDA:BOOL=ON
-DKokkos_ENABLE_CUDA_CONSTEXPR:BOOL=ON
-DTpetra_ASSUME_CUDA_AWARE_MPI:BOOL=OFF
-DKokkos_ARCH_VOLTA70:BOOL=ON
-DTrilinos_ENABLE_Amesos:BOOL=ON
-DTrilinos_ENABLE_Amesos2:BOOL=ON
-DTrilinos_ENABLE_Anasazi:BOOL=ON
-DTrilinos_ENABLE_AztecOO:BOOL=ON
-DTrilinos_ENABLE_Belos:BOOL=ON
-DTrilinos_ENABLE_Epetra:BOOL=ON
-DTrilinos_ENABLE_EpetraExt:BOOL=ON
-DTrilinos_ENABLE_FEI:BOOL=ON
-DTrilinos_ENABLE_Galeri:BOOL=OFF
-DTrilinos_ENABLE_Gtest:BOOL=OFF
-DTrilinos_ENABLE_Ifpack:BOOL=ON
-DTrilinos_ENABLE_Ifpack2:BOOL=ON
-DTrilinos_ENABLE_Intrepid:BOOL=ON
-DTrilinos_ENABLE_Intrepid2:BOOL=ON
-DTrilinos_ENABLE_Isorropia:BOOL=OFF
-DTrilinos_ENABLE_Kokkos:BOOL=ON
-DTrilinos_ENABLE_KokkosCore:BOOL=ON
-DTrilinos_ENABLE_KokkosContainers:BOOL=ON
-DTrilinos_ENABLE_MiniTensor:BOOL=ON
-DTrilinos_ENABLE_ML:BOOL=ON
-DTrilinos_ENABLE_MOOCHO:BOOL=OFF
-DTrilinos_ENABLE_MueLu:BOOL=ON
-DTrilinos_ENABLE_OpenMP:BOOL=OFF
-DTrilinos_ENABLE_Pamgen:BOOL=ON
-DTrilinos_ENABLE_ROL:BOOL=ON
-DTrilinos_ENABLE_RTOp:BOOL=ON
-DTrilinos_ENABLE_Rythmos:BOOL=OFF
-DTrilinos_ENABLE_Sacado:BOOL=ON
-DTrilinos_ENABLE_SEACAS:BOOL=OFF
-DTrilinos_ENABLE_Shards:BOOL=ON
-DTrilinos_ENABLE_ShyLU_NodeTacho:BOOL=ON
-DTrilinos_ENABLE_ShyLU_NodeHTS:BOOL=ON
-DTrilinos_ENABLE_STK:BOOL=OFF
-DTrilinos_ENABLE_Stokhos:BOOL=OFF
-DTrilinos_ENABLE_SuperLU5_API:BOOL=ON
-DTrilinos_ENABLE_Teuchos:BOOL=ON
-DTrilinos_ENABLE_Teko:BOOL=ON
-DTrilinos_ENABLE_ThreadPool:BOOL=ON
-DTrilinos_ENABLE_Thyra:BOOL=ON
-DTrilinos_ENABLE_ThyraEpetraExtAdapters:BOOL=ON
-DTrilinos_ENABLE_Tpetra:BOOL=ON
-DTrilinos_ENABLE_TrilinosCouplings:BOOL=OFF
-DTrilinos_ENABLE_Triutils:BOOL=ON
-DTrilinos_ENABLE_Xpetra:BOOL=ON
-DTrilinos_ENABLE_Zoltan:BOOL=ON
-DTrilinos_ENABLE_Zoltan2:BOOL=ON
-DTrilinos_ENABLE_ShyLU_NodeFastILU:BOOL=ON
-DTrilinos_ENABLE_PanzerExprEval:BOOL=ON
-DAmesos_ENABLE_SuperLU:BOOL=ON
-DAmesos2_ENABLE_ShyLU_NodeTacho:BOOL=ON
-DAmesos2_ENABLE_SuperLU:BOOL=ON
-DAztecOO_ENABLE_AZLU:BOOL=ON
-DBelos_ENABLE_Xpetra:BOOL=OFF
-DIfpack2_ENABLE_ThyraTpetraAdapters=ON
-DKokkos_ENABLE_CUDA:BOOL=ON
-DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE:BOOL=ON
-DKokkos_ENABLE_DEPRECATED_CODE:BOOL=OFF
-DKokkos_ENABLE_OPENMP:BOOL=OFF
-DKokkos_ENABLE_PROFILING:BOOL=ON
-DKokkosKernels_ENABLE_SUPERNODAL_SPTRSV=OFF
-DML_ENABLE_METIS:BOOL=OFF
-DML_ENABLE_ParMETIS:BOOL=OFF
-DPamgen_ENABLE_Boost:BOOL=OFF
-DPamgen_ENABLE_Extras:BOOL=OFF
-DROL_ENABLE_EXAMPLES:BOOL=OFF
-DROL_ENABLE_TESTS:BOOL=OFF
-DSacado_ENABLE_KokkosCore:BOOL=ON
-DTacho_ENABLE_INT_INT:BOOL=ON
-DTpetra_INST_INT_LONG:BOOL=ON
-DTpetra_INST_INT_LONG_LONG:BOOL=OFF
-DTpetra_INST_OPENMP:BOOL=OFF
-DTpetra_INST_SERIAL:BOOL=ON
-DZoltan_ENABLE_METIS:BOOL=OFF
-DZoltan_ENABLE_Scotch:BOOL=OFF
-DZoltan2_ENABLE_Experimental:BOOL=ON
-DIfpack2_ENABLE_ShyLU_NodeFastILU:BOOL=ON
-DAmesos2_ENABLE_UMFPACK:BOOL=ON
-DZoltan2_ENABLE_Scotch:BOOL=ON
What is the exact link error you are seeing? What *.o file in Trilinos is this coming from? From what I am looking at, you should be getting std::complex<float>
instantiations from these Teuchos macros.
For more detail about the status of this, see below ...
So it seems that the configure options:
Trilinos_ENABLE_COMPLEX_FLOAT
Trilinos_ENABLE_COMPLEX_COMPLEX
exist at the global Trilinos level but only the configure options:
Teuchos_ENABLE_FLOAT
Teuchos_ENABLE_COMPLEX
exist at the Teuchos level as seen at:
It seems that no one ever duplicated those finer-grained Trilinos options at the Teuchos level with the todo shown at:
which references https://github.com/trilinos/Trilinos/issues/362 (see the full discussion there). (I suspect I wrote that "ToDo" over 6 years aog.)
So at the Teuchos level, it assumes that if you want "float" and "complex", you will get float
, std::complex<double>
and std::complex<float>
. And the Teuchos ETI macros seems to enforce that so you should not be seeing link errors (just extra std::complex<float>
instantiations you may not want) as shown at:
We need to see the exact link errors to decide how to move forward.
The link errors are references to instantiations of Tpetra classes for std::complex<float>
which Tpetra did not instantiate (as desired). For example:
/fgs/vebruni/src/code_clean/objs/tpls/spack/spack/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_pat/linux-rhel7-x86_64/gcc-8.3.0/trilinos-2022.06.17.03-lq4lfl5k6staichygb35tieg4d2wg7h2/lib/libstratimikosbelos.a(Thyra_BelosLinearOpWithSolve.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<std::complex<float>, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeDenseHeader(std::ostream&, Tpetra::MultiVector<std::complex<float>, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&)':
tmpxft_0000f09d_00000000-6_Thyra_BelosLinearOpWithSolve.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixISt7complexIfEilN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIS4_ilSA_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESP_RKN7Teuchos3RCPINSQ_18basic_FancyOStreamIcSL_EEEESW_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixISt7complexIfEilN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIS4_ilSA_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESP_RKN7Teuchos3RCPINSQ_18basic_FancyOStreamIcSL_EEEESW_]+0x3bd): undefined reference to `Tpetra::MultiVector<std::complex<float>, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getGlobalLength() const'
tmpxft_0000f09d_00000000-6_Thyra_BelosLinearOpWithSolve.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixISt7complexIfEilN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIS4_ilSA_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESP_RKN7Teuchos3RCPINSQ_18basic_FancyOStreamIcSL_EEEESW_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixISt7complexIfEilN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIS4_ilSA_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESP_RKN7Teuchos3RCPINSQ_18basic_FancyOStreamIcSL_EEEESW_]+0x3e5): undefined reference to `Tpetra::MultiVector<std::complex<float>, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getNumVectors() const'
There are a bunch more, but all appear to be from files in Thyra missing references to things in Tpetra.
There are a bunch more, but all appear to be from files in Thyra missing references to things in Tpetra.
Okay, that makes sense. That means that I can likely reproduce this locally just with the Trilinos test suite. (Which means that there are no PR builds that test this combination of configure options.)
The fix is a little more tricky if we want to maintain backward compatibility. But it just so happens that we are allowed to break backward compatibility until Trilinos 14.0 is released in the next 3 months or so.
This is the fix I propose:
1) Add the cache vars Teuchos_ENABLE_COMPLEX_DOUBLE
and Teuchos_ENABLE_COMPLEX_FLOAT
which map to #defines
HAVE_TEUCHOS_COMPLEX_DOUBLE
and HAVE_TEUCHOS_COMPLEX_FLOAT
, respectively, in the configured Teuchos_config.h
file.
2) Set the defaults for Teuchos_ENABLE_COMPLEX_DOUBLE
and Teuchos_ENABLE_COMPLEX_FLOAT
to match the same logic for the global options here.
3) Upgrade the macros in Teuchos_ExplicitInstantiationHelpers.hpp
to key off of HAVE_TEUCHOS_COMPLEX_FLOAT
and HAVE_TEUCHOS_COMPLEX_DOUBLE
instead of HAVE_TEUCHOS_COMPLEX
and Trilinos_ENABLE_COMPLEX
.
The above has the following advantages:
Trilinos_ENABLE_FLOAT
, Trilinos_ENABLE_COMPLEX
, Trilinos_ENABLE_COMPLEX_FLOAT
and Trilinos_ENABLE_COMPLEX_DOUBLE
.Trilinos_ENABLE_FLOAT=ON
and Trilinos_ENABLE_COMPLEX=ON
in that it will enable std::complex<float>
support (and ETI instantiations).Trilinos_ENABLE_FLOAT=ON
and Trilinos_ENABLE_COMPLEX_DOUBLE=ON
will only give ETI instantiations for float
, double
and std::complex<double>
but not std::complex<float>
.@e10harvey, did you mention some issue with Teuchos related to the Trilinos 'develop' to 'master' promotion? Was it an ETI issue like this?
I think this issue has been latent in Teuchos for a long time, but is now impacting us after commit https://github.com/trilinos/Trilinos/commit/e08b53677de4ac72b9e060001dd5e63b6c2b167f which started using these Teuchos macros in stratimikos.
I think the problem is not so much that this is a defect in Teuchos, it is that a package downstream from Tpetra used the Teuchos ETI macros to do the instantations. Perhaps the Stratimikos commit https://github.com/trilinos/Trilinos/commit/e08b53677de4ac72b9e060001dd5e63b6c2b167f should have used Tpetra ETI macros instead of the Teuchos ETI macros?
But in any case, Teuchos and Tpetra should have identical logic for what instantiations to produce based on the top-level options Trilinos_ENABLE_FLOAT
, Trilinos_ENABLE_COMPLEX
, Trilinos_ENABLE_COMPLEX_FLOAT
and Trilinos_ENABLE_COMPLEX_DOUBLE
so the change described above should be made in any case.
How urgent is the fix of this issues?
@bartlettroscoe: No - it was a link time problem with abs
in intel-19. We should have filed a bug report for it but here is the comment showing the error: https://github.com/trilinos/Trilinos/pull/10486#issuecomment-1116083746.
@bartlettroscoe: No - it was a link time problem with
abs
in intel-19. We should have filed a bug report for it but here is the comment showing the error: #10486 (comment).
Okay, that means that there is not even a 'develop' to 'master' build for this configuration Trilinos_ENABLE_FLOAT=ON
and Trilinos_ENABLE_COMPLEX_DOUBLE=ON
that would have caught this bug before it was synced/deployed to 'master'. Given that Sierra is an important Trilinos customer, I will put in a request to add/update and 'develop' to 'master' build to cover this.
I'd say fairly urgent. The issue will block integrating any version of Trilinos after the stratimikos change into Sierra, and we're going to need to do that within the next month or so for milestone work. If necessary we can apply a patch to our spack builds as a workaround but the shorter lived patches are the better, and not needing to would be ideal.
I'd say fairly urgent.
Okay, I will try to reproduce and address this later this week.
Yes, I've been trying to clean up bugs so we can do another integration, and this bug is blocking that effort.
Sorry for the delay. I will work on this tomorrow unless a (figurative) bomb goes off.
FYI: I have reproduced the build error and am working on a solution along the lines above. However, there a lot of code (in 117 files) is keying off of HAVE_TEUCHOS_COMPLEX
as shown in the following find-grep operation.
This is why no one pulled the trigger on this refactoring way back in #362.
With having to edit all 117 of those files and having to test all of that, this could take some time. (And if we keep HAVE_TEUCHOS_COMPLEX
in addition to the new defines HAVE_TEUCHOS_COMPLEX_FLOAT
and HAVE_TEUCHOS_COMPLEX_COMPLEX
, then what does HAVE_TEUCHOS_COMPLEX
mean at that point?)
This is even harder than I thought. In addition to C++ files referencing the define HAVE_TEUCHOS_COMPLEX
, there are also a total of 37 files spanning 11 packages mentioning Teuchos_ENABLE_COMPLEX
as shown below.
In total, there are 147 files mentioning HAVE_TEUCHOS_COMPLEX
or Teuchos_ENABLE_COMPLEX
as shown in:
$ find packages -type f -exec grep -l "\(Teuchos_ENABLE_COMPLEX\|HAVE_TEUCHOS_COMPLEX\)" {} \; | wc -l
147
The strategy of outlined above would take several days of detailed code inspection, code change, and testing. Also that change would undoubtedly break some existing Trilinos customers that may be keying off of at least the define HAVE_TEUCHOS_COMPLEX
in the configured header file Teuchos_config.h
.
Therefore, instead of refactoring Teuchos in the way described above, we should instead copy the file Teuchos_ExplicitInstantiationHelpers.hpp
over to Tpetra_ExplicitInstantiationHelpers.hpp
and update the macros to key off of the appropriate Tpetra CMake variables like Tpetra_INST_COMPLEX_FLOAT
and Tpetra_INST_COMPLEX_DOUBLE
. That way, we can fix the instantiation problem reported in this issue without requiring large code changes and lots of testing and debugging.
Basically, the current Teuchos logic assumes that if Teuchos_ENABLE_COMPLEX=TRUE
and Teuchos_ENABLE_FLOAT=TRUE
, then you get support for complex<float>
enabled (in Teuchos) and you can't turn that off if those two vars are enabled. But since Teuchos support for complex needs to be a superset of what is required downstream, that should be fine.
Giving the reproduction info I used to produced the same link errors reported above. Basically, you just configure Trilinos with:
-DTrilinos_ENABLE_TESTS=ON \
-D Trilinos_ENABLE_Stratimikos=ON \
-D Trilinos_ENABLE_COMPLEX_DOUBLE=ON \
-D Trilinos_ENABLE_FLOAT=ON \
and you get the link errors.
FYI: PR #10720 contains the changes that should fix the build issue for Sierra. I am not ready to merge yet as I am fixing the Trilinos test suite for this use case.
NOTE: It appears there are not automated Trilinos builds that enable float
! That is a huge hole in the testing coverage for Trilinos given that Sierra requires float
. I will post an issue about that.
FYI: I created the GitHub Issue #10742 and the internal issue:
for the gap in Trilinos testing that let this defect get on the 'develop' and the 'master' branches.
@vbrunini and @jclause, with the merge of #10720, can we close this issue? Can you confirm that Trilinos 'develop' has this issue resolved?
Sorry for the delay, yes this is all set as far as I'm concerned. Thanks!
Bug Report
@trilinos/teuchos
Description
Teuchos_ExplicitInstantiationHelpers.hpp checks. This incorrectly enables those instantiations when the user wants to only instantiate for float, double, and complex.
#if defined(HAVE_TEUCHOS_COMPLEX) && defined(HAVE_TEUCHOS_FLOAT)
to determine whether or not to instantiate for complexThis is causing linking errors when trying to build Sierra against Trilinos/develop. I think this issue has been latent in Teuchos for a long time, but is now impacting us after commit e08b53677de4ac72b9e060001dd5e63b6c2b167f which started using these Teuchos macros in stratimikos.