search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.21k stars 566 forks source link

MueLu: Linker errors from missing explicit template instations for `Kokkos::CudaUVMSpace` #12846

Closed MalachiTimothyPhillips closed 7 months ago

MalachiTimothyPhillips commented 7 months ago

Bug Report

@MueLu @jhux2

Description

I am running into some linker issues in a nvidia Sierra spack build of Trilinos. It looks as if MueLu is missing explicit template instantiations for Kokkos::CudaUVMSpace (see linker output below). I will attempt to resolve the issue on my end, but I am not familiar with how MueLu handles the explicit template instantiations.

/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `MueLu::HierarchyManager<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::CreateHierarchy(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE15CreateHierarchyERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE15CreateHierarchyERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1e): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Hierarchy(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `MueLu::HierarchyManager<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::CreateHierarchy() const':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE15CreateHierarchyEv[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE15CreateHierarchyEv]+0x1a): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Hierarchy()'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Teuchos::RCP<MueLu::SmootherFactory<float, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > MueLu::AdaptiveDetails::getSmoother<float, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Teuchos::ParameterList&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN5MueLu15AdaptiveDetails11getSmootherIfilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos6SerialENS5_9HostSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNS9_13ParameterListE[_ZN5MueLu15AdaptiveDetails11getSmootherIfilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos6SerialENS5_9HostSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNS9_13ParameterListE]+0x74): undefined reference to `MueLu::ParameterListInterpreter<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::ParameterListInterpreter(Teuchos::ParameterList&, Teuchos::RCP<Teuchos::Comm<int> const>, Teuchos::RCP<MueLu::FactoryFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >, Teuchos::RCP<MueLu::FacadeClassFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Teuchos::RCP<MueLu::SmootherFactory<float, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > > MueLu::AdaptiveDetails::getSmoother<float, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >(Teuchos::ParameterList&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN5MueLu15AdaptiveDetails11getSmootherIfilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNS9_13ParameterListE[_ZN5MueLu15AdaptiveDetails11getSmootherIfilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNS9_13ParameterListE]+0x74): undefined reference to `MueLu::ParameterListInterpreter<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::ParameterListInterpreter(Teuchos::ParameterList&, Teuchos::RCP<Teuchos::Comm<int> const>, Teuchos::RCP<MueLu::FactoryFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >, Teuchos::RCP<MueLu::FacadeClassFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Teuchos::RCP<MueLu::SmootherFactory<double, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > MueLu::AdaptiveDetails::getSmoother<double, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Teuchos::ParameterList&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN5MueLu15AdaptiveDetails11getSmootherIdilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos6SerialENS5_9HostSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNS9_13ParameterListE[_ZN5MueLu15AdaptiveDetails11getSmootherIdilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos6SerialENS5_9HostSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNS9_13ParameterListE]+0x74): undefined reference to `MueLu::ParameterListInterpreter<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::ParameterListInterpreter(Teuchos::ParameterList&, Teuchos::RCP<Teuchos::Comm<int> const>, Teuchos::RCP<MueLu::FactoryFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >, Teuchos::RCP<MueLu::FacadeClassFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Teuchos::RCP<MueLu::SmootherFactory<double, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > > MueLu::AdaptiveDetails::getSmoother<double, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >(Teuchos::ParameterList&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN5MueLu15AdaptiveDetails11getSmootherIdilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNS9_13ParameterListE[_ZN5MueLu15AdaptiveDetails11getSmootherIdilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNS9_13ParameterListE]+0x74): undefined reference to `MueLu::ParameterListInterpreter<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::ParameterListInterpreter(Teuchos::ParameterList&, Teuchos::RCP<Teuchos::Comm<int> const>, Teuchos::RCP<MueLu::FactoryFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >, Teuchos::RCP<MueLu::FacadeClassFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Teuchos::RCP<MueLu::SmootherFactory<std::complex<float>, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > MueLu::AdaptiveDetails::getSmoother<std::complex<float>, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(Teuchos::ParameterList&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN5MueLu15AdaptiveDetails11getSmootherISt7complexIfEilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos6SerialENS7_9HostSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNSB_13ParameterListE[_ZN5MueLu15AdaptiveDetails11getSmootherISt7complexIfEilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos6SerialENS7_9HostSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNSB_13ParameterListE]+0x74): undefined reference to `MueLu::ParameterListInterpreter<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::ParameterListInterpreter(Teuchos::ParameterList&, Teuchos::RCP<Teuchos::Comm<int> const>, Teuchos::RCP<MueLu::FactoryFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >, Teuchos::RCP<MueLu::FacadeClassFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o):tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN5MueLu15AdaptiveDetails11getSmootherISt7complexIfEilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS7_12CudaUVMSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNSB_13ParameterListE[_ZN5MueLu15AdaptiveDetails11getSmootherISt7complexIfEilN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS7_12CudaUVMSpaceEEEEEN7Teuchos3RCPINS_15SmootherFactoryIT_T0_T1_T2_EEEERNSB_13ParameterListE]+0x74): more undefined references to `MueLu::ParameterListInterpreter<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::ParameterListInterpreter(Teuchos::ParameterList&, Teuchos::RCP<Teuchos::Comm<int> const>, Teuchos::RCP<MueLu::FactoryFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >, Teuchos::RCP<MueLu::FacadeClassFactory<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >)' follow
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `MueLu::HierarchyManager<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::ExportDataSetKeepFlags(MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >&, Teuchos::Array<int> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE22ExportDataSetKeepFlagsERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE22ExportDataSetKeepFlagsERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x64): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetNumLevels() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE22ExportDataSetKeepFlagsERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE22ExportDataSetKeepFlagsERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x78): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetLevel(int)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `MueLu::HierarchyManager<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::WriteDataAggregates(MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >&, Teuchos::Array<int> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE19WriteDataAggregatesERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE19WriteDataAggregatesERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1ff): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetNumLevels() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE19WriteDataAggregatesERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE19WriteDataAggregatesERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x21e): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetLevel(int)'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE19WriteDataAggregatesERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE19WriteDataAggregatesERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x295): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetNumLevels() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE19WriteDataAggregatesERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE19WriteDataAggregatesERNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x417): undefined reference to `MueLu::Aggregates<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::print(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, Teuchos::EVerbosityLevel) const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Xpetra::Matrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetFixedBlockSize() const':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK6Xpetra6MatrixIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE17GetFixedBlockSizeEv[_ZNK6Xpetra6MatrixIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE17GetFixedBlockSizeEv]+0x21d): undefined reference to `Xpetra::StridedMap<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getFixedBlockSize() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK6Xpetra6MatrixIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE17GetFixedBlockSizeEv[_ZNK6Xpetra6MatrixIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE17GetFixedBlockSizeEv]+0x229): undefined reference to `Xpetra::StridedMap<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getFixedBlockSize() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK6Xpetra6MatrixIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE17GetFixedBlockSizeEv[_ZNK6Xpetra6MatrixIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE17GetFixedBlockSizeEv]+0x23c): undefined reference to `Xpetra::StridedMap<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getFixedBlockSize() const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeMap(std::ostream&, Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, bool)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x34): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x4fc): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::isContiguous() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x1806): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getLocalElementList() const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Xpetra::IO<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Xpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_3MapIiiS7_EE[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_3MapIiiS7_EE]+0xa4): undefined reference to `Xpetra::TpetraMap<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getTpetra_Map() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_3MapIiiS7_EE[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_3MapIiiS7_EE]+0xb6): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> Tpetra::Details::computeGatherMap<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >(Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const>, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, bool)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x36): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0xc1): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::isContiguous() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0xec): undefined reference to `Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> Tpetra::createOneToOne<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >(Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> const&)'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x22a): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getLocalElementList() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra7Details16computeGatherMapINS_3MapIiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEEEN7Teuchos3RCPIKT_EESE_RKNSB_INSA_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x465): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Map(unsigned long, Teuchos::ArrayView<int const> const&, int, Teuchos::RCP<Teuchos::Comm<int> const> const&)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeSparse(std::ostream&, Tpetra::CrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b]+0x266): undefined reference to `Tpetra::Import<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Import(Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> const&, Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> const&)'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b]+0x2b5): undefined reference to `Tpetra::CrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::CrsMatrix(Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> const&, Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> const&, unsigned long, Teuchos::RCP<Teuchos::ParameterList> const&)'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b]+0x336): undefined reference to `Tpetra::DistObject<char, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::doImport(Tpetra::SrcDistObject const&, Tpetra::Import<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, Tpetra::CombineMode, bool)'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b]+0x365): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Map(unsigned long, unsigned long, int, Teuchos::RCP<Teuchos::Comm<int> const> const&)'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b]+0x412): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Map(unsigned long, unsigned long, int, Teuchos::RCP<Teuchos::Comm<int> const> const&)'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b]+0x4dc): undefined reference to `Tpetra::CrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::fillComplete(Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> const&, Teuchos::RCP<Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const> const&, Teuchos::RCP<Teuchos::ParameterList> const&)'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b]+0x1165): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getGlobalElement(int) const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE11writeSparseERSoRKS9_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESL_b]+0x1258): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getGlobalElement(int) const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Xpetra::IO<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Xpetra::Matrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, bool const&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_6MatrixIdiiS7_EERKb[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_6MatrixIdiiS7_EERKb]+0x272): undefined reference to `Xpetra::CrsMatrixWrap<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getCrsMatrix() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_6MatrixIdiiS7_EERKb[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_6MatrixIdiiS7_EERKb]+0x343): undefined reference to `Xpetra::TpetraCrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getTpetra_CrsMatrix() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_6MatrixIdiiS7_EERKb[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_6MatrixIdiiS7_EERKb]+0x6b7): undefined reference to `Xpetra::TpetraBlockCrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getTpetra_BlockCrsMatrix() const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `void MueLu::HierarchyManager<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::WriteData<Xpetra::Matrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >(MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >&, Teuchos::Array<int> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE9WriteDataIN6Xpetra6MatrixIdiiS7_EEEEvRNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE9WriteDataIN6Xpetra6MatrixIdiiS7_EEEEvRNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1e6): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetNumLevels() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE9WriteDataIN6Xpetra6MatrixIdiiS7_EEEEvRNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE9WriteDataIN6Xpetra6MatrixIdiiS7_EEEEvRNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x353): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetLevel(int)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeDenseHeader(std::ostream&, Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIdiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIdiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_]+0x135): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIdiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIdiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_]+0x3ec): undefined reference to `Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getGlobalLength() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIdiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIdiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_]+0x41c): undefined reference to `Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getNumVectors() const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeDenseColumn(std::ostream&, Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIdiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIdiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_]+0x43): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIdiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIdiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_]+0x1a0): undefined reference to `Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getLocalLength() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIdiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIdiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_]+0x1ab): undefined reference to `Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getNumVectors() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIdiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIdiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIdiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_]+0x781): undefined reference to `Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::get1dCopy(Teuchos::ArrayView<double> const&, unsigned long) const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Xpetra::IO<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Xpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE]+0xf9): undefined reference to `Xpetra::TpetraMultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getTpetra_MultiVector() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE]+0x2d0): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE]+0x343): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE]+0x456): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE]+0x5bf): undefined reference to `Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getNumVectors() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE[_ZN6Xpetra2IOIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE5WriteERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_11MultiVectorIdiiS7_EE]+0x607): undefined reference to `Tpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getVector(unsigned long) const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `void MueLu::HierarchyManager<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::WriteData<Xpetra::MultiVector<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >(MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >&, Teuchos::Array<int> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE9WriteDataIN6Xpetra11MultiVectorIdiiS7_EEEEvRNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE9WriteDataIN6Xpetra11MultiVectorIdiiS7_EEEEvRNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1e7): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetNumLevels() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE9WriteDataIN6Xpetra11MultiVectorIdiiS7_EEEEvRNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE9WriteDataIN6Xpetra11MultiVectorIdiiS7_EEEEvRNS_9HierarchyIdiiS7_EERKN7Teuchos5ArrayIiEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x35b): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::GetLevel(int)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<int, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeMap(std::ostream&, Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, bool)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x34): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x4fc): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::isContiguous() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE8writeMapERSoRKNS_3MapIiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEEb]+0x1806): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getLocalElementList() const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<int, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeDenseHeader(std::ostream&, Tpetra::MultiVector<int, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIiiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIiiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_]+0x135): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getComm() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIiiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIiiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_]+0x3ec): undefined reference to `Tpetra::MultiVector<int, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getGlobalLength() const'
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIiiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseHeaderERSoRKNS_11MultiVectorIiiiS8_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESN_RKN7Teuchos3RCPINSO_18basic_FancyOStreamIcSJ_EEEESU_]+0x41c): undefined reference to `Tpetra::MultiVector<int, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::getNumVectors() const'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `Tpetra::MatrixMarket::Writer<Tpetra::CrsMatrix<int, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > >::writeDenseColumn(std::ostream&, Tpetra::MultiVector<int, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&, Teuchos::RCP<Teuchos::basic_FancyOStream<char, std::char_traits<char> > > const&)':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIiiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_[_ZN6Tpetra12MatrixMarket6WriterINS_9CrsMatrixIiiiNS_12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS5_12CudaUVMSpaceEEEEEE16writeDenseColumnERSoRKNS_11MultiVectorIiiiS8_EERKN7Teuchos3RCPINSG_18basic_FancyOStreamIcSt11char_traitsIcEEEEESO_]+0x43): undefined reference to `Tpetra::Map<int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaU
github-actions[bot] commented 7 months ago

Automatic mention of the @trilinos/muelu team

jhux2 commented 7 months ago

@trilinos/muelu

@MalachiTimothyPhillips Is this with a recent integration of Trilinos?

MalachiTimothyPhillips commented 7 months ago

@trilinos/muelu

@MalachiTimothyPhillips Is this with a recent integration of Trilinos?

Correct -- this isn't currently preventing Sierra from building, but we are looking to do an integration soon. The last integration we did was November 9, 2013.

csiefer2 commented 7 months ago

@MalachiTimothyPhillips Can you share a configure script?

MalachiTimothyPhillips commented 7 months ago

@MalachiTimothyPhillips Can you share a configure script?

These are the configure flags passed to cmake: spack-configure-args-cleaned.txt

Note the full spack paths for 3rd party library/include dirs is abbreviated <spack_path>. If it helps, I can widdle down the flags to try and get you a minimal reproducer.

cgcgcg commented 7 months ago

@MalachiTimothyPhillips Could you try with these flags?

-D KokkosKernels_INST_MEMSPACE_CUDAUVMSPACE=ON
-D Tpetra_ALLOCATE_IN_SHARED_SPACE=ON

See #12622.

MalachiTimothyPhillips commented 7 months ago

@MalachiTimothyPhillips Could you try with these flags?

-D KokkosKernels_INST_MEMSPACE_CUDAUVMSPACE=ON
-D Tpetra_ALLOCATE_IN_SHARED_SPACE=ON

See #12622.

No luck. I've generated a smaller reproducer that is accessible here: https://github.com/MalachiTimothyPhillips/trilinos-issue-12846.

edit: After taking a closer look, it looks like the linker errors in the stub above (and my example) stem from trying to use int for the global ordinal type.

e.g.:

/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `MueLu::HierarchyManager<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::CreateHierarchy(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const':
tmpxft_000388d7_00000000-6_ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cudafe1.cpp:(.text._ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE15CreateHierarchyERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZNK5MueLu16HierarchyManagerIdiiN6Tpetra12KokkosCompat23KokkosDeviceWrapperNodeIN6Kokkos4CudaENS4_12CudaUVMSpaceEEEE15CreateHierarchyERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x1e): undefined reference to `MueLu::Hierarchy<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::Hierarchy(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/sierra/build/linux_rh7/nightly/nvidia_trilinos_develop_master/objs/tpls/spack/spack/__spack_path_placeholder__/__spack_path_pla/linux-rhel7-x86_64/gcc-8.3.0/trilinos-develop-b3pf3gpcojl3qf3u4qymdldxxpldgb6b/lib64/libmuelu.a(ETI_MueLu_AdaptiveSaMLParameterListInterpreter.cpp.o): In function `MueLu::HierarchyManager<double, int, int, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace> >::CreateHierarchy() const':

The cmake configuration indicates that the only support global ordinal type is long long:

-- MueLu: Enabling ETI support
--    <float,   int, int>       : OFF
--    <float,   int, long long> : ON
--    <double,  int, int>       : OFF
--    <double,  int, long>      : OFF
--    <double,  int, long long> : ON
--    <complex, int, int>       : OFF
--    <complex, int, long long> : ON

I am not sure where/why the int type for the global ordinal is coming from, but I do not yet have a simple reproducer prepared for that yet.

I should also note we are not seeing these linker errors on GCC/Clang, for example.

MalachiTimothyPhillips commented 7 months ago

Setting:

  -DTpetra_INST_INT_INT:BOOL=OFF
  -DTpetra_INST_INT_LONG:BOOL=ON

I see the following CMake configure output:

-- MueLu: Enabling ETI support
--    <float,   int, int>       : OFF
--    <float,   int, long long> : OFF
--    <double,  int, int>       : OFF
--    <double,  int, long>      : ON
--    <double,  int, long long> : OFF
--    <complex, int, int>       : OFF
--    <complex, int, long long> : OFF
-- MueLu: Default GO: int

That has to be wrong, right? I think this is why I'm hitting the linker errors.

spack-build-01-cmake-out.txt

cgcgcg commented 7 months ago

@MalachiTimothyPhillips I'm not sure if this is your issue, but the way the variables work is

Tpetra_INST_LO_GO

So setting Tpetra_INST_INT_LONG results in LO=int, GO=long.

MalachiTimothyPhillips commented 7 months ago

@MalachiTimothyPhillips I'm not sure if this is your issue, but the way the variables work is

Tpetra_INST_LO_GO

So setting Tpetra_INST_INT_LONG results in LO=int, GO=long.

Right -- why is MueLu configured with Default GO: int in that case? e.g., https://github.com/trilinos/Trilinos/blob/88b2d6fa20695c34bdc44a9074a6ab42294446df/packages/muelu/CMakeLists.txt#L242

cgcgcg commented 7 months ago

Oh, no I see that. :eyes:

cgcgcg commented 7 months ago

Can you try to change from: https://github.com/trilinos/Trilinos/blob/88b2d6fa20695c34bdc44a9074a6ab42294446df/packages/muelu/CMakeLists.txt#L241-L242 to 

 IF(Tpetra_INST_DOUBLE AND Tpetra_INST_INT_LONG) 
  GLOBAL_SET(${PACKAGE_NAME}_HAVE_GO_LONG ON)

? Cause that looks fishy.

cgcgcg commented 7 months ago

NVM, I think something is quite wrong in our CMake for long. Well, actually, it might just be weird.

MalachiTimothyPhillips commented 7 months ago

NVM, I think something is quite wrong in our CMake for long. Well, actually, it might just be weird.

Sure!

https://github.com/trilinos/Trilinos/blob/88b2d6fa20695c34bdc44a9074a6ab42294446df/packages/muelu/CMakeLists.txt#L283, too? Or something else?

Previous Trilinos versions (e.g., de68716d971f08d995548a6bdc37f205dbf2ba50, which is our current version in Sierra) had the same configuration. I'm wondering if there's some change in ETI handling for the Kokkos::CudaUVMSpace specifically that is allowing use to still link on cpu-only builds and (previously) nvidia builds.

edit: I guess another complication is the fact that Epetra seems to require <double, int, int>, right?

cgcgcg commented 7 months ago

Nope, only that spot. Yeah, I'm also not sure how this could have worked before.

edit: Yes, if you want Epetra+Tpetra you need int-int.

MalachiTimothyPhillips commented 7 months ago

Good news -- that seems to have fixed the linker errors on our end! See: #12881.

Thank you for your help!

MalachiTimothyPhillips commented 7 months ago

Marking as closed now that #12881 has been merged.