search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.19k stars 565 forks source link

Tpetra: Compilation error with TpetraUtils_WrappedDualView.cpp with cuda/9.2 + gcc/7.2.0 #9237

Closed ndellingwood closed 3 years ago

ndellingwood commented 3 years ago

Bug Report

@trilinos/tpetra

Description

Following merge of PR #9038 some nightly testing with Trilinos began failing with cuda/9.2 + gcc/7.2.0 (sems modules) on Kepler architecture with error messages like:

../../packages/tpetra/core/test/Utils/TpetraUtils_WrappedDualView.cpp:1339:17: error: ‘__T0’ has not been declared
   static_assert(decltype(owningView)::rank == 1,
                 ^~~~

I reproduced this on develop branch with the SHA posted below. This was detected on the kokkos-dev test machine, but since it uses sems modules I'm assuming this could be reproduced elsewhere that has the sems env available.

Steps to Reproduce

SHA1: 3f683fc90aa9740f002b56190cbe883f0e188b9d

Modules: 

[ndellin@kokkos-dev Test]$ module list
Currently Loaded Modulefiles:
  1) sems-env                     5) sems-gcc/7.2.0               9) sems-openmpi/4.0.2          13) sems-boost/1.59.0/base
  2) sems-git/2.10.1              6) sems-cuda/9.2               10) sems-hdf5/1.10.6/parallel   14) sems-superlu/4.3/base
  3) sems-cmake/3.17.1            7) atdm-env                    11) sems-netcdf/4.7.3/parallel
  4) sems-ninja_fortran/1.8.2     8) atdm-openblas/0.3.6         12) sems-zlib/1.2.8/base

Configure script: (note this loads modules/env specific to kokkos-dev)

export ATDM_CONFIG_REGISTER_CUSTOM_CONFIG_DIR=${TRILINOS_DIR}/cmake/std/atdm/contributed/kokkos-dev
source ${TRILINOS_DIR}/cmake/std/atdm/load-env.sh kokkos-dev-cuda-opt

module list
export OMPI_CXX=$TRILINOS_DIR/packages/kokkos/bin/nvcc_wrapper

cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DCMAKE_INSTALL_PREFIX="${PWD}/install" \
 -DCMAKE_CXX_STANDARD="14" \
 -DTrilinos_ENABLE_TESTS=OFF \
 -DTrilinos_ENABLE_ALL_PACKAGES=OFF \
 -DTrilinos_ENABLE_Kokkos=ON \
  -DKokkos_ARCH_KEPLER35=ON \
 -DTrilinos_ENABLE_KokkosKernels=ON \
  -DKokkosKernels_ENABLE_TESTS=ON \
 -DTrilinos_ENABLE_Tpetra=ON \
   -DTpetra_ENABLE_TESTS=ON \
 -DKokkos_ENABLE_CUDA=ON \
$TRILINOS_DIR

Configure output

Expand dropdown ``` [ndellin@kokkos-dev Test]$ . ./configure-nightly.sh 2>&1 | tee configure.out Selecting custom system configuration 'kokkos-dev' Setting compiler and build options for build-name 'kokkos-dev-cuda-opt' CUDA ATDM COMPILER: CUDA Using SEMS RHEL7 compiler stack CUDA-9.2 to build RELEASE code with Kokkos node type CUDA Currently Loaded Modulefiles: 1) sems-env 8) atdm-openblas/0.3.6 2) sems-git/2.10.1 9) sems-openmpi/4.0.2 3) sems-cmake/3.17.1 10) sems-hdf5/1.10.6/parallel 4) sems-ninja_fortran/1.8.2 11) sems-netcdf/4.7.3/parallel 5) sems-gcc/7.2.0 12) sems-zlib/1.2.8/base 6) sems-cuda/9.2 13) sems-boost/1.59.0/base 7) atdm-env 14) sems-superlu/4.3/base Configuring Trilinos build directory -- PROJECT_SOURCE_DIR='/ascldap/users/ndellin/trilinos/Trilinos' -- PROJECT_BINARY_DIR='/ascldap/users/ndellin/trilinos/Trilinos/BUILD/Test' -- Trilinos_TRIBITS_DIR='/ascldap/users/ndellin/trilinos/Trilinos/cmake/tribits' -- TriBITS_VERSION_STRING='0.9 (Dev)' -- CMAKE_VERSION='3.17.1' -- CMAKE_GENERATOR='Ninja' -- Reading in configuration options from cmake/std/atdm/ATDMDevEnv.cmake ... -- ATDM_BUILD_NAME_KEYS_STR='CUDA-9.2_RELEASE_CUDA' -- ATDM_TWEAKS_FILES='' -- Setting default TPL_ENABLE_MPI=ON -- Setting default Trilinos_ENABLE_Fortran=OFF -- Setting default Trilinos_ENABLE_SECONDARY_TESTED_CODE=ON -- Setting default TPL_ENABLE_CUDA=ON -- Setting default TPL_ENABLE_CUSPARSE=ON -- Setting default TPL_ENABLE_Pthread=OFF -- Setting default TPL_ENABLE_BinUtils=OFF -- Setting default TPL_ENABLE_BLAS=ON -- Setting default TPL_ENABLE_Boost=ON -- Setting default TPL_ENABLE_BoostLib=ON -- Setting default TPL_ENABLE_METIS=OFF -- Setting default TPL_ENABLE_ParMETIS=OFF -- Setting default TPL_ENABLE_HWLOC=OFF -- Setting default TPL_ENABLE_LAPACK=ON -- Setting default TPL_ENABLE_CGNS=OFF -- Setting default TPL_ENABLE_HDF5=ON -- Setting default TPL_ENABLE_Netcdf=ON -- Setting default TPL_ENABLE_SuperLUDist=OFF -- Setting default TPL_ENABLE_Matio=OFF -- Setting default Trilinos_MAKE_INSTALL_GROUP_WRITABLE=TRUE -- Setting default Teko_ModALPreconditioner_MPI_1_DISABLE=ON -- Setting default Zoltan2_XpetraEpetraMatrix_EXE_DISABLE=ON -- Setting default Zoltan2_XpetraEpetraMatrix_MPI_4_DISABLE=ON -- Setting default Belos_Tpetra_HybridGMRES_hb_test_1_MPI_4_DISABLE=ON -- Setting default Belos_Tpetra_HybridGMRES_hb_test_0_MPI_4_DISABLE=ON -- Setting default Belos_rcg_hb_MPI_4_DISABLE=ON -- Setting default Piro_ThyraSolver_EXE_DISABLE=ON -- Setting default Piro_ThyraSolver_MPI_4_DISABLE=ON -- Setting default Piro_AnalysisDriverTpetra_EXE_DISABLE=ON -- Setting default Piro_AnalysisDriverTpetra_MPI_4_DISABLE=ON -- Setting default ROL_adapters_tpetra_test_vector_SimulatedVectorTpetraBatchManagerInterface_EXE_DISABLE=ON -- Setting default ROL_adapters_tpetra_test_vector_SimulatedVectorTpetraBatchManagerInterface_MPI_4_DISABLE=ON -- Setting default KokkosCore_UnitTest_CudaTimingBased_MPI_1_DISABLE=ON -- Setting default ROL_example_PDE-OPT_0ld_adv-diff-react_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_0ld_adv-diff-react_example_02_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_0ld_poisson_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_0ld_stefan-boltzmann_example_03_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_navier-stokes_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_navier-stokes_example_02_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_nonlinear-elliptic_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_nonlinear-elliptic_example_02_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_obstacle_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_stefan-boltzmann_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_stefan-boltzmann_example_03_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_topo-opt_poisson_example_01_MPI_4_DISABLE=ON -- Setting default ROL_test_elementwise_TpetraMultiVector_MPI_4_DISABLE=ON -- Setting default TrilinosCouplings_Example_Maxwell_MueLu_MPI_1_DISABLE=ON -- Setting default TrilinosCouplings_Example_Maxwell_MueLu_MPI_4_DISABLE=ON -- Setting default Zoltan_ch_ewgt_zoltan_parallel_DISABLE=ON -- Setting default Zoltan_ch_grid20x19_zoltan_parallel_DISABLE=ON -- Setting default Zoltan_ch_nograph_zoltan_parallel_DISABLE=ON -- Setting default Zoltan_ch_simple_zoltan_parallel_DISABLE=ON -- Setting default KokkosContainers_PerformanceTest_Cuda_DISABLE=ON -- Setting default KokkosCore_UnitTest_CudaInterOpStreams_MPI_1_SET_RUN_SERIAL=ON -- Setting default KokkosCore_UnitTest_CudaInterOpInit_MPI_1_SET_RUN_SERIAL=ON -- CMAKE_HOST_SYSTEM_NAME='Linux' -- Trilinos_HOSTNAME='kokkos-dev.sandia.gov' -- Found PythonInterp: /usr/bin/python (found suitable version "2.7.5", minimum required is "2.6") -- PYTHON_EXECUTABLE='/usr/bin/python' -- Found Git: /projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/bin/git (found version "2.10.1") Setting up major user options ... Trilinos repos versions: -------------------------------------------------------------------------------- *** Base Git Repo: Trilinos 3f683fc [Mon Jun 7 09:54:07 2021 -0600] Intrepid2: Structured Integration Performance Test (add OpenMP test case) (#9226 -------------------------------------------------------------------------------- Reading list of native TPLs from /ascldap/users/ndellin/trilinos/Trilinos/TPLsList.cmake -- Trilinos_NUM_TPLS='116' Reading list of native packages from /ascldap/users/ndellin/trilinos/Trilinos/PackagesList.cmake -- Trilinos_NUM_PACKAGES='64' Processing Project, Repository, and Package dependency files and building internal dependencies graph ... -- Trilinos_NUM_SE_PACKAGES='159' -- Tentatively enabling TPL 'CUBLAS' -- Tentatively enabling TPL 'CUSOLVER' -- Tentatively enabling TPL 'DLlib' Explicitly enabled packages on input (by user): Kokkos KokkosKernels Tpetra 3 Explicitly enabled SE packages on input (by user): Kokkos KokkosKernels Tpetra 3 Explicitly disabled packages on input (by user or by default): TrilinosFrameworkTests MiniTensor Domi Isorropia Pliris ShyLU_Node Komplex FEI TriKota Compadre Moertel ShyLU_DD ShyLU Tempus Stokhos ROL PyTrilinos NewPackage TrilinosCouplings Pike 20 Explicitly disabled SE packages on input (by user or by default): TrilinosFrameworkTests MiniTensor Domi Isorropia Pliris ShyLU_NodeTacho ShyLU_NodeBasker ShyLU_NodeFastILU ShyLU_Node SEACASExotec2 SEACASSlice Komplex FEI TriKota Compadre STKSearchUtil STKUnit_tests STKDoc_tests Moertel ShyLU_DDCore ShyLU_DD ShyLU Tempus Stokhos ROL PanzerExprEval PyTrilinos NewPackage TrilinosCouplings Pike 30 Explicitly enabled TPLs on input (by user): CUDA CUBLAS CUSOLVER CUSPARSE MPI BLAS LAPACK Boost HDF5 Netcdf BoostLib DLlib 12 Explicitly disabled TPLs on input (by user or by default): yaml-cpp Pthread HWLOC BinUtils METIS ParMETIS CGNS SuperLUDist SuperLU Matio X11 GLM 12 Disabling all packages that have a required dependency on disabled TPLs and optional package TPL support based on TPL_ENABLE_=OFF ... -- Setting Trilinos_ENABLE_SEACASSVDI=OFF because SEACASSVDI has a required library dependence on disabled TPL X11 -- Setting Trilinos_ENABLE_SEACASExo2mat=OFF because SEACASExo2mat has a required library dependence on disabled TPL Matio -- Setting Trilinos_ENABLE_SEACASMat2exo=OFF because SEACASMat2exo has a required library dependence on disabled TPL Matio Disabling subpackages for hard disables of parent packages due to Trilinos_ENABLE_=OFF ... -- Setting subpackage enable Trilinos_ENABLE_ShyLU_NodeHTS=OFF because parent package Trilinos_ENABLE_ShyLU_Node=OFF -- Setting subpackage enable Trilinos_ENABLE_ShyLU_DDBDDC=OFF because parent package Trilinos_ENABLE_ShyLU_DD=OFF -- Setting subpackage enable Trilinos_ENABLE_ShyLU_DDFROSch=OFF because parent package Trilinos_ENABLE_ShyLU_DD=OFF -- Setting subpackage enable Trilinos_ENABLE_ShyLU_DDCommon=OFF because parent package Trilinos_ENABLE_ShyLU_DD=OFF -- Setting subpackage enable Trilinos_ENABLE_PikeBlackBox=OFF because parent package Trilinos_ENABLE_Pike=OFF -- Setting subpackage enable Trilinos_ENABLE_PikeImplicit=OFF because parent package Trilinos_ENABLE_Pike=OFF Disabling forward required SE packages and optional intra-package support that have a dependancy on disabled SE packages Trilinos_ENABLE_=OFF ... -- Setting Trilinos_ENABLE_SEACASPLT=OFF because SEACASPLT has a required library dependence on disabled package SEACASSVDI -- Setting Trilinos_ENABLE_SEACASBlot=OFF because SEACASBlot has a required library dependence on disabled package SEACASPLT -- Setting Trilinos_ENABLE_SEACASFastq=OFF because SEACASFastq has a required library dependence on disabled package SEACASPLT Enabling subpackages for hard enables of parent packages due to Trilinos_ENABLE_=ON ... -- Setting subpackage enable Trilinos_ENABLE_KokkosCore=ON because parent package Trilinos_ENABLE_Kokkos=ON -- Setting subpackage enable Trilinos_ENABLE_KokkosContainers=ON because parent package Trilinos_ENABLE_Kokkos=ON -- Setting subpackage enable Trilinos_ENABLE_KokkosAlgorithms=ON because parent package Trilinos_ENABLE_Kokkos=ON -- Setting subpackage enable Trilinos_ENABLE_TpetraClassic=ON because parent package Trilinos_ENABLE_Tpetra=ON -- Setting subpackage enable Trilinos_ENABLE_TpetraTSQR=ON because parent package Trilinos_ENABLE_Tpetra=ON -- Setting subpackage enable Trilinos_ENABLE_TpetraCore=ON because parent package Trilinos_ENABLE_Tpetra=ON Enabling all required (and optional since Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES=ON) upstream SE packages for current set of enabled packages (Trilinos_ENABLE_SECONDARY_TESTED_CODE=ON) ... -- Setting Trilinos_ENABLE_Teuchos=ON because TpetraCore has a required dependence on Teuchos -- Setting Trilinos_ENABLE_TeuchosKokkosCompat=ON because TpetraCore has a required dependence on TeuchosKokkosCompat -- Setting Trilinos_ENABLE_TeuchosKokkosComm=ON because TpetraCore has a required dependence on TeuchosKokkosComm -- Setting Trilinos_ENABLE_Epetra=ON because TpetraCore has an optional dependence on Epetra -- Setting Trilinos_ENABLE_TeuchosNumerics=ON because TpetraCore has an optional dependence on TeuchosNumerics -- Setting Trilinos_ENABLE_TeuchosCore=ON because Teuchos has a required dependence on TeuchosCore -- Setting Trilinos_ENABLE_TeuchosParser=ON because Teuchos has a required dependence on TeuchosParser -- Setting Trilinos_ENABLE_TeuchosParameterList=ON because Teuchos has a required dependence on TeuchosParameterList -- Setting Trilinos_ENABLE_TeuchosComm=ON because Teuchos has a required dependence on TeuchosComm -- Setting Trilinos_ENABLE_TeuchosRemainder=ON because Teuchos has a required dependence on TeuchosRemainder Enabling all optional intra-package enables _ENABLE_ that are not currently disabled if both sets of packages are enabled ... -- Setting Kokkos_ENABLE_KokkosContainers=ON since Trilinos_ENABLE_Kokkos=ON AND Trilinos_ENABLE_KokkosContainers=ON -- Setting Kokkos_ENABLE_KokkosAlgorithms=ON since Trilinos_ENABLE_Kokkos=ON AND Trilinos_ENABLE_KokkosAlgorithms=ON -- Setting TeuchosCore_ENABLE_KokkosCore=ON since Trilinos_ENABLE_TeuchosCore=ON AND Trilinos_ENABLE_KokkosCore=ON -- Setting Teuchos_ENABLE_TeuchosKokkosCompat=ON since Trilinos_ENABLE_Teuchos=ON AND Trilinos_ENABLE_TeuchosKokkosCompat=ON -- Setting Teuchos_ENABLE_TeuchosKokkosComm=ON since Trilinos_ENABLE_Teuchos=ON AND Trilinos_ENABLE_TeuchosKokkosComm=ON -- Setting Epetra_ENABLE_Teuchos=ON since Trilinos_ENABLE_Epetra=ON AND Trilinos_ENABLE_Teuchos=ON -- Setting TpetraCore_ENABLE_Epetra=ON since Trilinos_ENABLE_TpetraCore=ON AND Trilinos_ENABLE_Epetra=ON -- Setting TpetraCore_ENABLE_TpetraTSQR=ON since Trilinos_ENABLE_TpetraCore=ON AND Trilinos_ENABLE_TpetraTSQR=ON -- Setting TpetraCore_ENABLE_TeuchosNumerics=ON since Trilinos_ENABLE_TpetraCore=ON AND Trilinos_ENABLE_TeuchosNumerics=ON -- Setting Tpetra_ENABLE_TpetraTSQR=ON since Trilinos_ENABLE_Tpetra=ON AND Trilinos_ENABLE_TpetraTSQR=ON Enabling all remaining required TPLs for current set of enabled packages ... Enabling all optional package TPL support _ENABLE_ not currently disabled for enabled TPLs ... -- Setting KokkosCore_ENABLE_CUDA=ON since TPL_ENABLE_CUDA=ON -- Setting KokkosCore_ENABLE_DLlib=ON since TPL_ENABLE_DLlib=ON -- Setting KokkosCore_ENABLE_CUSPARSE=ON since TPL_ENABLE_CUSPARSE=ON -- Setting KokkosContainers_ENABLE_CUDA=ON since TPL_ENABLE_CUDA=ON -- Setting KokkosContainers_ENABLE_CUSPARSE=ON since TPL_ENABLE_CUSPARSE=ON -- Setting KokkosAlgorithms_ENABLE_CUDA=ON since TPL_ENABLE_CUDA=ON -- Setting KokkosAlgorithms_ENABLE_CUSPARSE=ON since TPL_ENABLE_CUSPARSE=ON -- Setting TeuchosCore_ENABLE_Boost=ON since TPL_ENABLE_Boost=ON -- Setting TeuchosCore_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON -- Setting TeuchosKokkosComm_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON -- Setting KokkosKernels_ENABLE_BLAS=ON since TPL_ENABLE_BLAS=ON -- Setting KokkosKernels_ENABLE_LAPACK=ON since TPL_ENABLE_LAPACK=ON -- Setting KokkosKernels_ENABLE_CUSPARSE=ON since TPL_ENABLE_CUSPARSE=ON -- Setting Epetra_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON -- Setting TpetraTSQR_ENABLE_CUBLAS=ON since TPL_ENABLE_CUBLAS=ON -- Setting TpetraTSQR_ENABLE_CUSOLVER=ON since TPL_ENABLE_CUSOLVER=ON -- Setting TpetraCore_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON -- Setting TpetraCore_ENABLE_CUDA=ON since TPL_ENABLE_CUDA=ON Enabling TPLs based on _ENABLE_=ON if TPL is not explicitly disabled ... Set cache entries for optional packages/TPLs and tests/examples for packages actually enabled ... Enabling the shell of non-enabled parent packages (mostly for show) that have at least one subpackage enabled ... Final set of enabled packages: Kokkos Teuchos KokkosKernels Epetra Tpetra 5 Final set of enabled SE packages: KokkosCore KokkosContainers KokkosAlgorithms Kokkos TeuchosCore TeuchosParser TeuchosParameterList TeuchosComm TeuchosNumerics TeuchosRemainder TeuchosKokkosCompat TeuchosKokkosComm Teuchos KokkosKernels Epetra TpetraClassic TpetraTSQR TpetraCore Tpetra 19 Final set of non-enabled packages: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest RTOp Sacado MiniTensor Zoltan Shards Triutils EpetraExt TrilinosSS Domi Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_Node Amesos2 SEACAS Komplex Anasazi Ifpack2 Stratimikos FEI Teko TriKota Intrepid Intrepid2 Compadre STK Percept Phalanx NOX Moertel MueLu Zoltan2Sphynx Zoltan2 ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro Panzer PyTrilinos NewPackage Adelus TrilinosCouplings Pike TrilinosBuildStats 59 Final set of non-enabled SE packages: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest RTOp Sacado MiniTensor Zoltan Shards Triutils EpetraExt TrilinosSS Domi ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_NodeHTS ShyLU_NodeTacho ShyLU_NodeBasker ShyLU_NodeFastILU ShyLU_Node Amesos2 SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASNemesis SEACASIoss SEACASChaco SEACASAprepro_lib SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASPLT SEACASAlgebra SEACASAprepro SEACASBlot SEACASConjoin SEACASEjoin SEACASEpu SEACASExo2mat SEACASExodiff SEACASExomatlab SEACASExotxt SEACASExo_format SEACASEx1ex2v2 SEACASExotec2 SEACASFastq SEACASGjoin SEACASGen3D SEACASGenshell SEACASGrepos SEACASExplore SEACASMapvarlib SEACASMapvar SEACASMapvar-kd SEACASMat2exo SEACASNas2exo SEACASZellij SEACASNemslice SEACASNemspread SEACASNumbers SEACASSlice SEACASTxtexo SEACASEx2ex1v2 SEACAS Komplex Anasazi Ifpack2 Stratimikos FEI Teko TriKota Intrepid Intrepid2 Compadre STKUtil STKCoupling STKMath STKSimd STKNGP_TEST STKTopology STKMesh STKIO STKUnit_test_utils STKSearch STKSearchUtil STKTransfer STKTools STKBalance STKUnit_tests STKDoc_tests STKExprEval STK Percept Phalanx NOX Moertel MueLu Zoltan2Sphynx Zoltan2 ShyLU_DDBDDC ShyLU_DDFROSch ShyLU_DDCore ShyLU_DDCommon ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro PanzerCore PanzerDofMgr PanzerDiscFE PanzerAdaptersSTK PanzerMiniEM PanzerExprEval Panzer PyTrilinos NewPackage Adelus TrilinosCouplings PikeBlackBox PikeImplicit Pike TrilinosBuildStats 140 Final set of enabled TPLs: CUDA CUBLAS CUSOLVER CUSPARSE MPI BLAS LAPACK Boost HDF5 Netcdf BoostLib DLlib 12 Final set of non-enabled TPLs: MKL yaml-cpp Peano Thrust Cusp TBB Pthread HWLOC QTHREAD BinUtils ARPREC QD Scotch OVIS gpcd METIS MTMETIS ParMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF Zlib CGNS Pnetcdf ADIOS2 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS STRUMPACK PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Clp GLPK qpOASES Matio PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX CGAL CGALCore VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField Valgrind QUO ViennaCL Avatar mlpack pebbl MAGMASparse Check SARMA 104 Setting up export dependencies for all enabled SE packages ... Probing the environment ... -- USE_XSDK_DEFAULTS='FALSE' -- BUILD_SHARED_LIBS='OFF' -- CMAKE_BUILD_TYPE='RELEASE' -- MPI_USE_COMPILER_WRAPPERS='ON' -- Leaving current CMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/7.2.0/openmpi/4.0.2/bin/mpicc since it is already set! -- Leaving current CMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/7.2.0/openmpi/4.0.2/bin/mpicxx since it is already set! -- MPI_EXEC='mpiexec' -- MPI_EXEC='/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/7.2.0/openmpi/4.0.2/bin/mpiexec' -- The C compiler identification is GNU 7.2.0 -- Check for working C compiler: /projects/sems/install/rhel7-x86_64/sems/compiler/gcc/7.2.0/openmpi/4.0.2/bin/mpicc -- Check for working C compiler: /projects/sems/install/rhel7-x86_64/sems/compiler/gcc/7.2.0/openmpi/4.0.2/bin/mpicc - works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- CMAKE_C_COMPILER_ID='GNU' -- CMAKE_C_COMPILER_VERSION='7.2.0' -- The CXX compiler identification is GNU 7.2.0 -- Check for working CXX compiler: /projects/sems/install/rhel7-x86_64/sems/compiler/gcc/7.2.0/openmpi/4.0.2/bin/mpicxx -- Check for working CXX compiler: /projects/sems/install/rhel7-x86_64/sems/compiler/gcc/7.2.0/openmpi/4.0.2/bin/mpicxx - works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- CMAKE_CXX_COMPILER_ID='GNU' -- CMAKE_CXX_COMPILER_VERSION='7.2.0' -- Skip adding flags for OpenMP because Kokkos flags does that ... -- Trilinos_SET_INSTALL_RPATH='TRUE' -- CMAKE_INSTALL_RPATH_USE_LINK_PATH='TRUE' -- Setting default for CMAKE_INSTALL_RPATH pointing to Trilinos_INSTALL_LIB_DIR -- CMAKE_INSTALL_RPATH='/ascldap/users/ndellin/trilinos/Trilinos/BUILD/Test/install/lib' -- Trilinos_MAKE_INSTALL_GROUP_READABLE='TRUE' -- Trilinos_MAKE_INSTALL_WORLD_READABLE='ON' -- CMAKE_INSTALL_DEFAULT_DIRECTORY_PERMISSIONS = (OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE) -- Looking for C++ include sys/time.h -- Looking for C++ include sys/time.h - found -- Looking for C++ include time.h -- Looking for C++ include time.h - found -- Looking for C++ include stdint.h -- Looking for C++ include stdint.h - found -- Looking for C++ include inttypes.h -- Looking for C++ include inttypes.h - found -- Found Perl: /usr/bin/perl (found version "5.16.3") -- Performing Test MATH_LIBRARY_IS_SUPPLIED -- Performing Test MATH_LIBRARY_IS_SUPPLIED - Success -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN - Success -- Performing Test FINITE_VALUE_HAVE_STD_ISNAN -- Performing Test FINITE_VALUE_HAVE_STD_ISNAN - Success -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF - Success -- Performing Test FINITE_VALUE_HAVE_STD_ISINF -- Performing Test FINITE_VALUE_HAVE_STD_ISINF - Success -- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") found components: doxygen missing components: dot Getting information for all enabled TPLs ... Processing enabled TPL: CUDA (enabled explicitly, disable with -DTPL_ENABLE_CUDA=OFF) -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE -- Found CUDA: /projects/sems/install/rhel7-x86_64/sems/compiler/cuda/9.2/base (found version "9.2") Processing enabled TPL: CUBLAS (enabled explicitly, disable with -DTPL_ENABLE_CUBLAS=OFF) Processing enabled TPL: CUSOLVER (enabled explicitly, disable with -DTPL_ENABLE_CUSOLVER=OFF) Processing enabled TPL: CUSPARSE (enabled explicitly, disable with -DTPL_ENABLE_CUSPARSE=OFF) Processing enabled TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF) Processing enabled TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF) -- BLAS_LIBRARY_NAMES='blas blas_win32' -- TPL_BLAS_LIBRARIES='/projects/sems/install/rhel7-x86_64/atdm/tpl/openblas/0.3.6/gcc/7.2.0/lib/libopenblas.so' Processing enabled TPL: LAPACK (enabled explicitly, disable with -DTPL_ENABLE_LAPACK=OFF) -- LAPACK_LIBRARY_NAMES='lapack lapack_win32' -- TPL_LAPACK_LIBRARIES='/projects/sems/install/rhel7-x86_64/atdm/tpl/openblas/0.3.6/gcc/7.2.0/lib/libopenblas.so' Processing enabled TPL: Boost (enabled explicitly, disable with -DTPL_ENABLE_Boost=OFF) -- Searching for headers in Boost_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include' -- Searching for a header file in the set "boost/version.hpp": -- Searching for header 'boost/version.hpp' ... -- Found header '/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include/boost/version.hpp' -- Searching for a header file in the set "boost/mpl/at.hpp": -- Searching for header 'boost/mpl/at.hpp' ... -- Found header '/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include/boost/mpl/at.hpp' -- Found TPL 'Boost' include dirs '/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include' -- TPL_Boost_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include' Processing enabled TPL: HDF5 (enabled explicitly, disable with -DTPL_ENABLE_HDF5=OFF) -- HDF5_LIBRARY_NAMES='hdf5;z;hdf5_hl' -- TPL_HDF5_LIBRARIES='/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/7.2.0/openmpi/4.0.2/parallel/lib/libhdf5_hl.a;/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/7.2.0/openmpi/4.0.2/parallel/lib/libhdf5.a;/projects/sems/install/rhel7-x86_64/sems/tpl/zlib/1.2.8/gcc/7.2.0/base/lib/libz.a;-ldl' -- Searching for headers in HDF5_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/7.2.0/openmpi/4.0.2/parallel/include' -- Searching for a header file in the set "hdf5.h": -- Searching for header 'hdf5.h' ... -- Found header '/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/7.2.0/openmpi/4.0.2/parallel/include/hdf5.h' -- Found TPL 'HDF5' include dirs '/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/7.2.0/openmpi/4.0.2/parallel/include' -- TPL_HDF5_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/7.2.0/openmpi/4.0.2/parallel/include' Processing enabled TPL: Netcdf (enabled explicitly, disable with -DTPL_ENABLE_Netcdf=OFF) -- Netcdf_LIBRARY_NAMES='netcdf;pnetcdf' -- TPL_Netcdf_LIBRARIES='/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/7.2.0/openmpi/4.0.2/parallel/lib/libnetcdf.a;/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/7.2.0/openmpi/4.0.2/parallel/lib/libpnetcdf.a;/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/7.2.0/openmpi/4.0.2/parallel/lib/libhdf5_hl.a;/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/7.2.0/openmpi/4.0.2/parallel/lib/libhdf5.a;/projects/sems/install/rhel7-x86_64/sems/tpl/zlib/1.2.8/gcc/7.2.0/base/lib/libz.a;-ldl;-lcurl' -- Searching for headers in Netcdf_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/7.2.0/openmpi/4.0.2/parallel/include' -- Searching for a header file in the set "netcdf.h": -- Searching for header 'netcdf.h' ... -- Found header '/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/7.2.0/openmpi/4.0.2/parallel/include/netcdf.h' -- Found TPL 'Netcdf' include dirs '/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/7.2.0/openmpi/4.0.2/parallel/include' -- TPL_Netcdf_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/7.2.0/openmpi/4.0.2/parallel/include' -- TPL_Netcdf_PARALLEL is ON Processing enabled TPL: BoostLib (enabled explicitly, disable with -DTPL_ENABLE_BoostLib=OFF) -- BoostLib_LIBRARY_NAMES='boost_program_options;boost_system' -- TPL_BoostLib_LIBRARIES='/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/lib/libboost_program_options.a;/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/lib/libboost_system.a' -- Searching for headers in BoostLib_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include' -- Searching for a header file in the set "boost/version.hpp": -- Searching for header 'boost/version.hpp' ... -- Found header '/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include/boost/version.hpp' -- Searching for a header file in the set "boost/mpl/at.hpp": -- Searching for header 'boost/mpl/at.hpp' ... -- Found header '/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include/boost/mpl/at.hpp' -- Found TPL 'BoostLib' include dirs '/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include' -- TPL_BoostLib_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/boost/1.59.0/gcc/7.2.0/base/include' Processing enabled TPL: DLlib (enabled explicitly, disable with -DTPL_ENABLE_DLlib=OFF) -- Attempting to tentatively enable TPL 'DLlib' ... -- DLlib_LIBRARY_NAMES='dl' -- TPL_DLlib_LIBRARIES='-ldl' -- Attempt to tentatively enable TPL 'DLlib' passed! Setting up testing support ... -- CTEST_DROP_METHOD='http' -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' Configuring individual enabled Trilinos packages ... Processing enabled package: Kokkos (Core, Containers, Algorithms) -- Setting default Kokkos CXX standard to 14 -- Setting policy CMP0074 to use _ROOT variables -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- The project name is: Trilinos -- Compiler Version: 9.2.148 -- Using -std=c++14 for C++14 standard as feature -- Built-in Execution Spaces: -- Device Parallel: Kokkos::Cuda -- Host Parallel: NoTypeDefined -- Host Serial: SERIAL -- -- Architectures: -- KEPLER35 -- Kokkos Devices: CUDA;SERIAL, Kokkos Backends: CUDA;SERIAL Processing enabled package: Teuchos (Core, Parser, ParameterList, Comm, Numerics, Remainder, KokkosCompat, KokkosComm) -- Performing Test HAVE_GCC_ABI_DEMANGLE -- Performing Test HAVE_GCC_ABI_DEMANGLE - Success -- Performing Test HAVE_TEUCHOSCORE_BOOST_IS_POLYMORPHIC -- Performing Test HAVE_TEUCHOSCORE_BOOST_IS_POLYMORPHIC - Success -- Performing Test HAVE_TEUCHOS_BLASFLOAT -- Performing Test HAVE_TEUCHOS_BLASFLOAT - Success -- Performing Test LAPACK_SLAPY2_WORKS -- Performing Test LAPACK_SLAPY2_WORKS - Success -- Performing Test HAVE_TEUCHOS_LAPACKLARND -- Performing Test HAVE_TEUCHOS_LAPACKLARND - Success -- Performing Test HAVE_CXX_ATTRIBUTE_CONSTRUCTOR -- Performing Test HAVE_CXX_ATTRIBUTE_CONSTRUCTOR - Success -- C++ compiler supports __attribute__((constructor)) syntax -- Performing Test HAVE_CXX_ATTRIBUTE_WEAK -- Performing Test HAVE_CXX_ATTRIBUTE_WEAK - Success -- C++ compiler supports __attribute__((weak)) syntax and testing weak functions -- Performing Test HAVE_CXX_PRAGMA_WEAK -- Performing Test HAVE_CXX_PRAGMA_WEAK - Failed -- C++ compiler does NOT support #pragma weak syntax and testing weak functions -- Wrapping every Teuchos timer with a Kokkos profiling region. Processing enabled package: KokkosKernels (Libs, Tests) -- Setting policy CMP0074 to use _ROOT variables -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- The project name is: Trilinos -- CTEST_DROP_SITE='testing.sandia.gov' -- CTEST_PROJECT_NAME='Trilinos' -- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' -- CTEST_TRIGGER_SITE='' -- CTEST_DROP_SITE_CDASH='TRUE' -- TRIBITS_2ND_CTEST_DROP_SITE='testing-dev.sandia.gov' -- TRIBITS_2ND_CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos' Overriding KOKKOSKERNELS_ENABLE_TPL_BLAS=OFF with TPL_ENABLE_BLAS=ON Overriding KOKKOSKERNELS_ENABLE_TPL_LAPACK=OFF with TPL_ENABLE_LAPACK=ON -- Performing Test KOKKOSKERNELS_TPL_BLAS_RETURN_COMPLEX -- Performing Test KOKKOSKERNELS_TPL_BLAS_RETURN_COMPLEX - Success ======================= KokkosKernels ETI Types Devices: ;; Scalars: double Ordinals: int Offsets: int;size_t Layouts: LayoutLeft KokkosKernels TPLs BLAS: LAPACK: CUBLAS: CUSPARSE: ======================= -- Creating ETI files for Blas1_abs -- Creating ETI files for Blas1_abs_mv -- Creating ETI files for Blas1_scal -- Creating ETI files for Blas1_scal_mv -- Creating ETI files for Blas1_dot -- Creating ETI files for Blas1_dot_mv -- Creating ETI files for Blas_gesv -- Creating ETI files for Blas1_axpby -- Creating ETI files for Blas1_axpby_mv -- Creating ETI files for Blas1_update -- Creating ETI files for Blas1_update_mv -- Creating ETI files for Blas1_sum -- Creating ETI files for Blas1_sum_mv -- Creating ETI files for Blas1_nrm1 -- Creating ETI files for Blas1_nrm1_mv -- Creating ETI files for Blas1_nrm2w -- Creating ETI files for Blas1_nrm2w_mv -- Creating ETI files for Blas1_nrminf -- Creating ETI files for Blas1_nrminf_mv -- Creating ETI files for Blas1_iamax -- Creating ETI files for Blas1_iamax_mv -- Creating ETI files for Blas1_nrm2 -- Creating ETI files for Blas1_nrm2_mv -- Creating ETI files for Blas1_mult -- Creating ETI files for Blas1_mult_mv -- Creating ETI files for Blas1_reciprocal -- Creating ETI files for Blas1_reciprocal_mv -- Creating ETI files for Blas2_gemv -- Creating ETI files for Blas3_gemm -- Creating ETI files for Blas3_trsm -- Creating ETI files for Blas3_trmm -- Creating ETI files for Blas_trtri -- Creating ETI files for Sparse_sptrsv_solve -- Creating ETI files for Sparse_spmv_struct -- Creating ETI files for Sparse_spmv_mv_struct -- Creating ETI files for Sparse_spmv -- Creating ETI files for Sparse_spmv_mv -- Creating ETI files for Sparse_spgemm_symbolic -- Creating ETI files for Sparse_spgemm_numeric -- Creating ETI files for Sparse_spgemm_jacobi -- Creating ETI files for Sparse_spiluk_symbolic -- Creating ETI files for Sparse_spiluk_numeric -- Creating ETI files for Sparse_sptrsv_symbolic -- Creating ETI files for Sparse_trsv -- Creating ETI files for Sparse_gauss_seidel_symbolic -- Creating ETI files for Sparse_gauss_seidel_numeric -- Creating ETI files for Sparse_gauss_seidel_apply Processing enabled package: Epetra (Libs) -- Looking for dggsvd3 -- Looking for dggsvd3 - not found -- Looking for dggsvd3_ -- Looking for dggsvd3_ - found -- Looking for DGGSVD3 -- Looking for DGGSVD3 - not found -- Looking for DGGSVD3_ -- Looking for DGGSVD3_ - not found Found new version of lapack. dggsvd3 is available. Processing enabled package: Tpetra (Classic, TSQR, Core, Tests) -- Tpetra: Enabling deprecated code CUDA_VERSION=9.2 -- Determine whether Tpetra will assume that MPI is CUDA aware: -- - Attempt to detect whether MPI is CUDA aware, by searching for "ompi_info" executable. -- - Calling "ompi_info" (full path: /projects/sems/install/rhel7-x86_64/sems/compiler/gcc/7.2.0/openmpi/4.0.2/bin/ompi_info) to find out whether MPI is CUDA aware. -- - "ompi_info" explicitly claims that your MPI implementation is NOT CUDA aware. You may want to use a different OpenMPI installation that is CUDA aware, or reconfigure and rebuild OpenMPI in order to make it CUDA aware. For details, please refer to OpenMPI's website: https://www.open-mpi.org/faq/?category=runcuda -- Tpetra execution space availability (ON means available): -- - Serial: ON -- - Threads: OFF -- - OpenMP: OFF -- - Cuda: ON -- - HIP: off -- Tpetra: Tpetra_INST_INT_LONG_LONG is enabled by default. -- Tpetra: Tpetra_INST_INT_UNSIGNED is disabled by default. -- Tpetra: Tpetra_INST_INT_UNSIGNED_LONG is disabled by default. -- Tpetra: Tpetra_INST_INT_INT is disabled by default. -- Tpetra: Tpetra_INST_INT_LONG is disabled by default. -- -- Tpetra: Validate global ordinal setting ... -- Tpetra: global ordinal setting is OK -- -- Setting default Node to Kokkos::Compat::KokkosCudaWrapperNode. -- Tpetra: Performance-CGSolve test ENABLED -- Tpetra: Performance-CGSolve CUDA_LAUNCH_BLOCKING test ENABLED -- TpetraCore_TpetraUtils_UnitTests: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'! -- TpetraCore_TpetraUtils_merge: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'! -- TpetraCore_createPrefix_no_MPI: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'! Processing explicit instantiation support for enabled packages ... Processing ETI support: TpetraCore -- TpetraCore: Processing ETI / test support -- Enabled Scalar types: long long|double -- Enabled LocalOrdinal types: int -- Enabled GlobalOrdinal types: long long -- Enabled Node types: Kokkos::Compat::KokkosSerialWrapperNode|Kokkos::Compat::KokkosCudaWrapperNode -- Excluded type combinations: : -- Set of enabled types, before exclusions: S={long long} N={Kokkos::Compat::KokkosSerialWrapperNode} LO={int} GO={long long};S={long long} N={Kokkos::Compat::KokkosCudaWrapperNode} LO={int} GO={long long};S={double} N={Kokkos::Compat::KokkosSerialWrapperNode} LO={int} GO={long long};S={double} N={Kokkos::Compat::KokkosCudaWrapperNode} LO={int} GO={long long} Generating dummy makefiles in each directory to call Ninja ... Set up for creating a distribution ... Finished configuring Trilinos! -- Configuring done -- Generating done -- Build files have been written to: /ascldap/users/ndellin/trilinos/Trilinos/BUILD/Test [ndellin@kokkos-dev Test]$ ```
brian-kelley commented 3 years ago

@ndellingwood I'm taking a look at this. @jennloe had the same issue on Weaver, also with cuda 9.2. This seems to be a compiler bug btw, same symptoms as this: https://github.com/pytorch/pytorch/issues/3243 but if it's just a particular usage of decltype, it should be easy to work around.

ndellingwood commented 3 years ago

If useful, here is a more generic configuration (not connected to kokkos-dev, and assuming blas+lapack libs don't need to be explicitly provided):

module load sems-cuda/9.2 sems-gcc/7.2.0 sems-cmake/3.17.1 sems-openmpi/4.0.2

export OMPI_CXX=$TRILINOS_DIR/packages/kokkos/bin/nvcc_wrapper

cmake \
 -DCMAKE_CXX_COMPILER="$OMPI_CXX" \
 -DCMAKE_C_COMPILER="`which gcc`" \
 -DCMAKE_Fortran_COMPILER="`which gfortran`" \
 -DCMAKE_INSTALL_PREFIX="${PWD}/install" \
 -DCMAKE_CXX_STANDARD="14" \
 -DTrilinos_ENABLE_TESTS=OFF \
 -DTrilinos_ENABLE_ALL_PACKAGES=OFF \
 -DTrilinos_ENABLE_Kokkos=ON \
  -DKokkos_ENABLE_CUDA=ON \
  -DKokkos_ARCH_KEPLER35=ON \
  -DKokkos_ENABLE_CUDA_UVM=ON \
  -DKokkos_ENABLE_CUDA_LAMBDA=ON \
 -DTrilinos_ENABLE_KokkosKernels=ON \
 -DTrilinos_ENABLE_Tpetra=ON \
   -DTpetra_ENABLE_TESTS=ON \
 -DTPL_ENABLE_BLAS:BOOL=ON \
 -DTPL_ENABLE_LAPACK:BOOL=ON \
$TRILINOS_DIR
kddevin commented 3 years ago

Switching the decltype to 

WrappedDualViewType::DeviceViewType::rank

fixed the problem for me. @brian-kelley Should I PR that change?

brian-kelley commented 3 years ago

@kddevin Yes, I think that's the fix.

ndellingwood commented 3 years ago

9238 resolved the issue, thanks!