Closed ndellingwood closed 3 years ago
@ndellingwood I'm taking a look at this. @jennloe had the same issue on Weaver, also with cuda 9.2. This seems to be a compiler bug btw, same symptoms as this: https://github.com/pytorch/pytorch/issues/3243 but if it's just a particular usage of decltype, it should be easy to work around.
If useful, here is a more generic configuration (not connected to kokkos-dev, and assuming blas+lapack libs don't need to be explicitly provided):
module load sems-cuda/9.2 sems-gcc/7.2.0 sems-cmake/3.17.1 sems-openmpi/4.0.2
export OMPI_CXX=$TRILINOS_DIR/packages/kokkos/bin/nvcc_wrapper
cmake \
-DCMAKE_CXX_COMPILER="$OMPI_CXX" \
-DCMAKE_C_COMPILER="`which gcc`" \
-DCMAKE_Fortran_COMPILER="`which gfortran`" \
-DCMAKE_INSTALL_PREFIX="${PWD}/install" \
-DCMAKE_CXX_STANDARD="14" \
-DTrilinos_ENABLE_TESTS=OFF \
-DTrilinos_ENABLE_ALL_PACKAGES=OFF \
-DTrilinos_ENABLE_Kokkos=ON \
-DKokkos_ENABLE_CUDA=ON \
-DKokkos_ARCH_KEPLER35=ON \
-DKokkos_ENABLE_CUDA_UVM=ON \
-DKokkos_ENABLE_CUDA_LAMBDA=ON \
-DTrilinos_ENABLE_KokkosKernels=ON \
-DTrilinos_ENABLE_Tpetra=ON \
-DTpetra_ENABLE_TESTS=ON \
-DTPL_ENABLE_BLAS:BOOL=ON \
-DTPL_ENABLE_LAPACK:BOOL=ON \
$TRILINOS_DIR
Switching the decltype to
WrappedDualViewType::DeviceViewType::rank
fixed the problem for me. @brian-kelley Should I PR that change?
@kddevin Yes, I think that's the fix.
Bug Report
@trilinos/tpetra
Description
Following merge of PR #9038 some nightly testing with Trilinos began failing with cuda/9.2 + gcc/7.2.0 (sems modules) on Kepler architecture with error messages like:
I reproduced this on develop branch with the SHA posted below. This was detected on the kokkos-dev test machine, but since it uses sems modules I'm assuming this could be reproduced elsewhere that has the sems env available.
Steps to Reproduce
SHA1: 3f683fc90aa9740f002b56190cbe883f0e188b9d
Modules:
Configure script: (note this loads modules/env specific to kokkos-dev)
Configure output
Expand dropdown
``` [ndellin@kokkos-dev Test]$ . ./configure-nightly.sh 2>&1 | tee configure.out Selecting custom system configuration 'kokkos-dev' Setting compiler and build options for build-name 'kokkos-dev-cuda-opt' CUDA ATDM COMPILER: CUDA Using SEMS RHEL7 compiler stack CUDA-9.2 to build RELEASE code with Kokkos node type CUDA Currently Loaded Modulefiles: 1) sems-env 8) atdm-openblas/0.3.6 2) sems-git/2.10.1 9) sems-openmpi/4.0.2 3) sems-cmake/3.17.1 10) sems-hdf5/1.10.6/parallel 4) sems-ninja_fortran/1.8.2 11) sems-netcdf/4.7.3/parallel 5) sems-gcc/7.2.0 12) sems-zlib/1.2.8/base 6) sems-cuda/9.2 13) sems-boost/1.59.0/base 7) atdm-env 14) sems-superlu/4.3/base Configuring Trilinos build directory -- PROJECT_SOURCE_DIR='/ascldap/users/ndellin/trilinos/Trilinos' -- PROJECT_BINARY_DIR='/ascldap/users/ndellin/trilinos/Trilinos/BUILD/Test' -- Trilinos_TRIBITS_DIR='/ascldap/users/ndellin/trilinos/Trilinos/cmake/tribits' -- TriBITS_VERSION_STRING='0.9 (Dev)' -- CMAKE_VERSION='3.17.1' -- CMAKE_GENERATOR='Ninja' -- Reading in configuration options from cmake/std/atdm/ATDMDevEnv.cmake ... -- ATDM_BUILD_NAME_KEYS_STR='CUDA-9.2_RELEASE_CUDA' -- ATDM_TWEAKS_FILES='' -- Setting default TPL_ENABLE_MPI=ON -- Setting default Trilinos_ENABLE_Fortran=OFF -- Setting default Trilinos_ENABLE_SECONDARY_TESTED_CODE=ON -- Setting default TPL_ENABLE_CUDA=ON -- Setting default TPL_ENABLE_CUSPARSE=ON -- Setting default TPL_ENABLE_Pthread=OFF -- Setting default TPL_ENABLE_BinUtils=OFF -- Setting default TPL_ENABLE_BLAS=ON -- Setting default TPL_ENABLE_Boost=ON -- Setting default TPL_ENABLE_BoostLib=ON -- Setting default TPL_ENABLE_METIS=OFF -- Setting default TPL_ENABLE_ParMETIS=OFF -- Setting default TPL_ENABLE_HWLOC=OFF -- Setting default TPL_ENABLE_LAPACK=ON -- Setting default TPL_ENABLE_CGNS=OFF -- Setting default TPL_ENABLE_HDF5=ON -- Setting default TPL_ENABLE_Netcdf=ON -- Setting default TPL_ENABLE_SuperLUDist=OFF -- Setting default TPL_ENABLE_Matio=OFF -- Setting default Trilinos_MAKE_INSTALL_GROUP_WRITABLE=TRUE -- Setting default Teko_ModALPreconditioner_MPI_1_DISABLE=ON -- Setting default Zoltan2_XpetraEpetraMatrix_EXE_DISABLE=ON -- Setting default Zoltan2_XpetraEpetraMatrix_MPI_4_DISABLE=ON -- Setting default Belos_Tpetra_HybridGMRES_hb_test_1_MPI_4_DISABLE=ON -- Setting default Belos_Tpetra_HybridGMRES_hb_test_0_MPI_4_DISABLE=ON -- Setting default Belos_rcg_hb_MPI_4_DISABLE=ON -- Setting default Piro_ThyraSolver_EXE_DISABLE=ON -- Setting default Piro_ThyraSolver_MPI_4_DISABLE=ON -- Setting default Piro_AnalysisDriverTpetra_EXE_DISABLE=ON -- Setting default Piro_AnalysisDriverTpetra_MPI_4_DISABLE=ON -- Setting default ROL_adapters_tpetra_test_vector_SimulatedVectorTpetraBatchManagerInterface_EXE_DISABLE=ON -- Setting default ROL_adapters_tpetra_test_vector_SimulatedVectorTpetraBatchManagerInterface_MPI_4_DISABLE=ON -- Setting default KokkosCore_UnitTest_CudaTimingBased_MPI_1_DISABLE=ON -- Setting default ROL_example_PDE-OPT_0ld_adv-diff-react_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_0ld_adv-diff-react_example_02_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_0ld_poisson_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_0ld_stefan-boltzmann_example_03_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_navier-stokes_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_navier-stokes_example_02_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_nonlinear-elliptic_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_nonlinear-elliptic_example_02_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_obstacle_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_stefan-boltzmann_example_01_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_stefan-boltzmann_example_03_MPI_4_DISABLE=ON -- Setting default ROL_example_PDE-OPT_topo-opt_poisson_example_01_MPI_4_DISABLE=ON -- Setting default ROL_test_elementwise_TpetraMultiVector_MPI_4_DISABLE=ON -- Setting default TrilinosCouplings_Example_Maxwell_MueLu_MPI_1_DISABLE=ON -- Setting default TrilinosCouplings_Example_Maxwell_MueLu_MPI_4_DISABLE=ON -- Setting default Zoltan_ch_ewgt_zoltan_parallel_DISABLE=ON -- Setting default Zoltan_ch_grid20x19_zoltan_parallel_DISABLE=ON -- Setting default Zoltan_ch_nograph_zoltan_parallel_DISABLE=ON -- Setting default Zoltan_ch_simple_zoltan_parallel_DISABLE=ON -- Setting default KokkosContainers_PerformanceTest_Cuda_DISABLE=ON -- Setting default KokkosCore_UnitTest_CudaInterOpStreams_MPI_1_SET_RUN_SERIAL=ON -- Setting default KokkosCore_UnitTest_CudaInterOpInit_MPI_1_SET_RUN_SERIAL=ON -- CMAKE_HOST_SYSTEM_NAME='Linux' -- Trilinos_HOSTNAME='kokkos-dev.sandia.gov' -- Found PythonInterp: /usr/bin/python (found suitable version "2.7.5", minimum required is "2.6") -- PYTHON_EXECUTABLE='/usr/bin/python' -- Found Git: /projects/sems/install/rhel7-x86_64/sems/utility/git/2.10.1/bin/git (found version "2.10.1") Setting up major user options ... Trilinos repos versions: -------------------------------------------------------------------------------- *** Base Git Repo: Trilinos 3f683fc [Mon Jun 7 09:54:07 2021 -0600]