Closed ndellingwood closed 1 year ago
@ndellingwood, why are we not seeing this with the Trilinos PR CUDA build? What is that build doing differently from this build? What happens when you use the versions of Kokkos and KokkosKernels that are in the Trilinos 'develop' branch with this exact same build and test process?
@ndellingwood, can you run make dashboard
in the Trilinos build dir so this posts to CDash?
@bartlettroscoe
why are we not seeing this with the Trilinos PR CUDA build? What is that build doing differently from this build?
I'm not certain
What happens when you use the versions of Kokkos and KokkosKernels that are in the Trilinos 'develop' branch with this exact same build and test process?
I'm in progress with this install now for testing, will post back shortly
can you run
make dashboard
in the Trilinos build dir so this posts to CDash
I don't have the nightly properly configured to post to CDash, I have this on my list of TODO's; I haven't set this up before, hopefully you don't mind if I follow-up by email with you as questions arise?
an you run make dashboard in the Trilinos build dir so this posts to CDash
I don't have the nightly properly configured to post to CDash, I have this on my list of TODO's; I haven't set this up before, hopefully you don't mind if I follow-up by email with you as questions arise?
@ndellingwood, every Trilinos build has the ability to post an experimental submission for the current Trilinos configuration to CDash (as long as it can reach https://testing.sandia.gov/cdash). There is zero you should have to do (except perhaps mess with the proxy). See:
Just try make dashboard
and see what happens.
P.S. This requires you have a working configuration of Trilinos that writes the build files to the build dir. This feature can't post results if there are configure failures.
@bartlettroscoe The issue is also present with Trilinos develop branch using the kokkos and kokkos-kernels packages (no source override)
@ndellingwood, every Trilinos build has the ability to post an experimental submission for the current Trilinos configuration to CDash (as long as it can reach https://testing.sandia.gov/cdash). There is zero you should have to do (except perhaps mess with the proxy). See:
* https://docs.trilinos.org/files/TrilinosBuildReference.html#dashboard-submissions
Oh, that's convenient! I hadn't seen this document before, I had assumed I would need to write a ctest driver script. I'll test this out to get this added to the Jenkins nightly builds
As far as running make dashboard
, I'm assuming I'll need to call this for two separate stages - once when building and testing Trilinos, a second time for the app test building against the Trilinos install, and this will result in two cdash submissions, does that sound correct?
I'm assuming I'll need to call this for two separate stages - once when building and testing Trilinos, a second time for the app test building against the Trilinos install, and this will result in two cdash submissions, does that sound correct?
@ndellingwood, no, you can drive the install as well with posting to CDash. Just run with:
$ env CTEST_DO_INSTALL=TRUE make dashboard
See:
Just make sure that you set -DCMAKE_INSTALL_PREFIX=<trilinosInstallDir>
when you configure Trilinos (which you are already doing).
I tried running make dashboard
in my local build where I reproduced using the sems modules but ran into some issues I need to better understand. I'll come back to this after afternoon meetings, here's a snip of the output
CONFIGURE_OPTIONS = '-DTrilinos_TRIBITS_DIR=/ascldap/users/ndellin/Trilinos/cmake/tribits;-DCTEST_USE_LAUNCHERS:BOOL=1;-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON;-DTrilinos_WARNINGS_AS_ERRORS_FLAGS:STRING='';-DTrilinos_ALLOW_NO_PACKAGES:BOOL=ON;-DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON;-DTrilinos_ENABLE_SECONDARY_TESTED_CODE:BOOL=ON;-DTrilinos_EXTRAREPOS_FILE:STRING=/ascldap/users/ndellin/Trilinos/cmake/ExtraRepositoriesList.cmake;-DTrilinos_IGNORE_MISSING_EXTRA_REPOSITORIES:BOOL=ON;-DTrilinos_ENABLE_KNOWN_EXTERNAL_REPOS_TYPE:STRING=Experimental;-DTrilinos_ENABLE_ALL_PACKAGES=ON;-DTrilinos_ENABLE_TESTS:BOOL=ON'
Configure project
Each . represents 1024 bytes of output
.................................................. Size: 50K
.................................................. Size: 100K
................................................ Size of output: 147K
Error(s) when configuring the project
Generating the file '/home/ndellin/Trilinos/Build/InstallTestTrilinos/CMakeCache.clean.txt' ...
Configure FAILED!
I tried running
make dashboard
in my local build where I reproduced using the sems modules but ran into some issues I need to better understand.
@ndellingwood, can you attach the full STDOUT from above and the generated configure output file Testing/Temporary/LastConfigure_<date-time>.log
?
@bartlettroscoe The issue is also present with Trilinos develop branch using the kokkos and kokkos-kernels packages (no source override)
@ndellingwood, then the difference is likely how the environment is set up and the configuration is being done. Is the right CUDA version being found shown in:
20:17:24 -- Found CUDAToolkit: /home/projects/ppc64le-pwr9-nvidia/spack-installs/cuda/11.2.2/gcc/8.3.1/base/uflzqtv/include (found version "11.2.152")
20:17:24 CMake Warning at /projects/ppc64le/spack-installs/cmake/3.23.1/gcc/7.2.0/base/knlpf2j/share/cmake-3.23/Modules/FindCUDAToolkit.cmake:919 (message):
20:17:24 Could not find librt library, needed by CUDA::cudart_static
That looks to be a env problem. (This is an error coming from inside of the standard CMake module cmake-3.23/Modules/FindCUDAToolkit.cmake:919
.)
@bartlettroscoe the configure output I posted was from a nightly build using cuda/11.2.2 on Weaver (sems modules not available); the reproducer notes I posted were from a different machine using sems modules (using cuda/11.4.2), that is the source of the discrepancy. I posted the sems-based reproducer because I thought that would be more portable for testing on various machines
Here's the stdout output in collapsible block (lots of stuff)
My <build-dir>/Testing/Temporary
directory is empty, no LastConfigure_*.log file
@bartlettroscoe I saw this line in the stdout output:
-- CTEST_NOTES_FILES='/home/ndellin/Trilinos/Build/InstallTestTrilinos/do-configure'
The name of my configuration script is configure.sh, should I rename to do-configure?
@bartlettroscoe here is my LastConfigure_*.log file (I was distracted and looked in the wrong build dir before)
I experimented by copying my configure.sh to do-configure, blew away the CMake* directories, manually reconfigured then tried running make dashboard
but same issue. I'm reading through the docs and may answer my own question, but how do I get the command to use my configuration script? The configure output (when run through make dashboard) does not match when I run my configure.sh script, in particular lots of stuff shows up under Final set of enabled top-level packages
in the make dashboard
output
make dashboard snip:
Final set of enabled top-level packages: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest Kokkos Teuchos KokkosKernels RTOp Sacado MiniTensor Epetra Zoltan Shards Triutils EpetraExt Tpetra TrilinosSS Domi Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_Node Amesos2 SEACAS Komplex Anasazi Ifpack2 Stratimikos FEI Teko TriKota Intrepid Intrepid2 Compadre STK Percept Krino Phalanx NOX Moertel MueLu Zoltan2Sphynx Zoltan2 ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro Panzer PyTrilinos Adelus TrilinosCouplings Pike TrilinosBuildStats TrilinosInstallTests 65
Final set of enabled packages: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest Kokkos TeuchosCore TeuchosParser TeuchosParameterList TeuchosComm TeuchosNumerics TeuchosRemainder TeuchosKokkosCompat TeuchosKokkosComm Teuchos KokkosKernels RTOp Sacado MiniTensor Epetra Zoltan Shards Triutils EpetraExt TpetraTSQR TpetraCore Tpetra TrilinosSS Domi ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_NodeHTS ShyLU_NodeTacho ShyLU_Node Amesos2 SEACASExodus SEACASNemesis SEACASIoss SEACASChaco SEACASAprepro_lib SEACASSuplibC SEACASSuplibCpp SEACASAprepro SEACASConjoin SEACASEjoin SEACASEpu SEACASCpup SEACASExo2mat SEACASExodiff SEACASExomatlab SEACASExo_format SEACASMat2exo SEACASNas2exo SEACASZellij SEACASNemslice SEACASNemspread SEACASSlice SEACAS Komplex Anasazi Ifpack2 Stratimikos FEI Teko TriKota Intrepid Intrepid2 Compadre STKUtil STKCoupling STKMath STKSimd STKNGP_TEST STKTopology STKMesh STKIO STKSearch STKTransfer STKTools STKBalance STKUnit_test_utils STKUnit_tests STKDoc_tests STKExprEval STKEmend STK Percept Krino Phalanx NOX Moertel MueLu Zoltan2Sphynx Zoltan2 ShyLU_DDFROSch ShyLU_DDCommon ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro PanzerCore PanzerDofMgr PanzerDiscFE PanzerAdaptersSTK PanzerMiniEM Panzer PyTrilinos Adelus TrilinosCouplings PikeBlackBox PikeImplicit Pike TrilinosBuildStats TrilinosInstallTests 129
Final set of non-enabled top-level packages: NewPackage 1
Final set of non-enabled packages: ShyLU_NodeBasker ShyLU_NodeFastILU SEACASExodus_for SEACASExoIIv2for32 SEACASSupes SEACASSuplib SEACASSVDI SEACASPLT SEACASAlgebra SEACASBlot SEACASExotxt SEACASEx1ex2v2 SEACASExotec2 SEACASFastq SEACASGjoin SEACASGen3D SEACASGenshell SEACASGrepos SEACASExplore SEACASMapvarlib SEACASMapvar SEACASMapvar-kd SEACASNumbers SEACASTxtexo SEACASEx2ex1v2 STKMiddle_mesh STKSearchUtil STKTransferUtil ShyLU_DDCore PanzerExprEval NewPackage 31
Final set of enabled top-level external packages/TPLs: MPI BLAS LAPACK Boost Netcdf Matio DLlib 7
Final set of enabled external packages/TPLs: MPI BLAS LAPACK Boost Netcdf Matio DLlib 7
Final set of non-enabled top-level external packages/TPLs: MKL yaml-cpp Peano CUDA CUBLAS CUSOLVER CUSPARSE Thrust Cusp ROCBLAS ROCSPARSE TBB Pthread HWLOC QTHREAD BinUtils ARPREC QD Scotch OVIS gpcd DataWarp METIS MTMETIS ParMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF Zlib HDF5 CGNS Pnetcdf ADIOS2 Faodel Cereal Catalyst2 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS STRUMPACK PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostLib BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Clp GLPK qpOASES PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField Valgrind QUO ViennaCL Avatar mlpack pebbl MAGMASparse Check SARMA CDT mpi_advance 115
Final set of non-enabled external packages/TPLs: MKL yaml-cpp Peano CUDA CUBLAS CUSOLVER CUSPARSE Thrust Cusp ROCBLAS ROCSPARSE TBB Pthread HWLOC QTHREAD BinUtils ARPREC QD Scotch OVIS gpcd DataWarp METIS MTMETIS ParMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF Zlib HDF5 CGNS Pnetcdf ADIOS2 Faodel Cereal Catalyst2 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS STRUMPACK PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostLib BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Clp GLPK qpOASES PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField Valgrind QUO ViennaCL Avatar mlpack pebbl MAGMASparse Check SARMA CDT mpi_advance 115
Final set of packages to be explicitly processed by CTest/CDash: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest Kokkos Teuchos KokkosKernels RTOp Sacado MiniTensor Epetra Zoltan Shards Triutils EpetraExt Tpetra TrilinosSS Domi Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_Node Amesos2 SEACAS Komplex Anasazi Ifpack2 Stratimikos FEI Teko TriKota Intrepid Intrepid2 Compadre STK Percept Krino Phalanx NOX Moertel MueLu Zoltan2Sphynx Zoltan2 ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro Panzer PyTrilinos Adelus TrilinosCouplings Pike TrilinosBuildStats TrilinosInstallTests 65
(Lots of stuff enabled that I don't enable in my script)L
manually running configure.sh snip:
Final set of enabled top-level packages: Kokkos Teuchos KokkosKernels Epetra Tpetra 5
Final set of enabled packages: Kokkos TeuchosCore TeuchosParser TeuchosParameterList TeuchosComm TeuchosNumerics TeuchosRemainder TeuchosKokkosCompat TeuchosKokkosComm Teuchos KokkosKernels Epetra TpetraTSQR TpetraCore Tpetra 15
Final set of non-enabled top-level packages: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest RTOp Sacado MiniTensor Zoltan Shards Triutils EpetraExt TrilinosSS Domi Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_Node Amesos2 SEACAS Komplex Anasazi Ifpack2 Stratimikos FEI Teko TriKota Intrepid Intrepid2 Compadre STK Percept Krino Phalanx NOX Moertel MueLu Zoltan2Sphynx Zoltan2 ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro Panzer PyTrilinos NewPackage Adelus TrilinosCouplings Pike TrilinosBuildStats TrilinosInstallTests 61
Final set of non-enabled packages: TrilinosFrameworkTests TrilinosATDMConfigTests Gtest RTOp Sacado MiniTensor Zoltan Shards Triutils EpetraExt TrilinosSS Domi ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Thyra Xpetra Isorropia Pliris AztecOO Galeri Amesos Pamgen Zoltan2Core Ifpack ML Belos ShyLU_NodeHTS ShyLU_NodeTacho ShyLU_NodeBasker ShyLU_NodeFastILU ShyLU_Node Amesos2 SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASNemesis SEACASIoss SEACASChaco SEACASAprepro_lib SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASPLT SEACASAlgebra SEACASAprepro SEACASBlot SEACASConjoin SEACASEjoin SEACASEpu SEACASCpup SEACASExo2mat SEACASExodiff SEACASExomatlab SEACASExotxt SEACASExo_format SEACASEx1ex2v2 SEACASExotec2 SEACASFastq SEACASGjoin SEACASGen3D SEACASGenshell SEACASGrepos SEACASExplore SEACASMapvarlib SEACASMapvar SEACASMapvar-kd SEACASMat2exo SEACASNas2exo SEACASZellij SEACASNemslice SEACASNemspread SEACASNumbers SEACASSlice SEACASTxtexo SEACASEx2ex1v2 SEACAS Komplex Anasazi Ifpack2 Stratimikos FEI Teko TriKota Intrepid Intrepid2 Compadre STKUtil STKCoupling STKMath STKSimd STKNGP_TEST STKTopology STKMesh STKIO STKSearch STKMiddle_mesh STKTransfer STKTools STKBalance STKUnit_test_utils STKSearchUtil STKTransferUtil STKUnit_tests STKDoc_tests STKExprEval STKEmend STK Percept Krino Phalanx NOX Moertel MueLu Zoltan2Sphynx Zoltan2 ShyLU_DDFROSch ShyLU_DDCore ShyLU_DDCommon ShyLU_DD ShyLU Rythmos Tempus Stokhos ROL Piro PanzerCore PanzerDofMgr PanzerDiscFE PanzerAdaptersSTK PanzerMiniEM PanzerExprEval Panzer PyTrilinos NewPackage Adelus TrilinosCouplings PikeBlackBox PikeImplicit Pike TrilinosBuildStats TrilinosInstallTests 145
Final set of enabled top-level external packages/TPLs: CUDA MPI BLAS LAPACK DLlib 5
Final set of enabled external packages/TPLs: CUDA MPI BLAS LAPACK DLlib 5
Final set of non-enabled top-level external packages/TPLs: MKL yaml-cpp Peano CUBLAS CUSOLVER CUSPARSE Thrust Cusp ROCBLAS ROCSPARSE TBB Pthread HWLOC QTHREAD BinUtils ARPREC QD Boost Scotch OVIS gpcd DataWarp METIS MTMETIS ParMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF Zlib HDF5 CGNS Pnetcdf Netcdf ADIOS2 Faodel Cereal Catalyst2 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS STRUMPACK PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostLib BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Clp GLPK qpOASES Matio PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField Valgrind QUO ViennaCL Avatar mlpack pebbl MAGMASparse Check SARMA CDT mpi_advance 117
Final set of non-enabled external packages/TPLs: MKL yaml-cpp Peano CUBLAS CUSOLVER CUSPARSE Thrust Cusp ROCBLAS ROCSPARSE TBB Pthread HWLOC QTHREAD BinUtils ARPREC QD Boost Scotch OVIS gpcd DataWarp METIS MTMETIS ParMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF Zlib HDF5 CGNS Pnetcdf Netcdf ADIOS2 Faodel Cereal Catalyst2 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS STRUMPACK PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostLib BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Clp GLPK qpOASES Matio PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField Valgrind QUO ViennaCL Avatar mlpack pebbl MAGMASparse Check SARMA CDT mpi_advance 117
@bartlettroscoe here is my LastConfigure_*.log file (I was distracted and looked in the wrong build dir before)
@ndellingwood, if you look at that file, you will see:
Processing enabled external package/TPL: Netcdf (enabled by SEACASNemesis, disable with -DTPL_ENABLE_Netcdf=OFF)
-- Using find_package(Netcdf ...) ...
CMake Error at cmake/tribits/common_tpls/find_modules/FindNetCDF.cmake:163 (message):
Can not locate NetCDF include directory
Call Stack (most recent call first):
cmake/tribits/common_tpls/FindTPLNetcdf.cmake:92 (find_package)
cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:219 (include)
cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:169 (tribits_process_enabled_tribits_find_tpl_mod_file)
cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:104 (tribits_process_enabled_standard_tpl)
cmake/tribits/core/package_arch/TribitsProjectImpl.cmake:193 (tribits_process_enabled_tpls)
cmake/tribits/core/package_arch/TribitsProject.cmake:92 (tribits_project_impl)
CMakeLists.txt:126 (TRIBITS_PROJECT)
CMake Error at cmake/tribits/common_tpls/find_modules/FindNetCDF.cmake:276 (message):
This looks to be an env problem. How is it that find_package(Netcdf ...)
does not find Netcdf
when the initial configure did?
Can you please provide the STDOUT from the from-scratch configure?
$ rm -r CMake*
$ ./do-configure &> configure.out
Then, what happens when you just run:
$ cmake . &> reconfigure.out
instead of make dashboard
? What is inside of that STDOUT file reconfigure.out
?
@bartlettroscoe in my configuration I'm not enabling SEACAS or any SEACAS sub-packages, I don't understand why that is being triggered when I call make dashboard
, which is resulting in the NETCDF issues (I'm not loading NETCDF nor enabling it). I'm only enabling Kokkos, KokkosKernels and Tpetra explicitly.
cat do-configure
:
export TRILINOS_DIR=$PWD/../..
export KOKKOS_PATH=$TRILINOS_DIR/packages/kokkos
module purge
module load sems-cmake/3.24.3 sems-gcc/8.3.0 sems-cuda/11.4.2 sems-openmpi/4.0.5 sems-python/3.7.9 sems-ninja/1.10.1
export OMPI_CXX=$KOKKOS_PATH/bin/nvcc_wrapper
# configure
cmake \
-GNinja \
-DCMAKE_CXX_STANDARD=17 \
-DCMAKE_INSTALL_PREFIX="${PWD}/install" \
-DTPL_ENABLE_MPI=ON \
-DTrilinos_ENABLE_TESTS=OFF \
-DTrilinos_ENABLE_Kokkos=ON \
-DKokkos_ENABLE_CUDA=ON \
-DKokkos_ENABLE_CUDA_UVM=ON \
-DKokkos_ENABLE_CUDA_LAMBDA=ON \
-DTrilinos_ENABLE_KokkosKernels=ON \
-DKokkosKernels_INST_MEMSPACE_CUDAUVMSPACE=ON \
-DTrilinos_ENABLE_Tpetra=ON \
$TRILINOS_DIR
@bartlettroscoe in my configuration I'm not enabling SEACAS or any SEACAS sub-packages, I don't understand why that is being triggered when I call
make dashboard
, which is resulting in the NETCDF issues (I'm not loading NETCDF nor enabling it). I'm only enabling Kokkos, KokkosKernels and Tpetra explicitly.
@ndellingwood, well, that is the issue then. Let me look into what is happening and will let you know if I need any more info. (I need to see if there is a missing use case in TriBITS testing of this feature.)
@ndellingwood, looks like there is a defect in tribits_ctest_driver()
for the case where no packages have tests enabled shown above which shows:
...
**************************************************
*** Running all-at-once experimental dashboard ***
**************************************************
Trilinos_ENABLED_PACKAGES_LIST=
***
*** Run the dashboard command setting the list of packages
***
Running: env ... Trilinos_PACKAGES= ... <base-cmake-dir>/ctest -V -S /ascldap/users/ndellin/Trilinos/cmake/tribits/ctest_driver/experimental_build_test.cmake
...
***
*** Determining what packages to enable based what was set in Trilinos_PACKAGES by the user ...
***
Setting Trilinos_ENABLE_ALL_PACKAGES=ON since Trilinos_PACKAGES_USER_SELECTED=''
...
Above, note the empty lists Trilinos_ENABLED_PACKAGES_LIST=
and Trilinos_ENABLED_PACKAGES_LIST=
which happens to trigger setting Trilinos_ENABLE_ALL_PACKAGES=ON
inside of tribits_ctest_driver()
.
The dashboard
target assumes that you are testing something and not just building. I have never tried to run the dashboard
target when there are no tests enabled (and there are not TriBITS tests for that use case). It determines what packages to test based on which packages have tests enabled (which is why Trilinos_ENABLED_PACKAGES_LIST
is empty).
This use case can be supported for the all-at-once mode for when there are no tests enabled and the dashboard
target should be able to do the correct thing. (I think I would just error out for the package-by-package mode.)
For now, could you please add -DTpetra_ENABLE_TESTS=ON
to your configure script for Trilinos and try again? (The Kokkos develop + KokkosKerenels develop + Trilinos develop
testing should be running some tests or this is not much of a "test".)
I can add a test case to TriBITS where no tests are enabled and the make dashboard
target will only do a build and not try to run any tests (which I have never done before.)
Above, note the empty lists
Trilinos_ENABLED_PACKAGES_LIST=
andTrilinos_ENABLED_PACKAGES_LIST=
which happens to trigger settingTrilinos_ENABLE_ALL_PACKAGES=ON
inside oftribits_ctest_driver()
.
Thanks, that explains it!
For now, could you please add
-DTpetra_ENABLE_TESTS=ON
to your configure script for Trilinos and try again?
Will do this now
Nice, enabling the Tpetra tests worked successfully to post the configure, build, and unit test results to the cdash experimental track (this is from my local build on kokkos-dev-2):
I forgot to set CTEST_DO_INSTALL=TRUE
@bartlettroscoe it looks like setting CTEST_DO_INSTALL=TRUE
will run make install
, but is there an option that will trigger testing of e.g. simpleBuildAgainstTrilinos
against the install of Trilinos and post to cdash?
@bartlettroscoe it looks like setting
CTEST_DO_INSTALL=TRUE
will runmake install
, but is there an option that will trigger testing of e.g.simpleBuildAgainstTrilinos
against the install of Trilinos and post to cdash?
CCing @sebrowne to inform him about this as well ...
@ndellingwood, please set the following configure options:
-DTrilinos_ENABLE_TrilinosInstallTests=ON \
-TrilinosInstallTests_ENABLE_TESTS=ON \
and configure and run make dashboard
again and that will build and test simpleBuildAgainstTrilinos
against a local install of Trilinos under the build dir and also against Trilinos packages in the build dir. (That is a separate install from the one pointed to by CMAKE_PREFIX_PATH
.) See packages/TrilinosInstallTests/CMakeLists.txt
.
@bartlettroscoe thanks, I wasn't aware of the TrilinosInstallTests
package and tests. I added the options and reran make dashboard
, results are posted:
The install tests passed when run this way.
I reran them locally following the process in my OP:
mkdir -p app-test
cd app-test
cmake -DCMAKE_PREFIX_PATH=${TRILINOS_INSTALL} ${TRILINOS_DIR}/demos/simpleBuildAgainstTrilinos
make
and I am still encountering the same problems. Is there additional cmake options I should be passing on the cmake line to get this to work? What I have posted worked prior to #11863
To clarify, one of the problems is that the correct compiler is not being selected during configuration - g++ is being selected, rather than nvcc_wrapper (which was pointed at by OMPI_CXX
in my configuration and build of Trilinos)
Edit: Adding the configuration output and compilation error:
[ndellin@kokkos-dev-2 app-test]$ cmake -DCMAKE_PREFIX_PATH=${TRILINOS_INSTALL} ${TRILINOS_DIR}/demos/simpleBuildAgainstTrilinos
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /projects/sems/install/rhel7-x86_64/sems-compilers/tpl/gcc/8.3.0/gcc/4.8.5/base/odjqidm/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /projects/sems/install/rhel7-x86_64/sems-compilers/tpl/gcc/8.3.0/gcc/4.8.5/base/odjqidm/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDAToolkit: /projects/sems/install/rhel7-x86_64/sems/v2/tpl/cuda/11.4.2/gcc/8.3.0/base/7lflstk/include (found version "11.4.120")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Enabled Kokkos devices: CUDA;SERIAL
Found Trilinos! Here are the details:
Trilinos_DIR = /ascldap/users/ndellin/Trilinos/Build/InstallTestTrilinos/install/lib/cmake/Trilinos
Trilinos_VERSION = 14.1
Trilinos_PACKAGE_LIST = TrilinosInstallTests;Tpetra;TpetraCore;TpetraTSQR;Epetra;KokkosKernels;Teuchos;TeuchosKokkosComm;TeuchosKokkosCompat;TeuchosRemainder;TeuchosNumerics;TeuchosComm;TeuchosParameterList;TeuchosParser;TeuchosCore;Kokkos
Trilinos_LIBRARIES = Teuchos::all_libs;Tpetra::all_libs
Trilinos_INCLUDE_DIRS = /ascldap/users/ndellin/Trilinos/Build/InstallTestTrilinos/install/include
Trilinos_TPL_LIST =
Trilinos_TPL_LIBRARIES =
Trilinos_BUILD_SHARED_LIBS = FALSE
End of Trilinos details
-- Configuring done
-- Generating done
-- Build files have been written to: /ascldap/users/ndellin/Trilinos/Build/InstallTestTrilinos/app-test
[ndellin@kokkos-dev-2 app-test]$ make
[ 50%] Building CXX object CMakeFiles/MyApp.dir/app.cpp.o
g++: error: unrecognized command line option '-Wext-lambda-captures-this'
g++: error: unrecognized command line option '-arch=sm_70'
make[2]: *** [CMakeFiles/MyApp.dir/app.cpp.o] Error 1
make[1]: *** [CMakeFiles/MyApp.dir/all] Error 2
make: *** [all] Error 2
To clarify, one of the problems is that the correct compiler is not being selected during configuration - g++ is being selected, rather than nvcc_wrapper (which was pointed at by
OMPI_CXX
in my configuration and build of Trilinos)
@ndellingwood, you now have to pass the compilers to use explicitly as shown in the test TrilinosInstallTests_simpleBuildAgainstTrilinos showing:
Running: "/net/watson.sandia.gov/storage/fast/projects/sems/install/rhel7-x86_64/sems/utility/cmake/3.24.3/gcc/8.3.0/base/pwyrvxm/bin/cmake" "-DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/v2/tpl/openmpi/4.0.5/gcc/8.3.0/base/wtxd46r/bin/mpicc" "-DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/v2/tpl/openmpi/4.0.5/gcc/8.3.0/base/wtxd46r/bin/mpicxx" "-DCMAKE_Fortan_COMPILER=/projects/sems/install/rhel7-x86_64/sems/v2/tpl/openmpi/4.0.5/gcc/8.3.0/base/wtxd46r/bin/mpif90" "-DCMAKE_PREFIX_PATH=/ascldap/users/ndellin/Trilinos/Build/InstallTestTrilinos/install" "-DCMAKE_BUILD_TYPE=DEBUG" "/ascldap/users/ndellin/Trilinos/demos/simpleBuildAgainstTrilinos"
You can't get the compilers automatically from find_package(Trilinos)
anymore; you have to explicitly pass them in (as well as have your env vars set correctly like OMP_CXX
pointing to nvcc_wrapper
obviously). See the discussion about this in https://github.com/trilinos/Trilinos/pull/11863#issuecomment-1540403871. We may need to update the file demos/simpleBuildAgainstTrilinos/README.md to make this more explicit?
Thanks @bartlettroscoe , I missed that comment but confirmed including the compilers on the cmake line resolved the issue I was seeing (compilers are not properly set, code compiles, test passes)
cmake -D CMAKE_CXX_COMPILER=`which mpicxx` -D CMAKE_C_COMPILER=`which mpicc` -DCMAKE_PREFIX_PATH=${TRILINOS_INSTALL} ${TRILINOS_DIR}/demos/simpleBuildAgainstTrilinos
Re: README.md updates, I think it would help to have step 3 updated https://github.com/trilinos/Trilinos/blob/develop/demos/simpleBuildAgainstTrilinos/README.md#3-configure-the-example-project
I changed the labeling from bug to question, thanks @bartlettroscoe for your help!
Thanks @bartlettroscoe , I missed that comment but confirmed including the compilers on the cmake line resolved the issue I was seeing (compilers are not properly set, code compiles, test passes)
cmake -D CMAKE_CXX_COMPILER=`which mpicxx` -D CMAKE_C_COMPILER=`which mpicc` -DCMAKE_PREFIX_PATH=${TRILINOS_INSTALL} ${TRILINOS_DIR}/demos/simpleBuildAgainstTrilinos
Re: README.md updates, I think it would help to have step 3 updated https://github.com/trilinos/Trilinos/blob/develop/demos/simpleBuildAgainstTrilinos/README.md#3-configure-the-example-project
@ndellingwood,
See the new PR:
Can you please review?
But I think some positive things came out of this issue anyway:
dashbaord
target when no tests are enabled (see https://github.com/TriBITSPub/TriBITS/issues/584).TrilinosInstallTests
. (Perhaps we should advertise these somehow?)
Bug Report
After merge of #11863 I am encountering problems when testing a Trilinos install with the
simpleBuildAgainstTrilinos
demo with Cuda builds.I'm adding the @trilinos/tribits label for now, can re-label as needed, @bartlettroscoe
Here are some notes, reproducer details will follow below
simpleBuildAgainstTrilinos
configuration output: (leading to failed compilation)The configuration above leads to compilation errors, as g++ is being picked up as the compiler rather than nvcc_wrapper through the OMPI_CXX environment variable
Compilation error:
My cmake line for configuring
simpleBuildAgainstTrilinos
is as follows, which worked prior to the #11863 updates:From comparison, this is the configuration output from the previous round of passing tests:
In case this is relevant, these failures began showing in nightly tests against Kokkos Core+Kernels develop branches where I use source override to build using the source code from their respective repos (rather than the Trilinos packages). I still need to reproduce with just Trilinos' develop branch
Steps to Reproduce
Symbolic link to kokkos and kokkos-kernels repos in Trilinos source directory for source override
cd Trilinos ln -s/kokkos kokkos
ln -s /kokkos-kernels kokkos-kernels
cd $HOME mkdir -p build cd build
modules and env
sh /projects/sems/modulefiles/utils/sems-v2-modules-init.sh module load sems-cmake/3.24.3 sems-gcc/8.3.0 sems-cuda/11.4.2 sems-openmpi/4.0.5 sems-python/3.7.9 sems-ninja/1.10.1 export OMPI_CXX=$KOKKOS_PATH/bin/nvcc_wrapper
configure
cmake \ -GNinja \ -DCMAKE_CXX_STANDARD=17 \ -DCMAKE_INSTALL_PREFIX="${TRILINOS_INSTALL}" \ -DTPL_ENABLE_MPI=ON \ -DTrilinos_ENABLE_TESTS=OFF \ -DTrilinos_ENABLE_Kokkos=ON \ -DKokkos_ENABLE_CUDA=ON \ -DKokkos_ENABLE_CUDA_UVM=ON \ -DKokkos_ENABLE_CUDA_LAMBDA=ON \ -DTrilinos_ENABLE_KokkosKernels=ON \ -DKokkosKernels_INST_MEMSPACE_CUDAUVMSPACE=ON \ -DTrilinos_ENABLE_Tpetra=ON \ -DKokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \ -DKokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels \ $TRILINOS_DIR
install Trilinos
make install -j8
test app build against trilinos install
mkdir -p app-test cd app-test cmake -DCMAKE_PREFIX_PATH=${TRILINOS_INSTALL} ${TRILINOS_DIR}/demos/simpleBuildAgainstTrilinos make ctest