Closed bartlettroscoe closed 5 years ago
On this last Thursday, I had a long conversation with @jwillenbring about this CUDA PR build on 'ride' and 'white'. We decided that it was time to turn this over to the @trilinos/framework team. The candidate CUDA build Trilinos-atdm-white-ride-cuda-9.2-release-debug-pt
on 'ride' shown running for the last 32 days is shown at:
As you can see, from that query, the build and the tests all completed and submitted results every day. And recently, the build have only been taking about 2h 30m and running the tests are taking 1h 11m. That is well within a reasonable runtime for a Trilinos PR build. And the robustness has been excellent. In fact, if you look at all of the 186 ATDM Trilinos builds run on 'ride' since 9/26/2018 here, there was not a single case where build and test results did not get submitted.
The configuration for this build is simply:
$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-9.2-debug
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnvAllPtPackages.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_ALL_PACKAGES=ON \
$TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16
NOTE: You can load the env, configure, and build completely on a compute node. To do this, just put your CTest -S driver in a bash script and run it with:
$ bsub -x -Is -q rhel7F -n 16 <ctest_s_driver_script>
You can see exactly how the ATDM Trilinos CTest -S driver system (that uses the TriBITS CTest -S driver) drives this build on 'ride' by looking at this file:
The jenkins-srn.sandia.gov
project driver can be seen at:
(NOTE: We use the Jenkins job-inheritance plugin so you will have to trace subclass jobs.)
Hopefully that should be all of the information that the Framework team needs to be able to copy the settings for this build out into their own files and then stand this build up. If not, please ask me any question you might have about this.
@jwillenbring and I talked about the Framework team doing the following:
Copy the settings out of the above-referenced files under Trilinos/cmake/std/atdm/
into new files under Trilinos/cmake/std/
that the Framework team will own. (This will avoid depending on files under Trilinos/cmake/std/atdm/
that don't get tested in PR builds.)
Disable the current set of tests that don't build and the tests that don't run in the *.cmake
file for this PR CUDA build (see these Issues).
Set up a new CTest -S driver script for this CUDA build on 'ride' and set up a Jenkins job on 'ride' on jenkins-srn.sandia.gov to driver this build.
Start running this CUDA build as part of the develop to master promotion PR builds and see how it goes for a while.
Once the new CUDA build is shown to be robust running in develop to master promotion builds for a while, then add this CUDA build to the set of Trilinos PR builds.
NOTE: We have been seeing problems getting test results for this build on 'white' recently for the last three days since 10/23/2018 as shown here. The Jenkins jobs for these builds can be seen at:
(see build numbers #79
, #80
, and #81
). From looking at the Jenkins output, it seems that a bunch of tests are timing out at 10 minutes and for some reason and the jobs were killed at the SLUM job time limit of 12 hours. But since this PR build would run on 'ride', not 'white', we can track down those problems with the Test Bed team. I will bring this up at the ATDM PI meeting on Monday.
FYI: I removed the ATDM DevOps
label and added the Framework
label in order to officially hand this off to the @trilinos/framework team.
@trilinos/framework Please include the following option in the CUDA configure for this PR:
-D MueLu_ENABLE_Kokkos_Refactor_Use_By_Default:BOOL=YES
Thank you.
@jwillenbring : @bartlettroscoe brought this up in ATDM meeting. Can someone describe a plan for ths ? Also it would be nice to assign it to a person, so there is a responsible person we can talk to.
@jwillenbring @william76
@trilinos/framework,
For the reasons described in #3998, this needs to be a release-debug-pt
build, not just a debug
build. The cuda-9.2-release-debug
build runs more tests and catches more defects than either a cuda-9.2-opt
or cuda-9.2-debug
build (and should run faster than a cuda-9.2-debug
build).
I will update the instructions above accordingly.
@jwillenbring : @bartlettroscoe brought this up in ATDM meeting. Can someone describe a plan for ths ? Also it would be nice to assign it to a person, so there is a responsible person we can talk to.
@srajama1
I just assigned this to @ZUUL42 who was going to meet with @prwolfe yesterday to start work on this. We spoke a little about the general process and he recently set up the new 7.3 dev->master build.
@jwillenbring Thank you !
FYI: A CUDA PR build would have avoided #4050 that took out almost all of the Panzer tests (i.e. we lost a day of CUDA tests for Panzer we were hopping to get to help clean up our CUDA builds).
@trilinos/framework
FYI: The ascic-jenkins.sandia.gov slave for 'ride' looks like it has 16 executors from looking at:
?
That has a very strange curve for "available executors". It seems unlikely that your ascic-jenkins.sandia.gov jobs will experience the crashes that we have been experiencing on jenkins-srn.sandia.gov documented here:
Let me know if you have any questions about creating this CUDA PR build based on our cuda-9.2-release-debug
build that we have been running for a long time.
FYI: #4123 would have been avoided if this CUDA PR build would have been running. I lost at least 2+ hours last night and today on that one. Also, it looks like EMPIRE pulled that version of Trilinos (because I don't think they yet have a CUDA build).
CC: @rppawlo, @fryeguy52
@trilinos/framework,
As per https://github.com/trilinos/Trilinos/pull/4146#issuecomment-452097215 and can we make sure that this CUDA PR build getting set up on 'ride' sets:
set(Kokkos_ENABLE_Profiling OFF CACHE BOOL)
?
That will catch issues like #4145 before they they can break ATDM Trilinos builds.
@bartlettroscoe Did you mean "ON CACHE BOOL"?
Wait, why is ATDM turning off Kokkos profiling?!?
@mhoemmen said:
Wait, why is ATDM turning off Kokkos profiling?!?
Don't know. Need to find out from the EMPIRE developers. Note that the current native SPARC configuration for Trilinos does NOT set Kokkos_ENABLE_Profiling=OFF
.
I will create a new ATDM Trilinos GitHub issue to see about removing Kokkos_ENABLE_Profiling=OFF
from the ATDM Trilinos configuration.
CC: @srajama1
@trilinos/framework, please make sure the new CUDA PR build on 'ride' enables the ShyLU_DD package. As of today, it should be 100% clean in that CUDA 9.2 build on 'ride' (see #3541). We need the CUDA PR build to protect the ShyLU_DD packages's CUDA build.
FYI: While reviewing PR #4332, I just noticed that there is now a Trilinos_pullrequest_cuda_9.2-83
build as shown in https://github.com/trilinos/Trilinos/pull/4332#issuecomment-460972709. This is a PR that changes Teuchos so it should test everything downstsream in Trilinos. Looking at the configure output on CDash here, we can see the following explicit disables:
Explicitly disabled packages on input (by user or by default): Claps SEACAS Trios Komplex TriKota Moertel PyTrilinos NewPackage 8
This shows that SEACAS is being disabled. Since SEACAS plays a critical role in ATDM, we need to get SEACAS enables ASAP to protect ATDM and other important customers. This build should 100% match the build Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt
which builds and tests SEACAS just fine as shown, for example, today here.
As I told @jwillenbring, I will look into what the problem with this CUDA PR build is and get it to match the build Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt
and then create a PR to enable SEACAS again.
FYI: I am working on fixing the problems in the Trilinos CUDA PR build. A few things I am seeing right away:
The KOKKOS_ARCH was wrong. It was setting 'Power9' when the correct arch is 'Power8,Kepler37'. (The KOKKOS_ARCH can be seen on CDash in configure output.) This likely explained the set of the Tpetra tests that get disabled in the PR build and perhaps why STK tests got disabled (also since BoostLib was disabled). (There are no failing Tpetra tests in the matching ATDM Trilinos PR build).
The wrong set of TPLs were enabled. (The correct set can seen configure output on CDash)
The libraries for Netcdf where using "${
I will run the configures and builds and get these to match up. Once I have everything matched up and the correct set of test disables added, I will post a PR to merge in this configuration. I will also provide detailed instructions on how I did this so others can copy this process in the future for future PR builds.
I just posed PR #4592 which fixes the CUDA PR build. It enables all of SEACAS and STK (and all their tests) and it enables all 160 Panzer BASIC tests. And they all pass (except for one recent known STK test failure with CUDA described in #4551). The process I used to fix the build was pretty simple and is described in detail below (so that others can follow a similar process in the future). With all the configure iterations, it took about 4 hours to complete this matching (mostly because the configure is slow in the NFS mounted drive on 'ride'). Once I got the configure diffs to match up, the build and tests ran right out of the box (with the one expected failing STK test).
First, I set up the build dir:
/home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/cuda-9.2-gnu-7.2.0-release-debug-pt/
with the files load-env.sh
:
source /home/rabartl/Trilinos.base/Trilinos/cmake/std/atdm/load-env.sh \
cuda-9.2-gnu-7.2.0-release-debug-pt
and do-configure
:
cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnvAllPtPackages.cmake \
-DKokkos_ENABLE_Profiling=ON \
-DTrilinos_ENABLE_TESTS=ON \
-DTrilinos_TEST_CATEGORIES=BASIC \
-DTrilinos_TRACE_ADD_TEST=ON \
-DDART_TESTING_TIMEOUT:STRING=600.0 \
-DTrilinos_ENABLE_CONFIGURE_TIMING=ON \
"$@" \
/home/rabartl/Trilinos.base/Trilinos
(NOTE: I set some of these options to better match the Trilinos CUDA PR build settings.)
I ran the base configure with:
$ cd /home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/cuda-9.2-gnu-7.2.0-release-debug-pt/
$ . load-env.sh
Hostname 'ride6' matches known ATDM host 'ride' and system 'ride'
Setting compiler and build options for buld name 'cuda-9.2-gnu-7.2.0-release-debug-pt'
Using white/ride compiler stack CUDA-9.2_GNU-7.2.0 to build RELEASE-DEBUG code with Kokkos node type CUDA and KOKKOS_ARCH=Power8,Kepler37
$ time ./do-configure -DTrilinos_ENABLE_ALL_PACKAGES=ON \
-DTrilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES=ON \
&> configure.out
real 6m34.854s
user 2m56.180s
sys 1m19.912s
(NOTE: The Trilinos CUDA PR build always set Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES=ON
which is actually not desirable if you just want to reproduce you one package build.)
I then set up a configure and build directory for the Trilinos CUDA PR configuration:
/home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/pull-request-cuda-9.2/
with files load-env.sh
:
module purge
export WORKSPACE=/home/rabartl/Trilinos.base
source /home/rabartl/Trilinos.base/Trilinos/cmake/std/sems/PullRequestCuda9.2TestingEnv.sh
and do-configure
:
cmake \
-GNinja \
-C /home/rabartl/Trilinos.base/Trilinos/cmake/std/PullRequestLinuxCuda9.2TestingSettings.cmake \
-DTrilinos_ENABLE_TESTS:BOOL=ON \
-DTrilinos_TRACE_ADD_TEST=ON \
-DDART_TESTING_TIMEOUT:STRING=600.0 \
"$@" \
/home/rabartl/Trilinos.base/Trilinos
I ran the configure with:
$ cd /home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/pull-request-cuda-9.2/
$ time ./do-configure -DTrilinos_ENABLE_ALL_PACKAGES=ON &> configure.out
real 4m58.255s
user 2m35.155s
sys 0m26.741s
I created the script create_normalized_cmake_output_files.sh
:
#!/bin/bash
# Get build-dir name from argument
BUILD_DIR_NAME=$1
echo "BUILD_DIR_NAME='${BUILD_DIR_NAME}'"
set -x
cat CMakeCache.txt | grep -v "^$" | grep -v "^//" | grep -v "^#" | sort \
> CMakeCache.normalized.txt
~/Trilinos.base/Trilinos/commonTools/refactoring/token-replace.pl \
${BUILD_DIR_NAME} GENERIC_BUILD_DIR \
CMakeCache.normalized.txt CMakeCache.normalized.txt
~/Trilinos.base/Trilinos/commonTools/refactoring/token-replace.pl \
${BUILD_DIR_NAME} GENERIC_BUILD_DIR \
configure.out configure.normalized.out
I then ran it as:
$ cd /home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/
$ cd cuda-9.2-gnu-7.2.0-release-debug-pt/
$ ../../../create_normalized_cmake_output_files.sh cuda-9.2-gnu-7.2.0-release-debug-pt
$ cd ..
$ cd pull-request-cuda-9.2/
$ ../../../create_normalized_cmake_output_files.sh pull-request-cuda-9.2
$ cd ...
I then compare the two sets of files with:
$ cd /home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/
$ diff \
cuda-9.2-gnu-7.2.0-release-debug-pt/configure.normalized.out \
pull-request-cuda-9.2/configure.normalized.out \
| less
$ diff \
cuda-9.2-gnu-7.2.0-release-debug-pt/CMakeCache.normalized.txt \
pull-request-cuda-9.2/CMakeCache.normalized.txt \
| less
One difference I noted was:
74c50
< Explicitly disabled packages on input (by user or by default): Claps Trios TriKota NewPackage 4
---
> Explicitly disabled packages on input (by user or by default): Claps Trios TriKota PyTrilinos NewPackage 5
It is not necessary to explicitly disable PyTrilinos in this context because it is not a Primary Tested package (so it would not get enabled). But that should be harmless as it will not impact the final set of enabled and non-enabled SE Packages and TPLs.
Using the script configure-pr-build-and-diff.sh
:
#!/bin/bash
cd pull-request-cuda-9.2/
. load-env.sh
rm -r CMake*
time ./do-configure -DTrilinos_ENABLE_ALL_PACKAGES=ON &> configure.out
../../../create_normalized_cmake_output_files.sh pull-request-cuda-9.2
cd ..
diff \
cuda-9.2-gnu-7.2.0-release-debug-pt/configure.normalized.out \
pull-request-cuda-9.2/configure.normalized.out \
| less
diff \
cuda-9.2-gnu-7.2.0-release-debug-pt/CMakeCache.normalized.txt \
pull-request-cuda-9.2/CMakeCache.normalized.txt \
| less
I did several of iterations of modifying the file PullRequestLinuxCuda9.2TestingSettings.cmake
and running:
$ cd /home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/
$ configure-pr-build-and-diff.sh
and carefully inspecting the diffs until I got them pretty close. The final diffs are shown in :
The diffs that remained should not impact what builds and what passes and fails.
I then did a full build and ran the test suite on 'ride' using the script run_all.sh
:
#!/bin/bash -e
. load-env.sh
rm -r CMake* || echo "no CMake files to remove!"
time ./do-configure -DTrilinos_ENABLE_ALL_PACKAGES=ON &> configure.out
time ninja -j64 &> make.out
time ctest -j8 &> ctest.out
and I ran this with:
$ cd /home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/pull-request-cuda-9.2/
$ bsub -x -Is -q rhel7F -n 16 ./run_all.sh
***Forced exclusive execution
Job <854522> is submitted to queue <rhel7F>.
<<Waiting for dispatch ...>>
<<Starting on ride12>>
rm: cannot remove ‘CMake*’: No such file or directory
no CMake files to remove!
real 4m19.775s
user 2m36.281s
sys 0m36.791s
real 172m17.763s
user 8615m25.206s
sys 865m48.777s
That gave the test results:
99% tests passed, 1 tests failed out of 2936
Subproject Time Summary:
Amesos = 27.54 sec*proc (13 tests)
Amesos2 = 35.31 sec*proc (8 tests)
Anasazi = 332.37 sec*proc (74 tests)
AztecOO = 26.23 sec*proc (17 tests)
Belos = 415.65 sec*proc (100 tests)
Domi = 232.67 sec*proc (125 tests)
Epetra = 85.45 sec*proc (63 tests)
EpetraExt = 26.28 sec*proc (10 tests)
FEI = 43.59 sec*proc (43 tests)
Galeri = 12.38 sec*proc (9 tests)
GlobiPack = 2.83 sec*proc (6 tests)
Ifpack = 99.66 sec*proc (48 tests)
Ifpack2 = 363.30 sec*proc (45 tests)
Intrepid = 383.97 sec*proc (143 tests)
Intrepid2 = 544.01 sec*proc (267 tests)
Isorropia = 13.20 sec*proc (6 tests)
Kokkos = 170.39 sec*proc (27 tests)
KokkosKernels = 167.29 sec*proc (8 tests)
ML = 75.86 sec*proc (34 tests)
MiniTensor = 3.52 sec*proc (2 tests)
MueLu = 2782.78 sec*proc (105 tests)
NOX = 290.93 sec*proc (106 tests)
OptiPack = 7.93 sec*proc (5 tests)
Panzer = 8737.40 sec*proc (163 tests)
Phalanx = 19.35 sec*proc (27 tests)
Pike = 3.77 sec*proc (7 tests)
Piro = 47.12 sec*proc (13 tests)
ROL = 1306.07 sec*proc (164 tests)
RTOp = 19.57 sec*proc (24 tests)
Rythmos = 68.87 sec*proc (83 tests)
SEACAS = 22.87 sec*proc (23 tests)
STK = 95.71 sec*proc (15 tests)
Sacado = 169.57 sec*proc (300 tests)
Shards = 1.42 sec*proc (4 tests)
ShyLU_DD = 300.03 sec*proc (37 tests)
Stokhos = 156.84 sec*proc (84 tests)
Stratimikos = 37.45 sec*proc (39 tests)
Teko = 592.89 sec*proc (18 tests)
Tempus = 402.99 sec*proc (80 tests)
Teuchos = 161.02 sec*proc (137 tests)
Thyra = 102.25 sec*proc (82 tests)
Tpetra = 1158.23 sec*proc (201 tests)
TrilinosCouplings = 32.43 sec*proc (22 tests)
TrilinosFrameworkTests = 5.50 sec*proc (4 tests)
Triutils = 3.83 sec*proc (2 tests)
Xpetra = 262.52 sec*proc (18 tests)
Zoltan = 345.53 sec*proc (14 tests)
Zoltan2 = 553.05 sec*proc (111 tests)
Total Test time (real) = 2654.55 sec
The following tests FAILED:
2036 - STKUnit_tests_stk_ngp_test_utest_MPI_4 (Failed
Errors while running CTest
See, we now have 23 SEACAS tests, 15 STK tests and 163 Panzer tests! Before there were only 60 Panzer tests as shown, for example, in this recent CUDA PR build and no SEACAS or STK tests.
The only failing test was STKUnit_tests_stk_ngp_test_utest_MPI_4
which is already known to be failing as described in #4551. Therefore, I added a disable for that test as well. To check that disable I did a new configure with:
$ cd /home/rabartl/Trilinos.base/BUILDS/RIDE/CUDA/pull-request-cuda-9.2/
$ . load-env.sh
$ cmake . &> configure.reconfig.out
which showed:
$ grep STKUnit_tests_stk_ngp_test_utest_MPI_4 configure.reconfig.out
-- STKUnit_tests_stk_ngp_test_utest_MPI_4: Added test (BASIC, NUM_MPI_PROCS=4, PROCESSORS=4)!
I then cleaned up the commits and created the PR #4592.
As I said on the other issue, we will try to get a stk update in, to fix the failing test, asap.
@trilinos/framework,
With more code being enabled in the CUDA PR build due to the changes in PR #4592 the build times have gone up a lot as shown in the PR build #4592 on CDash. That shows a build time of 4h 49s
! But if you look at the history of the build Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt
this is supposed to be duplicating over the last 3 weeks you can see the build times are just under 3 hours.
This must mean that the number of build processes is not correct. If you look at the Jenkins build for these at:
and look at the output for example at:
you can see:
03:03:07 -- CTEST_BUILD_FLAGS='-j64 -k 999999'
...
03:03:07 -- CTEST_PARALLEL_LEVEL='8'
So you want to set 64 build processes and 8 parallel ctest MPI processes (i.e. ctest -j8
).
CC: @dridzal, @rppawlo
@trilinos/framework,
The updated CUDA PR build in PR #4592 failed with 6 failing tests, 1 timing-out ROL test, 3 timing out Panzer tests and 2 failing Panzer tests that show CUDA allocation "out of memory" failures. This is likely due to using too high of a parallel level with ctest -j<N>
. From looking at the Jenkins output for this build here it showed:
Parallel level = 29
If that means that is is using ctest -j29
that is way too high. This needs to be lowered to ctest -j8
as described above. That will increase the test wallclock time a little but it will result in all passing tests.
Can someone on the @trilinos/framework team please update this CUDA PR build to use 64 parallel processes and only 8 parallel ctest processes on 'ride'? That will result in a total wall-clock time of a little over 4 hours in the worst case where now the CUDA PR build looks like it takes almost 5 hours and results in failing and timing out tests.
@trilinos/framework, is this done done? Has the parallel test level been reduced to 8 (or so) and have all the temp disables been removed?
This has been done for a while. The CUDA PR build looks to be one of the most robust PR builds being used. Closing as complete.
Horray!
CC: @trilinos/framework, @mhoemmen, @rppawlo, @ibaned, @crtrott, @nmhamster
Description
This Issue is to scope out and track efforts to set up a CUDA build of Trilinos to be used as an auto PR build as described in https://github.com/trilinos/Trilinos/issues/2317#issuecomment-376551457.
For this build it was agreed to use that ATDM build on
white
that is currently running and submitting to CDash. Questions about how to extend this build to be used as an auto PR build include:bsub
command robust enough to be a reliable PR build?Tasks:
white
until it is 100% clean [Done]Related Issues: