bartlettroscoe
commented
6 years ago This test Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4 newly failed in the build Trilinos-atdm-white-ride-cuda-debug on 'white' today as shown at:
showing:
Generating Y1,Y2 for project() : testing...
|| <Y1,Y1> - I || : 7.13673e-16
|| <Y2,Y2> - I || : 7.85286e-16
|| <X1,Y2> || : 1.71386e-16
|| <X1b,Y2> || : 7.10285e-15
p=1: *** Caught standard std::exception of type 'std::runtime_error' :
/home/rabartl/WHITE/ATDM_Driver/Trilinos-atdm-white-ride-cuda-debug/SRC_AND_BUILD/Trilinos/packages/anasazi/epetra/test/OrthoManager/cxx_gentest.cpp:274:
Throw number = 1
Throw test that evaluated to true: err > TOL
New X1 did not meet tolerance: orthog(X1,Y2) == 0.356233
...It passed yesterday in the same build as shown at:
Looking at the history of this test on this build on 'white' in the query:
it fails three other times on various days going back to 3/12/2018. This suggests non-deterministic behavior causing the test to randomly fail.
Does this test cause some non-deterministic behavior about Anasazi or the underlying software being used? Could this be exposing a weakness in Trilinos software that could bite a user in a CUDA build?
In any case, I think this test should be disabled for now on these CUDA debug builds so that we can promote this build Trilinos-atdm-white-ride-cuda-debug to the "ATDM" CDash Track/Group which opens the door to using it as an auto PR build for Trilinos (which will be huge for stabilizing Trilinos for ATDM customers). Then, someone can debug this test offline when they get some time.
@mhoemmen, what do you think about this? Is it okay to disable this test for now until someone can debug what is causing the non-deterministic behavior?
mhoemmen
srajama1
hkthorn
CC: @trilinos/anasazi, @mhoemmen
Next Action Status
PR #2621 merged on 4/24/2018 that re-enables the tests
Anasazi_Epetra_ModalSolversTester_MPI_4andAnasazi_Epetra_OrthoManagerGenTester_[0,1]_MPI_4. Tests ran and passed in all promoted ATDM Trilinos builds between 5/20/2018 and 6/7/2018.Description
The tests:
Anasazi_Epetra_ModalSolversTester_MPI_4Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4Anasazi_Epetra_OrthoManagerGenTester_1_MPI_4failed in
Trilinos-atdm-hansen-shiller-cuda-debugbuild on 'ride' as shown at:This build is targeted to be an auto PR build for Trilinos (see #2464) so we desire to clean up this build more quickly.
Intrestingly, these tests did not fail in what should be the idential
Trilinos-atdm-hansen-shiller-cuda-debugbuild on the identical machine 'white' as shown at:Strangely, those tests did fail on
Trilinos-atdm-hansen-shiller-cuda-debugbuild on 'white' yestrday shown at:A) Anasazi_Epetra_ModalSolversTester_MPI_4:
Test failing test
Anasazi_Epetra_ModalSolversTester_MPI_4today with details shown at:showed the failure:
Looking at all of the builds today that ran that test shown at:
this test fails in the same way (i.e. a numerical problem) on the builds
Linux-gcc-4.8.4-MPI_RELEASE_12.12.1andLinux-gcc-4.8.4-MPI_RELEASE_12.12.1_SHAREDon the machinehansel.sandia.govso this problem is not isolated to ATDM builds of Trilinos.Also note that this test failed for the ATDM builds
Trilinos-atdm-white-ride-gnu-opt-openmpandTrilinos-atdm-white-ride-gnu-opt-openmpwith segfaults, but that is already being addressed by #2454 and is likely unrelated.B) Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4:
The failing test
Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4today with details shown at:showed:
Looking at all of the builds today that ran that test shown at:
you can see that this test also failed in a similar (numerical) way in the builds
Linux-gcc-4.9.3-Sierra_MPI_release_DEV_ETI_SERIAL-ON_OPENMP-ON_PTHREAD-OFF_CUDA-OFF_COMPLEX-ONandLinux-GCC-4.9.3-openmpi-1.8.7_Debug_DEV_Werrorso it looks like this problem is not isolated to ATDM builds of Trilinos. Note that one of those is a "Sierra' build of Trilinos.