??: Add label "client: ATDM">
??: Add label "ATDM Sev: Blocker" (by default but could be other "ATDM Sev: XXX")>
??: Add label "type: bug"?>
??: Add label "impacting: tests"?>
??: Add label for affected packages (e.g. "pkg: MueLu", "pkg: Tpetra", "pkg: Kokkos", etc.)>
??: Add label "PA: ???Project Area???" (e.g. "PA: Linear Solvers", "PA: Data Services")>
??: Add milestone "Initial cleanup of new ATDM ..." or "Keep promoted ATDM ...">
??: Once GitHub Issue is created, add entries for tests to TrilinosATDMStatus/*.csv files>
## Next Action Status
## Description
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&begin=2020-05-01&end=2020-12-14&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-&field3=testname&compare3=61&value3=NOX_Tpetra_HouseholderBorderedSolve_MPI_4&field4=status&compare4=62&value4=passed&field5=testoutput&compare5=97&value5=solver-.getSolverStatistics..-.numNonlinearIterations%20%3D%20.*%20%3D%3D%205%20%3D%205%20%3A%20FAILED%20%3D%3D.%20.*Tpetra_HouseholderBorderedSolve.cpp) (click "Shown Matching Output" in upper right) the test:
* `NOX_Tpetra_HouseholderBorderedSolve_MPI_4`
in the builds:
* `Trilinos-atdm-cee-rhel6_intel-19.0.3_mpich2-3.2_openmp_static_opt`
* `Trilinos-atdm-cts1-intel-19.0.4_openmpi-4.0.3_openmp_static_opt`
* `Trilinos-atdm-sems-rhel6-intel-17.0.1-openmp-release`
* `Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug`
* `Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug`
* `Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release`
* `Trilinos-atdm-van1-tx2_arm-20.0_openmpi-4.0.2_openmp_static_opt`
* `Trilinos-atdm-van1-tx2_arm-20.1_openmpi-4.0.3_openmp_static_opt`
* `Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.15_openmp_static_opt`
is randomly failing starting before 2020-10-03.
As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&begin=2020-05-01&end=2020-12-14&filtercount=3&showfilters=1&filtercombine=and&field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-&field3=testname&compare3=61&value3=NOX_Tpetra_HouseholderBorderedSolve_MPI_4), it failed 42 times out of 2227 test runs over this time period with the first failure coming on 2020-10-03. But CDash history only goes back to 2020-10-03 so it likely started randomly failing before that.
When it does fail, it shows different numbers of iterations with error messages like shown [here](https://testing.sandia.gov/cdash/test/46896937) showing:
```
solver->getSolverStatistics()->numNonlinearIterations = 8 == 5 = 5 : FAILED ==> /lustre/jenkins/stria/workspace/Trilinos-atdm-van1-tx2_arm-20.1_openmpi-4.0.3_openmp_static_opt/SRC_AND_BUILD/Trilinos/packages/nox/test/tpetra/tTpetra_HouseholderBorderedSolve.cpp:292
```
## Current Status on CDash
Run the [above query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&begin=2020-05-01&end=2021-02-8&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-&field3=testname&compare3=61&value3=NOX_Tpetra_HouseholderBorderedSolve_MPI_4&field4=status&compare4=62&value4=passed&field5=testoutput&compare5=97&value5=solver-.getSolverStatistics..-.numNonlinearIterations%20%3D%20.*%20%3D%3D%205%20%3D%205%20%3A%20FAILED%20%3D%3D.%20.*Tpetra_HouseholderBorderedSolve.cpp) adjusting the "Begin" and "End" dates to match today any other date range or just click "CURRENT" in the top bar to see results for the current testing day.
## Steps to Reproduce
One should be able to reproduce this failure as described in:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
and the system-specific instructions at:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system
Just log into any of the associated machines and copy and paste the full CDash build name `` listed above and run commands like:
```
$ cd /
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh
$ cmake \
-GNinja \
-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
-DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_=ON \
$TRILINOS_DIR
$ make NP=16
$ ctest -j4
```
where `` is any package that you want to enabled to reproduce build and/or test results.
Again, for exact system-specific details on what commands to run to build and run tests, see:
* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system
And if you can't figure out what commands to run to produce the issue given the above-referenced documentation, please post a comment here and we will give you the exact minimal commands to reproduce the failures.
CC: @trilinos/nox, @rppawlo (Trilinos Nonlinear Product Lead)