trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.2k stars 563 forks source link

NOX_Tpetra_HouseholderBorderedSolve_MPI_4 failing randomly in many ATDM Trilinos builds starting before 2020-10-03 #8492

Closed bartlettroscoe closed 2 years ago

bartlettroscoe commented 3 years ago

CC: @trilinos/nox, @rppawlo (Trilinos Nonlinear Product Lead)

## Next Action Status ## Description As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&begin=2020-05-01&end=2020-12-14&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-&field3=testname&compare3=61&value3=NOX_Tpetra_HouseholderBorderedSolve_MPI_4&field4=status&compare4=62&value4=passed&field5=testoutput&compare5=97&value5=solver-.getSolverStatistics..-.numNonlinearIterations%20%3D%20.*%20%3D%3D%205%20%3D%205%20%3A%20FAILED%20%3D%3D.%20.*Tpetra_HouseholderBorderedSolve.cpp) (click "Shown Matching Output" in upper right) the test: * `NOX_Tpetra_HouseholderBorderedSolve_MPI_4` in the builds: * `Trilinos-atdm-cee-rhel6_intel-19.0.3_mpich2-3.2_openmp_static_opt` * `Trilinos-atdm-cts1-intel-19.0.4_openmpi-4.0.3_openmp_static_opt` * `Trilinos-atdm-sems-rhel6-intel-17.0.1-openmp-release` * `Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug` * `Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug` * `Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release` * `Trilinos-atdm-van1-tx2_arm-20.0_openmpi-4.0.2_openmp_static_opt` * `Trilinos-atdm-van1-tx2_arm-20.1_openmpi-4.0.3_openmp_static_opt` * `Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.15_openmp_static_opt` is randomly failing starting before 2020-10-03. As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&begin=2020-05-01&end=2020-12-14&filtercount=3&showfilters=1&filtercombine=and&field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-&field3=testname&compare3=61&value3=NOX_Tpetra_HouseholderBorderedSolve_MPI_4), it failed 42 times out of 2227 test runs over this time period with the first failure coming on 2020-10-03. But CDash history only goes back to 2020-10-03 so it likely started randomly failing before that. When it does fail, it shows different numbers of iterations with error messages like shown [here](https://testing.sandia.gov/cdash/test/46896937) showing: ``` solver->getSolverStatistics()->numNonlinearIterations = 8 == 5 = 5 : FAILED ==> /lustre/jenkins/stria/workspace/Trilinos-atdm-van1-tx2_arm-20.1_openmpi-4.0.3_openmp_static_opt/SRC_AND_BUILD/Trilinos/packages/nox/test/tpetra/tTpetra_HouseholderBorderedSolve.cpp:292 ``` ## Current Status on CDash Run the [above query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&begin=2020-05-01&end=2021-02-8&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-&field3=testname&compare3=61&value3=NOX_Tpetra_HouseholderBorderedSolve_MPI_4&field4=status&compare4=62&value4=passed&field5=testoutput&compare5=97&value5=solver-.getSolverStatistics..-.numNonlinearIterations%20%3D%20.*%20%3D%3D%205%20%3D%205%20%3A%20FAILED%20%3D%3D.%20.*Tpetra_HouseholderBorderedSolve.cpp) adjusting the "Begin" and "End" dates to match today any other date range or just click "CURRENT" in the top bar to see results for the current testing day. ## Steps to Reproduce One should be able to reproduce this failure as described in: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md and the system-specific instructions at: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system Just log into any of the associated machines and copy and paste the full CDash build name `` listed above and run commands like: ``` $ cd / $ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh $ cmake \ -GNinja \ -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \ -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_=ON \ $TRILINOS_DIR $ make NP=16 $ ctest -j4 ``` where `` is any package that you want to enabled to reproduce build and/or test results. Again, for exact system-specific details on what commands to run to build and run tests, see: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system And if you can't figure out what commands to run to produce the issue given the above-referenced documentation, please post a comment here and we will give you the exact minimal commands to reproduce the failures.
bartlettroscoe commented 3 years ago

NOTE: Trilinos-atdm-sems-rhel6-intel-17.0.1-openmp-release is not a valid build anymore. That was just one of the builds where this test failed in the past.

NOTE: This was a bit of a harder one to triage for several reasons. But with all of that it took less than 40 minutes to get here to a very good Issue I think.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2020-12-21

Tests with issue trackers Passed: twip=3
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 5 24 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 24 1 29 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 8 2 28 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel6 Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 11 17 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2020-12-27

Tests with issue trackers Passed: twip=4

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel6 Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 10 18 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 5 24 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 3 27 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-01-10

Tests with issue trackers Passed: twip=5
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=5

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 3 5 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 18 2 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 1 5 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 2 8 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 3 27 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 2 5 1 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 3 27 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-01-17

Tests with issue trackers Passed: twip=7

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=7

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 4 11 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 25 1 29 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 9 1 12 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 10 3 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 11 2 11 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 3 27 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 5 3 27 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-01-24

Tests with issue trackers Passed: twip=3
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 2 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 2 18 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 3 17 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 2 6 16 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 5 16 4 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-01-31

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=3

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 11 2 25 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 5 22 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 18 3 24 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 8 3 24 #8492

Tests with issue trackers Failed: twif=3

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 7 20 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 3 25 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 2 21 6 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-02-07

Tests with issue trackers Passed: twip=6
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=6

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 3 25 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 2 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 23 7 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 8 3 27 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 24 1 25 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 3 23 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 7 21 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

e10harvey commented 3 years ago

@rppawlo is there an assignee for this? This cropped up last night in Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.15_openmp_static_opt: https://testing-dev.sandia.gov/cdash/test/42900279.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-02-14

Tests with issue trackers Passed: twip=5
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=5

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 4 24 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 8 2 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 22 7 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 3 26 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 26 0 26 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 8 20 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 3 23 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-02-21

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 2 22 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 6 4 24 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 25 0 25 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 2 22 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 11 17 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-02-28

Tests with issue trackers Passed: twip=3
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 2 23 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 13 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 18 1 24 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 4 21 4 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 3 22 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-03-07

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 21 2 23 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 2 23 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 4 21 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 2 10 10 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 3 20 5 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-03-14

Tests with issue trackers Passed: twip=7
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=7

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 2 24 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 9 11 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 10 2 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 9 2 24 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 9 4 22 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 16 0 16 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 13 1 14 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 10 22 4 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-03-21

Tests with issue trackers Passed: twip=5
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=5

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 11 2 28 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 9 11 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 15 2 24 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 16 4 23 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 18 0 18 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 17 24 3 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 1 17 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-03-28

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=4

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 9 14 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 4 25 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 1 19 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 20 0 20 #8492

Tests with issue trackers Failed: twif=4

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 4 24 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 2 25 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 3 26 4 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 4 26 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-04-04

Tests with issue trackers Passed: twip=6
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=6

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 4 24 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 1 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 8 2 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 1 29 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 1 26 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 27 0 27 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 12 18 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 4 26 4 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

rppawlo commented 3 years ago

I've spent some time on this with no luck so far. To get a consistent failure on my workstation, we have to run the problem 3000 times in a loop. I've fenced every kokkos/Tpetra operation in the nox code and still get the failure. Suspect that we need a fence either in thyra adapters or in the linear solver. The problem actually always converges to the correct solution, but it seems a race condition sometimes causes a Newton iteration return a different answer resulting in an increased total number of Newton iterations. The test checks the number of Newton steps as part of the solution and that is what is triggering the failed result in testing.

etphipp commented 3 years ago

How much does the number of iterations change?

rppawlo commented 3 years ago

Jumps from 5 to 10 Newton steps

rppawlo commented 3 years ago

Let me revise that. I just looked at some more of the failing tests and it does not always converge to the correct solution. 10 was the upper bound.

rppawlo commented 3 years ago

Here's the test on the same system (KNL with intel 19, results from different nights of the week) where it passed needing only a single extra iteration and another where it never recovered. My workstation consistently fails with 6 iterations with openmpi and gcc 10:

 solvStatus == NOX::StatusTest::Converged = 1 == true : passed
 solver->getSolverStatistics()->numNonlinearIterations = 6 == 5 = 5 : FAILED ==> /lscratch1/jenkins/workspace/Trilinos-atdm-ats1-driver/Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.15_openmp_static_opt/SRC_AND_BUILD/Trilinos/packages/nox/test/tpetra/tTpetra_HouseholderBorderedSolve.cpp:292

 Final Parameter Value for "k" = -0.599327582
 Final Parameter Value for "T_left" = 1

 Check: rel_err(c_group.getParam(0), -0.5993277206)
        = rel_err(-0.599327582, -0.5993277206) = 2.312379921e-07
          <= tol = 0.001 : passed

 Check: rel_err(c_group.getParam(1), 1.0)
        = rel_err(1, 1) = 1.110223025e-16
          <= tol = 0.001 : passed
 [FAILED]  (4.88 sec) NOX_Tpetra_Householder_BasicSolve_UnitTest
 solvStatus == NOX::StatusTest::Converged = 1 == true : passed
 solver->getSolverStatistics()->numNonlinearIterations = 10 == 5 = 5 : FAILED ==> /lscratch1/jenkins/workspace/Trilinos-atdm-ats1-driver/Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.15_openmp_static_opt/SRC_AND_BUILD/Trilinos/packages/nox/test/tpetra/tTpetra_HouseholderBorderedSolve.cpp:292

 Final Parameter Value for "k" = -0.5993274938
 Final Parameter Value for "T_left" = 1

 Check: rel_err(c_group.getParam(0), -0.5993277206)
        = rel_err(-0.5993274938, -0.5993277206) = 3.783598679e-07
          <= tol = 0.001 : passed

 Check: rel_err(c_group.getParam(1), 1.0)
        = rel_err(1, 1) = 0
          <= tol = 0.001 : passed
 [FAILED]  (5.03 sec) NOX_Tpetra_Householder_BasicSolve_UnitTest
rppawlo commented 3 years ago

We still haven't seen a failure in cuda. This only happens with openmp backend on intel/arm/clang.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-04-11

Tests with issue trackers Passed: twip=7
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=7

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 2 26 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 5 10 20 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 2 25 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 15 2 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 6 2 28 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 21 1 29 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 25 5 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-04-18

Tests with issue trackers Passed: twip=7
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=7

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 21 2 26 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 8 22 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 10 2 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 13 2 28 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 28 1 29 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 8 25 5 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-04-25

Tests with issue trackers Passed: twip=6
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=6

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 17 2 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 9 2 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 24 6 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 20 2 28 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 9 21 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-05-02

Tests with issue trackers Passed: twip=6
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=6

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 10 20 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 2 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 2 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 25 5 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 5 22 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 3 27 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-05-09

Tests with issue trackers Passed: twip=7
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=7

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 7 20 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 24 6 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 4 26 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 30 0 30 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 12 18 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-05-18

Tests with issue trackers Passed: twip=3

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 5 12 18 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 10 2 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 5 24 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-05-23

Tests with issue trackers Passed: twip=4

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 11 19 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 3 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 15 2 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 5 24 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-05-30

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 10 20 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 3 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 4 25 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 24 0 24 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 3 25 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 1 23 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-06-06

Tests with issue trackers Passed: twip=6

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=6

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 11 19 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 2 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 8 2 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 3 26 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 1 23 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 24 0 24 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-06-13

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 15 2 28 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-06-20

Tests with issue trackers Passed: twip=6
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=6

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 11 15 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 3 23 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 21 2 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 1 26 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 17 1 23 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 24 0 24 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 4 2 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-06-27

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 11 15 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 10 3 24 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 5 2 27 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 24 0 24 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 2 9 4 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-07-04

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 11 15 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 17 2 25 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 2 27 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 23 0 23 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 4 13 7 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-07-11

Tests with issue trackers Passed: twip=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=5

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 11 15 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 23 2 25 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 3 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 16 11 #8492
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 23 0 23 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-07-18

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 6 10 20 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 9 3 27 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 1 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 7 20 10 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-07-25

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 0 4 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 12 18 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 1 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 3 27 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 21 9 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-08-01

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 11 0 11 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 3 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 5 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 22 8 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 13 17 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-08-08

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 5 2 16 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 14 16 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 9 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 12 2 28 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 7 24 6 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-08-08

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 5 2 16 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 14 16 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 9 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 12 2 28 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 7 24 6 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-08-08

Tests with issue trackers Passed: twip=4
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=4

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 5 2 16 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 14 16 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 9 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 12 2 28 #8492

Tests with issue trackers Failed: twif=1

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 7 24 6 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-08-29

Tests with issue trackers Passed: twip=3
Tests with issue trackers Failed: twif=3

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 6 24 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 7 1 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 12 1 29 #8492

Tests with issue trackers Failed: twif=3

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 11 19 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 16 28 2 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 2 28 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-09-05

Tests with issue trackers Passed: twip=3
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 1 7 23 #8492
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 1 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 4 2 28 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 4 11 18 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 26 4 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-09-12

Tests with issue trackers Passed: twip=3
Tests with issue trackers Failed: twif=3

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 21 1 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 3 27 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 14 1 29 #8492

Tests with issue trackers Failed: twif=3

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
mutrino Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 9 21 #8492
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 2 14 15 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 8 27 3 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-10-03

Tests with issue trackers Passed: twip=3
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 28 0 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 4 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 1 29 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 16 14 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 6 28 2 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8492 as of 2021-10-03

Tests with issue trackers Passed: twip=3
Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=3

Site Build Name Test Name Status Details Consec­utive Pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
eclipse Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 28 0 28 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 2 4 26 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-intel-18.0.5-openmp-shared-release-debug NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Passed Completed 3 1 29 #8492

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
cee-rhel7 Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 1 16 14 #8492
sems-rhel7 Trilinos-atdm-sems-rhel7-clang-7.0.1-openmp-shared-release NOX_­Tpetra_­HouseholderBorderedSolve_­MPI_­4 Failed Completed (Failed) 6 28 2 #8492

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

JacobDomagala commented 2 years ago

I've spent some time trying to fix this issue, but no luck so far. I've tried to fence all potential race conditions in nox/src-loca/src-tpetra but the failure still occurs.

Few notes:

I'll try to maybe run it with Thread Sanitizer (though using TS with OpenMP with default setting, can produce lots of false positives, see this SO discussion)

rppawlo commented 2 years ago

This is fixed. Closing.