search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.2k stars 563 forks source link

ROL tests timing out in ASTRA van1-tx2 debug builds #8488

Closed bartlettroscoe closed 2 years ago

bartlettroscoe commented 3 years ago

CC: @trilinos/rol, @rppawlo (Trilinos Nonlinear Product Lead)

## Next Action Status ## Description As shown in [this query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&begin=2020-11-16&end=2020-12-14&filtercount=6&showfilters=1&filtercombine=and&field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-van1-tx2&field3=status&compare3=62&value3=passed&field4=details&compare4=63&value4=Timeout&field5=testoutput&compare5=96&value5=srun%3A%20error.*launch%20failed%3A%20Error%20configuring%20interconnect&field6=testoutput&compare6=94&value6=Signal%3A%20Aborted%20(6)) (click "Shown Matching Output" in upper right) the tests: * `ROL_example_PDE-OPT_helmholtz_example_02_MPI_1` * `ROL_example_PDE-OPT_navier-stokes_example_01_MPI_4` in the builds: * `Trilinos-atdm-van1-tx2_arm-20.0_openmpi-4.0.2_openmp_static_dbg` * `Trilinos-atdm-van1-tx2_arm-20.1_openmpi-4.0.3_openmp_static_dbg` are timing out since these 'van1-tx2' builds were first setup. NOTE: The timing out test `ROL_example_PDE-OPT_helmholtz_example_02_MPI_1` is also covered in #6216. ## Current Status on CDash Run the [above query](https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&begin=2020-11-16&end=2020-12-14&filtercount=6&showfilters=1&filtercombine=and&field1=groupname&compare1=62&value1=Experimental&field2=buildname&compare2=65&value2=Trilinos-atdm-van1-tx2&field3=status&compare3=62&value3=passed&field4=details&compare4=63&value4=Timeout&field5=testoutput&compare5=96&value5=srun%3A%20error.*launch%20failed%3A%20Error%20configuring%20interconnect&field6=testoutput&compare6=94&value6=Signal%3A%20Aborted%20(6)) adjusting the "Begin" and "End" dates to match today any other date range or just click "CURRENT" in the top bar to see results for the current testing day. ## Steps to Reproduce One should be able to reproduce this failure as described in: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md and the system-specific instructions at: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system Just log into any of the associated machines and copy and paste the full CDash build name `` listed above and run commands like: ``` $ cd / $ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh $ cmake \ -GNinja \ -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \ -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_=ON \ $TRILINOS_DIR $ make NP=16 $ ctest -j4 ``` where `` is any package that you want to enabled to reproduce build and/or test results. Again, for exact system-specific details on what commands to run to build and run tests, see: * https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system And if you can't figure out what commands to run to produce the issue given the above-referenced documentation, please post a comment here and we will give you the exact minimal commands to reproduce the failures.
rppawlo commented 3 years ago

@dridzal

dridzal commented 3 years ago

@bartlettroscoe @rppawlo We have had nothing but trouble with those tests, especially in debug mode. They tend to run 5x to 50x slower, but the slowdown is heavily compiler dependent and machine dependent. With ASTRA van1-tx2, I have no idea what to expect. Can you disable them for this platform? On a related note, ROL 2.0 is coming in early February. At that point we will significantly reduce test times across the board.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2020-12-21

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2020-12-27

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-01-10

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-01-17

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-01-31

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 27 27 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 27 27 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-02-07

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 26 26 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 26 26 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-02-14

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 26 26 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 26 26 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-02-21

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 25 25 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 24 24 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-03-14

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 16 16 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 14 14 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-03-21

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 18 18 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 17 17 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-03-28

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 20 20 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 20 20 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-04-04

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 27 27 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 27 27 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-04-11

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-04-18

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-04-25

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-05-02

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-05-09

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 30 30 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-05-30

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 24 24 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 24 24 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

ccober6 commented 3 years ago

@dridzal Do you have a timeframe when to can look at this?

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-06-06

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 24 24 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 24 24 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

ccober6 commented 3 years ago

@dridzal Sorry to bother you with this, but we are trying to clean things. It looks like a test is consistently failing on stria. Thanks, Curt.

dridzal commented 3 years ago

@ccober6 Is Stria's hardware significantly slower than that of other testbeds (e.g., 5 to 10 times)? These failures are due to timeouts in the debug build. I would like to reduce the default mesh size in the failing example, but it would be good to know what the hardware can handle.

ccober6 commented 3 years ago

Actually I am not sure on Stria's speed.

@rppawlo, do you know if Stria is significantly slower than that other testbeds (e.g., 5 to 10 times)?

rppawlo commented 3 years ago

I have not noticed a significant slow down on stria, but haven't look at it in detail at the debug builds. We have not had to disable any of the compute intensive high order panzer convergence tests for stria like we do on cuda for debug builds.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-06-20

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 24 24 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 24 24 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

ccober6 commented 3 years ago

@dridzal Sorry again. Do you have status on this?

dridzal commented 3 years ago

@ccober6 we haven't figured out why this example is so much slower on Stria. There is a Kokkos warning related to the build setup on this platform:

Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
  In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
  For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
  For unit testing set OMP_PROC_BIND=false

Do you know if there are certain platform settings we have to use in our cmake/ctest files? What are the defaults?

The order-of-magnitude slowdown versus conventional platforms is bothersome. I should note that the computational time for this example is in assembling Tpetra matrices and in solving the linear systems using Amesos2. There is very little time spent in ROL itself. If there are documented issues with Tpetra matrix assembly or Amesos2 on Astra/Stria, let us know. CC @trilinos/tpetra @trilinos/amesos2

ccober6 commented 3 years ago

@dridzal I think you mentioned the right folks (@trilinos/tpetra @trilinos/amesos2) hopefully they have some guidance.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-06-27

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 24 24 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 23 23 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-07-04

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 23 23 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 22 22 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

grover-trilinos commented 3 years ago

Test results for issue #8488 as of 2021-07-11

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site Build Name Test Name Status Details Consec­utive Non-pass Days Non-pass Last 30 Days Pass Last 30 Days Issue Tracker
stria Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 23 23 0 #8488
stria Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg ROL_­example_­PDE-OPT_­navier-stokes_­example_­01_­MPI_­4 Failed Completed (Timeout) 22 22 0 #8488

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

github-actions[bot] commented 2 years ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

github-actions[bot] commented 2 years ago

This issue was closed due to inactivity for 395 days.