search

E3SM-Project / scream

Fork of E3SM used to develop exascale global atmosphere model written in C++
https://e3sm-project.github.io/scream/
Other
80 stars 55 forks source link

List of unit tests that fail with specific random seeds #1506

Closed AaronDonahue closed 2 years ago

AaronDonahue commented 2 years ago

The shoc_tests test suite fails for the sp test suite on lassen when the random seed is 376024247.

The fail occurs in the update_prognostics_implicit_bfb TEST_CASE with the following:

SCREAM_SOURCE/components/scream/src/physics/shoc/tests/shoc_update_prognostics_implicit_tests.cpp:394: FAILED:
  REQUIRE( d_f90.thetal[k] == d_cxx.thetal[k] )
with expansion:
  nanf == nanf

The line to recreate is:

./shoc_tests --rng-seed 376024247

bartgol commented 2 years ago

Have you tried running ./shoc_tests --rng-seed 376024247 on another machine, to verify the problem are not the inputs? Might be that's a very unlucky inputs combination. You could also verify the same thing on lassen, but with a dp build.

AaronDonahue commented 2 years ago

@bartgol , just tried...

AaronDonahue commented 2 years ago

Adding to this issue, p3_main test fails on blake for the sp test suite with seed 653534356


 For test p3_main, random number generator seed: 653534356
    Note: catch rng seed was 0 (default). We interpret that as a request to pick a random seed.
    To reproduce a previous run, use --rng-seed N to provide the rng seed.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
p3_tests is a Catch v2.13.8 host application.
Run with -? for options

-------------------------------------------------------------------------------
p3_main
-------------------------------------------------------------------------------
/home/e3sm-jenkins/blake/workspace/SCREAM_NightlyTests_Blake/scream/components/scream/src/physics/p3/tests/p3_main_unit_tests.cpp:463
...............................................................................

/home/e3sm-jenkins/blake/workspace/SCREAM_NightlyTests_Blake/scream/components/scream/src/physics/p3/tests/p3_main_unit_tests.cpp:218: FAILED:
  REQUIRE( isds_fortran[i].qi[k] == isds_cxx[i].qi[k] )
with expansion:
  0.0f == 0.0f

see the CDash report for May 19, 2022 for more details.

AaronDonahue commented 2 years ago

sp fails on quartz for the shoc_main_bfb test in shoc_tests with seed 1657744704

/p/lustre2/donahue5/E3SM_simulations/SCREAM/nightly_tests/quartz/scream/components/scream/src/physics/shoc/tests/shoc_main_tests.cpp:439: FAILED:
  REQUIRE( d_f90.shoc_ql2[k] == d_cxx.shoc_ql2[k] )
with expansion:
  nanf == 0.0f
jgfouca commented 2 years ago

Thanks @AaronDonahue , I will look at this.

AaronDonahue commented 2 years ago 
 For test p3_main, random number generator seed: 3269801319

Failed on mappy for full_sp_debug suite.