LLNL / serac

Serac is a high order nonlinear thermomechanical simulation code
BSD 3-Clause "New" or "Revised" License
173 stars 30 forks source link

Nondeterministic test failure #1157

Open btalamini opened 2 weeks ago

btalamini commented 2 weeks ago

During a PR, I got an unexpected failure in a test in the file dynamic_solid_adjoint.cpp. I say "unexpected", becaues the code introduced in my PR isn't used in that test. The failure looks like this:

48: /usr/WS1/bramwell/gl/builds/xYGwyxK_/000/gitlab/smith/serac/src/serac/physics/tests/dynamic_solid_adjoint.cpp:295: Failure
48: The difference between directional_deriv and (qoi_plus - qoi_base) / eps is nan, which exceeds eps, where
48: directional_deriv evaluates to -nan,
48: (qoi_plus - qoi_base) / eps evaluates to -0.023019492978448852, and
48: eps evaluates to 1.9999999999999999e-07.
48: 
48: [  FAILED  ] SolidMechanicsSensitivityFixture.ShapeSensitivities (7280 ms)

I'm suspicious that we have some kind of non-deterministic problem here, like a race condition. The entire log is included below.

output.log.make.test.txt