Closed njzjz closed 3 months ago
source/tests/pt/model/test_linear_atomic_model.py (2)
`35-37`: Introduce a global seed for deterministic tests. This change is crucial for achieving deterministic behavior in tests, which aligns with the PR's objective to address non-deterministic unit tests. --- `197-198`: Use of global seed in random number generation. Proper use of the `GLOBAL_SEED` to ensure that the random number generation is consistent across test runs. This is a good practice for maintaining the reproducibility of tests.
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 82.66%. Comparing base (
eb474d4
) to head (c03ec57
). Report is 120 commits behind head on devel.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Several tests in test_smooth
failed using today as the seed...
FAILED source/tests/pt/model/test_smooth.py::TestEnergyModelDPA1::test - AssertionError: Tensor-likes are not close!
Mismatched elements: 2 / 30 (6.7%)
Greatest absolute difference: 4.697115381646633e-05 at index (4, 2) (up to 1e-05 allowed)
Greatest relative difference: 7.770803885056612e-05 at index (2, 2) (up to 0 allowed)
FAILED source/tests/pt/model/test_smooth.py::TestEnergyModelDPA1Excl12::test - AssertionError: Tensor-likes are not close!
Mismatched elements: 2 / 30 (6.7%)
Greatest absolute difference: 3.200459405494627e-05 at index (4, 2) (up to 1e-05 allowed)
Greatest relative difference: 5.073069526107276e-05 at index (4, 2) (up to 0 allowed)
FAILED source/tests/pt/model/test_smooth.py::TestEnergyModelDPA2_2::test - AssertionError: Tensor-likes are not close!
Mismatched elements: 4 / 30 (13.3%)
Greatest absolute difference: 5.451339182582782e-05 at index (4, 2) (up to 1e-05 allowed)
Greatest relative difference: 5.638892512663301e-05 at index (4, 2) (up to 0 allowed)
FAILED source/tests/pt/model/test_smooth.py::TestEnergyModelDPA1Excl1::test - AssertionError: Tensor-likes are not close!
Mismatched elements: 2 / 30 (6.7%)
Greatest absolute difference: 7.867066788103116e-05 at index (4, 2) (up to 1e-05 allowed)
Greatest relative difference: 0.00022819964422492295 at index (2, 2) (up to 0 allowed)
fyi @iProzd
Several tests in
test_smooth
failed using today as the seed...FAILED source/tests/pt/model/test_smooth.py::TestEnergyModelDPA1::test - AssertionError: Tensor-likes are not close! Mismatched elements: 2 / 30 (6.7%) Greatest absolute difference: 4.697115381646633e-05 at index (4, 2) (up to 1e-05 allowed) Greatest relative difference: 7.770803885056612e-05 at index (2, 2) (up to 0 allowed) FAILED source/tests/pt/model/test_smooth.py::TestEnergyModelDPA1Excl12::test - AssertionError: Tensor-likes are not close! Mismatched elements: 2 / 30 (6.7%) Greatest absolute difference: 3.200459405494627e-05 at index (4, 2) (up to 1e-05 allowed) Greatest relative difference: 5.073069526107276e-05 at index (4, 2) (up to 0 allowed) FAILED source/tests/pt/model/test_smooth.py::TestEnergyModelDPA2_2::test - AssertionError: Tensor-likes are not close! Mismatched elements: 4 / 30 (13.3%) Greatest absolute difference: 5.451339182582782e-05 at index (4, 2) (up to 1e-05 allowed) Greatest relative difference: 5.638892512663301e-05 at index (4, 2) (up to 0 allowed) FAILED source/tests/pt/model/test_smooth.py::TestEnergyModelDPA1Excl1::test - AssertionError: Tensor-likes are not close! Mismatched elements: 2 / 30 (6.7%) Greatest absolute difference: 7.867066788103116e-05 at index (4, 2) (up to 1e-05 allowed) Greatest relative difference: 0.00022819964422492295 at index (2, 2) (up to 0 allowed)
we may decrease the epsilon to reduce the difference..
shall we provide a global rand seed for all the tests?
so if one wants to change the seed he/she needs to change only one line, rather than replace all the
20240604
strings across the code.
we should parameterize the seed, also make it dynamic by pulling the current date if possible. That way, it's reproducible and covers more test cases.
make it dynamic by pulling the current date if possible
This will make the tests fail on random days, which should not be used in the automatic CI environment.
we may decrease the epsilon to reduce the difference..
I changed from 1e-5 to 1e-6 but seems not working.
Several tests in
test_smooth
failed using today as the seed...
I just realized that I forgot to set generator
for torch.randint
.
Fix #3847. This PR makes all unit tests deterministic. We have found non-deterministic unit tests have more disadvantages than advantages. Read more: https://softwareengineering.stackexchange.com/questions/221632/testing-deterministic-or-non-deterministic
Summary by CodeRabbit