fcs-proj / FastCaloSim

An experiment-independent library for fast calorimeter simulation
Apache License 2.0
2 stars 0 forks source link

Add reproducibility tests #27

Closed jbeirer closed 3 months ago

jbeirer commented 3 months ago

This pull requests adds reproducibility tests for the simulation and transportation tests. More precisely, a new JsonComparer class, compares the serialized transport and simulation data with the references which are saved in tests/refs. In case that differences are found, the tests will now fail and output logs like:

Value mismatch at [0].cell_energy: 8.743232727050781 vs 9.230735778808594 (difference: 5.2813%)
Value mismatch at [0].cell_id: 65414.0 vs 66942.0 (difference: 2.28257%)
Value mismatch at [0].dr: 170.65232849121094 vs 170.90232849121094 (difference: 0.146282%)
Value mismatch at [0].eta: -0.03586232289671898 vs -0.16087640821933746 (difference: 77.7082%)
Value mismatch at [0].phi: 0.16119804978370667 vs -0.0351688526570797 (difference: 121.817%)
Value mismatch at [0].x: 1734.4632568359375 vs 1756.896728515625 (difference: 1.27688%)
Value mismatch at [0].y: 282.03924560546875 vs -61.81352615356445 (difference: 121.917%)
Value mismatch at [0].z: -63.032386779785156 vs -284.0396423339844 (difference: 77.8086%)

Note: for some reason the refs are not reproducible when running the test executables directly instead of invoking them via ctest. Unclear why that is as the seed does not change. It seems that the differences are already at the transportation state.

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 62.41611% with 56 lines in your changes missing coverage. Please review.

Project coverage is 31.68%. Comparing base (b069644) to head (8e8543e).

Files Patch % Lines
test/TestHelpers/JsonComparer.h 55.31% 42 Missing :warning:
test/g4app/src/FastSimModel.cc 33.33% 12 Missing :warning:
test/source/AtlasSimTests.cxx 83.33% 1 Missing :warning:
test/source/AtlasTransportTests.cxx 83.33% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #27 +/- ## ========================================== + Coverage 31.39% 31.68% +0.28% ========================================== Files 217 218 +1 Lines 11596 11709 +113 Branches 1649 1669 +20 ========================================== + Hits 3641 3710 +69 - Misses 7955 7999 +44 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.