Fixed issue where regression tests are not called properly
Update the test artefacts to include the actual data rather than pointers to git lfs
Removed two tests whose behaviour is undesirable (outputs change from run to run)
Updating the datafile means there's a huge set of diffs to review. The key file changes are in:
tests/conftest.py
regression_tests/test_ustar_cp/test_ustar_cp.py
The following comment is a report on the findings from investigating the failing test issue:
I've done some experimenting and I think I have narrowed down the reasons for the regression test discrepancy. I've been running the failing cases in Matlab 2024 to reproduce the results.
The core issue seems to be the stochastic sampling nature of the code. For certain input files, the output seems to vary on each run and this is obviously problematic for a simple comparison based test framework.
The two failing tests are US-Syv and US-ARM. Looking at my previous analysis of the available cases, US-Syv can be replaced by two other sets of data (to give coverage of the displayed expected report messages), however unfortunately US-ARM covers a unique case not covered by the remaining cases we have access to.
Proposal:
Remove both these cases from the list of regression tests
Replace US-Syv with US-Elm or US-Nr1 for one type of error it captures. Replace with US-Myb for the other. This is dependent on these also being stable cases (to be investigated)
This PR closes #46
Three key steps have been taken:
Updating the datafile means there's a huge set of diffs to review. The key file changes are in:
The following comment is a report on the findings from investigating the failing test issue:
The proposals will be managed in a separate issue