Closed ofuhrer closed 4 years ago
This PR is done and ready to review. Could @lxavier or @kosterried pick this up. Should not be a big thing, since it basically adds regression testing. The edits in the testsuite Python code are minimal.
@ofuhrer seems to work, but I get an error on kesch when doing make test, see below:
test_regression.py::test_full_testlist_from_xml PASSED
=================================================== 39 passed in 47.93s ===================================================
Exception ignored in: <bound method TemporaryDataDirectory.__del__ of <test_regression.TemporaryDataDirectory object at 0x2aaab4bd2240>>
Traceback (most recent call last):
File "/scratch-shared/meteoswiss/scratch/lapixa/testsuite/tests/regression/test_regression.py", line 55, in __del__
AttributeError: 'NoneType' object has no attribute 'rmtree'
@lxavier Sorry, obviously my strategy for cleaning away the data
directory did not work. Should work now. Removed that cleanup section. Will have to use pytest
fixtures, but don't know how they work yet.
@guydemorsier there is a PR from Oli but it fails with int2lm here, do you see what could be wrong : http://jenkins-mch.cscs.ch/view/tools/job/testsuite/55/real_type=double,slave=kesch,target=gpu/console
the issue is in the was the testsuite is integrated in int2lm, I will fix it in int2lm. I manually run with int2lm test ok cosmo test ok (except for output_tolerance), but this was in fact broken, I will open an issue
@kosterried and @guydemorsier I merge this, but I will need to fix some of the cosmo test see https://github.com/C2SM-RCM/testsuite/issues/37 and change one configuraiton in int2lm
@ofuhrer congratulation first external contribution :)
This PR adds regression tests to the test suite repository. The regression tests use a surrogate model
model.py
which emulates the behavior of the COSMO model and is used for rapid regression testing. Some fixes had to be applied to the testsuite Python scripts in order to make all tests pass. Would be great if we could add this to the testing plan (which I don't think I have access to).Note, to run these tests on Piz Kesch, you can simply execute
make test
in the top-level testuite directory (see image below for results).