Revamp Current Regression Testing Framework

[mandli]

This issue is intended as a place to document and plan for changes to the regression testing framework used in the Fortran based codes in Clawpack. This includes classic, amrclaw , and geoclaw. These changes would be implemented along side of the switch to PyTest as our testing framework.

A couple of items of note:

• The code that implements testing is based here in clawutil and has extensions in clawpack/amrclaw and clawpack/geoclaw.
• Existing issues:

  • 87

  • 101

  • clawpack/geoclaw#377

[mandli]

Failure Handling

Handling test failure is one of the big items on the list of things to address. Currently everything that would have been placed in the _output directory should be copied into the current working directory by default. This tends to fail if a test is switching working directories during the test. Other suggestions for behavior is welcome.

[mandli]

Placement of tests

Currently all tests are in the $CLAW/package/tests directory. Another way of doing this is to put them in the corresponding example. There is not currently a 1-to-1 correspondence between all the tests and examples but we could do that. Testing data could also reside in these directories as well. One reason we did not do this previously was that nosetests needed some treatment to find tests without adding __init__ everywhere. PyTest does not have this issue.