This introduces the necessary functionality to write regression
tests, and implements regression tests for the NVT ensemble checks.
The regression tests use the pytest-regressions package to check
run outputs.
The regression test functionality includes a test system database
storing input files for the runs of physical validation. This
includes a System class, whose objects represent a test system
with potentially several simulation results that can be analyzed.
The database.py file includes a simple function that returns the
different test systems.
To run the NVT regression tests, energy results of a simulation of
900 water molecules was added to the database. Other tests wil be
able to use the same system or add new ones.
The regression tests run the ensemble check for the NVT case, both
using the flat file parser and setting up the simulation data object
using numpy arrays. The regression test fixtures allow checking of
the returned values, all printed output, and the plotted image.
As is the nature of regression tests, this will mean that some
changes will and should make these tests fail. If such a change
is made, the reference data can easily be regenerated using
$ pytest test_ensemble_regression.py --force-regen
These regression tests do, however, allow us to monitor exactly
what changes happened, and whether they were intended.
Refs #59
Todos
Notable points that this PR has either accomplished or will accomplish.
[x] Setup a system database for end-to-end regression tests
[x] Introduce end-to-end regression test for NVT ensemble check
Status
This PR is currently aimed at https://github.com/shirtsgroup/physical_validation/tree/ptmerz-master-after-85-86-88, which is the master branch plus the commits in PRs #85, #86, and #88. These three PRs are prerequisites for the current PR, as they add some options that increase reproducibility (#85, #86), or allow us to compare images programmatically (#88).
Description
This introduces the necessary functionality to write regression tests, and implements regression tests for the NVT ensemble checks. The regression tests use the pytest-regressions package to check run outputs.
The regression test functionality includes a test system database storing input files for the runs of physical validation. This includes a
Systemclass, whose objects represent a test system with potentially several simulation results that can be analyzed. Thedatabase.pyfile includes a simple function that returns the different test systems.To run the NVT regression tests, energy results of a simulation of 900 water molecules was added to the database. Other tests wil be able to use the same system or add new ones.
The regression tests run the ensemble check for the NVT case, both using the flat file parser and setting up the simulation data object using numpy arrays. The regression test fixtures allow checking of the returned values, all printed output, and the plotted image.
As is the nature of regression tests, this will mean that some changes will and should make these tests fail. If such a change is made, the reference data can easily be regenerated using $ pytest test_ensemble_regression.py --force-regen These regression tests do, however, allow us to monitor exactly what changes happened, and whether they were intended.
Refs #59
Todos
Notable points that this PR has either accomplished or will accomplish.
Status
This PR is currently aimed at https://github.com/shirtsgroup/physical_validation/tree/ptmerz-master-after-85-86-88, which is the master branch plus the commits in PRs #85, #86, and #88. These three PRs are prerequisites for the current PR, as they add some options that increase reproducibility (#85, #86), or allow us to compare images programmatically (#88).