UCL-ARC / python-tooling

Python package template for new research software projects
http://github-pages.arc.ucl.ac.uk/python-tooling/
MIT License
43 stars 2 forks source link

Add regression test for generated package data #432

Closed dstansby closed 4 weeks ago

dstansby commented 4 months ago

This adds a regression test for generated pacakge data. The test folder now has a data folder inside it which contains a reference copy of the generated package. This is then compared at test time to a newly generated package. Any differences are shown after the failing test with a diff that looks like:

E           RuntimeError: Non-zero diff between generated files and expected files.
E           Generated files can be found in /Users/dstansby/software/python-tooling/pytest_output/test_package_generation0/cookiecutter-test.
E
E           --- /Users/dstansby/software/python-tooling/pytest_output/test_package_generation0/cookiecutter-test/README.md
E           +++ /Users/dstansby/software/python-tooling/tests/data/test_package_generation/README.md
E           @@ -119,7 +119,7 @@
E            tox -e docs
E            ```
E            
E           -from the root of the repository. The built docs will be written to
E           +from the root of the repository. The built documentation will be written to
E            `site`.
E            
E            Alternatively to build and preview the documentation locally, in a Python

tests/test_package_generation.py:79: RuntimeError

To update the test files with an intentional change, when the test fails the test data is updated with the changes that can then be easily comitted (if desired).

Fixes https://github.com/UCL-ARC/python-tooling/issues/329

dstansby commented 1 month ago

Are we sure we want this? Every time there's a new change, this test will have to be updated.

See https://github.com/UCL-ARC/python-tooling/issues/329 for context and motivation.

Definitely appreciate that this adds a bit more work when changing anything in the template, but I think it's worth it to catch potential errors such as in https://github.com/UCL-ARC/python-tooling/pull/309 which slipped through the PR author and reviewer.

I've tried to make the loop here as simple as possible:

  1. Run tests
  2. If this test fails, it modifies the test data in place
  3. Review if the test data is modified as expected. If it is, commit and push the changes.
paddyroddy commented 1 month ago

See #329 for context and motivation.

Definitely appreciate that this adds a bit more work when changing anything in the template, but I think it's worth it to catch potential errors such as in #309 which slipped through the PR author and reviewer.

Consider me won over, I'd forgotten about #309

dstansby commented 1 month ago

Sigh, putting this back as draft until I can fix the test

paddyroddy commented 4 weeks ago

The tests are failing again @dstansby. main had updated.