catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
471 stars 108 forks source link

Simplify the organization of our test suites #942

Closed zaneselvans closed 3 years ago

zaneselvans commented 3 years ago

While updating the development documentation (#940) related to running our tests, it was obvious that our current testing setup is more complex than it needs to be, and it seems to have discouraged people from running the tests, since almost nobody can remember how to do it. There have also been several cases in which someone has gone to the trouble of writing new tests, but they never get run automatically because they aren't put somewhere that our CI setup will run them.

Rather than documenting a complicated system, I'd like to simplify it a bit, and then document a simple system. Changes I'd like to make include:

See also issue #941 about separating the data validation process from the Tox/pytest setup, since it requires a complete database, and is really a different kind of thing altogether.

zaneselvans commented 3 years ago

This all seems to be working well now! Minor updates to the new docs need to happen to reflect the current setup, which is way simpler.

There are some tests which, rarely, do need to be run against all years of data -- and we didn't really have a systematic way to ensure that was happening in the past. I will create a settings file for all of the years which is part of the test settings, and a tox test environment that uses those settings to check out everything together.