Closed zaneselvans closed 3 years ago
This all seems to be working well now! Minor updates to the new docs need to happen to reflect the current setup, which is way simpler.
There are some tests which, rarely, do need to be run against all years of data -- and we didn't really have a systematic way to ensure that was happening in the past. I will create a settings file for all of the years which is part of the test settings, and a tox test environment that uses those settings to check out everything together.
While updating the development documentation (#940) related to running our tests, it was obvious that our current testing setup is more complex than it needs to be, and it seems to have discouraged people from running the tests, since almost nobody can remember how to do it. There have also been several cases in which someone has gone to the trouble of writing new tests, but they never get run automatically because they aren't put somewhere that our CI setup will run them.
Rather than documenting a complicated system, I'd like to simplify it a bit, and then document a simple system. Changes I'd like to make include:
tox -av
--tmp-data
flag is set, and then download fresh data to a temporary directory.--live_pudl_db
is still behaving as expected by running the output tests or something.See also issue #941 about separating the data validation process from the Tox/pytest setup, since it requires a complete database, and is really a different kind of thing altogether.