catalyst-cooperative / pudl-archiver

A tool for capuring snapshots of public data sources and archiving them on Zenodo for programmatic use.
MIT License
4 stars 2 forks source link

Test individual archivers #45

Open zschira opened 1 year ago

zschira commented 1 year ago

Background

Right now the individual archivers (inheriting from AbstractDatasetArchiver) are largely untested. There's been discussion in #43 about the appropriate way to test these archivers. In the long run, we will want to have new archives tested through nightly builds to fully verify the entire workflow. This, however will likely be future work once we are actually generating new archives regularly, and have decided how we want to integrate those archives into PUDL. In the immediate future, there are likely helpful tests that we can create at the unit/integration level to perform some basic sanity checking.

Scope

This issue will only track the development of unit/integration tests, and will leave full integration into the nightly builds process for future work. Things we can check for at this level include:

zaneselvans commented 1 year ago

@zschira Are there unit or integration tests that you imagine falling under this issue that wouldn't be covered by what's described in #70?