Right now CI only tests whether the EPA CEMS parquet data is working, but we've included the pudl.sqlite and ferc1.sqlite databases in the manifest as well, so they also need to be tested.
Messing around with the v2022.11.30 data I found that there were a variety of issues with some tables in the PUDL DB, and none of the data in the ferc1 DB was accessible so... there's work to be done here. I've implemented just the most basic tests as an example of some of these problems in #75 and marked the ones that aren't working with xfail.
Some potential tests to implement
[ ] Check that urlpath to pudl.sqlite looks reasonable
[ ] Check that urlpath to ferc1.sqlite looks reasonable
[ ] Check that a few expected tables exist in pudl.sqlite
[ ] Check that a few expected tables exist in ferc1.sqlite
[ ] Check that the number of tables in pudl.sqlite is at least some minimum.
[ ] Check that the number of tables in ferc1.sqlite is at least some minimum.
[ ] Read a table from pudl.sqlite and check that it has a reasonable shape and contents.
[ ] Read a table from ferc1.sqlite and check that it has a reasonable shape and contents.
Right now CI only tests whether the EPA CEMS parquet data is working, but we've included the
pudl.sqliteandferc1.sqlitedatabases in the manifest as well, so they also need to be tested.Messing around with the
v2022.11.30data I found that there were a variety of issues with some tables in the PUDL DB, and none of the data in the ferc1 DB was accessible so... there's work to be done here. I've implemented just the most basic tests as an example of some of these problems in #75 and marked the ones that aren't working withxfail.Some potential tests to implement
urlpathtopudl.sqlitelooks reasonableurlpathtoferc1.sqlitelooks reasonablepudl.sqliteferc1.sqlitepudl.sqliteis at least some minimum.ferc1.sqliteis at least some minimum.pudl.sqliteand check that it has a reasonable shape and contents.ferc1.sqliteand check that it has a reasonable shape and contents.