Improve catalog-level CI tests

catalyst-cooperative / pudl-catalog

An Intake catalog for distributing open energy system data liberated by Catalyst Cooperative.

https://catalyst.coop/pudl/

MIT License

9 stars 2 forks source link

Improve catalog-level CI tests #29

Closed zaneselvans closed 2 years ago

zaneselvans commented 2 years ago

The CarbonPlan data catalog repo provides some examples of tests that could apply to our catalog as well.

These include:

[x] Ensure all catalog entries have a set of required metadata attributes.
[x] Verify that the expected files are actually available in cloud storage
[x] Use to_dask() to avoid the need to download data while verifying that EPA CEMS dataframes have the right columns.

zaneselvans commented 2 years ago

Looking at these tests (and generally mucking with the Intake catalogs) it feels like the catalogs should be encapsulated in a @dataclass or Pydantic model. They're kind of a chaotic freeform mess right now -- not just ours, but Intake catalogs in general.