Consolidate dataset level metadata across modules / packages

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

https://catalyst.coop/pudl

MIT License

456 stars 106 forks source link

Consolidate dataset level metadata across modules / packages #767

Closed zaneselvans closed 1 year ago

zaneselvans commented 3 years ago

We are currently storing metadata related to datasets (e.g. eia923, ferc714) in 3 different places:

constants.py in the main PUDL repo
both the frictionless and the zs modules within the pudl-zenodo-storage repository.

This is inefficient and will result in the metadata getting out of sync. Somehow we need to consolidate this and store it in one place that's better accessible to all the packages. See also #52 and #523.

zaneselvans commented 2 years ago

I think the solution here is probably to make the other repositories / packages depend on the main pudl package / repo so that we can directly access the metadata structures and re-use them in those other contexts.

zaneselvans commented 1 year ago

Lots of this consolidation has been done, and the metadata mostly lives under pudl.metadata now.