Use Docker container to generate PUDL data release for Zenodo

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

https://catalyst.coop/pudl

MIT License

471 stars 108 forks source link

Use Docker container to generate PUDL data release for Zenodo #799

Closed zaneselvans closed 3 years ago

zaneselvans commented 3 years ago

Create one or more Docker containers to coordinate the full data release generation process, including:

Software environment setup.
Data acquisition (or point it at at a gs:// bucket)
Generate FERC 1 DB
Run the full ETL to generate data packages.
Populate the SQLite DB
Generate Parquet files for CEMS
Run full data validation.
Maybe upload to draft Zenodo archive?

zaneselvans commented 3 years ago

We decided to use the by-hand method for ETL for 0.4.0. Will use cloud ETL for 0.5.0