catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
458 stars 105 forks source link

Create Epic for parallelization efforts #1171

Closed bendnorman closed 2 years ago

bendnorman commented 2 years ago

Fully scope the Prefect parallelization Sloan project. Goals and a description can be found in the grant and our project planning document.

This issue should be completed by 8/7/21.

bendnorman commented 2 years ago

It seems like the main goal for this project is to cut down on the time it takes to run the full ETL, integration, and validation tests.

There are still some questions we need to answer before we can scope this epic:

bendnorman commented 2 years ago

Mostly covered by #840 and #899.