Replace pudl.extract.excel.Extractor with the new Datastore

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

https://catalyst.coop/pudl

MIT License

456 stars 105 forks source link

Replace pudl.extract.excel.Extractor with the new Datastore #580

Closed ptvirgo closed 4 years ago

ptvirgo commented 4 years ago

pudl/extract/excel Extractor is used in the eia860 and eia923 to get file paths. Replace this with the new datastore. Prove it works by re-running the ETL process with the datastore.

src/pudl/package_data/meta/xlsx_maps contains csv mappings that tell the extract process how to find necessary files.

ptvirgo commented 4 years ago

Todo:

[ ] /src/pudl/extract/excel.py - _load_excel_file() contents should be replaced to use the datastore
[ ] /src/pudl/extract/excel.py - eliminate verifiy_years() in favor of the datastore self validation
[ ] the page to file glob structure should be replaced with year/tab -> excel sheet csv mapping, used by the _load_excel to get the right contents.