search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
456 stars 106 forks source link

Debug multi-year import of EIA923 data into PUDL DB #62

Closed zaneselvans closed 7 years ago

zaneselvans commented 7 years ago

Once we have the EIA923/FERC1 id mapping done for our full dataset, we will need to try and pull in multiple years worth of EIA923 data simultaneously. This will almost certainly result in some unforeseen errors, which we'll need to clean up.

Initial testing can happen with the already ingestible 2014-2016 data, all of which has the same format in the EIA923 spreadsheets.

zaneselvans commented 7 years ago

As of 2017-03-28 we've got 2011 to 2015 importing. 2009-2010 need the infrastructure to populate without the plant_frame table, and 2016 still has some IDs which aren't mapped, bringing up some foreign key constraint errors. (e.g. plant_id=8809 "Bent Mountain")