search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
456 stars 105 forks source link

Allow EIA860m data from 1 or 2 years following current EIA860 data #1780

Open zaneselvans opened 1 year ago

zaneselvans commented 1 year ago

Right now we have hard-coded an assumption that the EIA-860m data will always include data from the 1 year following the data which is available from the EIA-860. However there are several months each year (approximately March-September) when the previous year's complete EIA-860 data has not yet been released, and there's data from the current year, meaning that the EIA-860m contains data from 2 calendar years following the most recent EIA-860 data.

Accommodating this will require changing the Eia860Settings validation to allow for 1 or 2 years of EIA-860m data, and also updating some of the ETL process to intelligently select the right EIA-860m data to supplement the most recent EIA-860 data.

cmgosnell commented 1 year ago

just fyi @zaneselvans , I know you just updated the archive but there were two months of m's missing so I regenerated a new archive (w/ a DOI of [3:37 PM] Christina Gosnell alex! hello! here is the [new eia80m archive](https://zenodo.org/record/6929086) with a DOI of10.5281/zenodo.6929086`.