search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
456 stars 106 forks source link

Integrate EIA 861 into Entity Harvesting Process #614

Open zaneselvans opened 4 years ago

zaneselvans commented 4 years ago

The EIA 861 brings a new kind of entity the Balancing Authority. It also brings a huge number of new tables related to Utilities, so it's important to integrate it into the post-Transform harvesting process. However, some of these tables also have structures unlike any in EIA 860 and EIA 923 (see Issue #613) and so it's not just a matter of adding new tables and entities into the existing harvesting process framework. We may need to change the entity harvesting to know specifically which columns it is supposed to harvest from each table.

jrea-rmi commented 1 year ago

I wish EIA861 entities were included in pudl.sqlite.utilities_eia and had utility_id_pudl connected to them. Is that likely to happen in the future, before or alongside full integration of EIA861 into the pudl ETL?

@aesharpe

zaneselvans commented 1 year ago

This is certainly the intention, alongside the full integration of eia861.