catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
481 stars 110 forks source link

Revise FERC-714 Table Structures #3198

Open zaneselvans opened 10 months ago

zaneselvans commented 10 months ago

Some suggested changes to our old FERC-714 table structures, based on investigations for GridLab:

- [ ] Create an output table that is just the FERC-714 respondent IDs along with all of the other IDs that they get associated with for use in merging with other tables. Some version of this probably already exists in the intermediate tables.
- [ ] Add a categorical `state` column to the estimated demand table in addition to the `state_id_fips` since state abbreviations are much more legible.
- [ ] Clarify name & meaning of `respondent_type` field. [see this comment](https://github.com/catalyst-cooperative/pudl/pull/3283#discussion_r1467024452)
mikejrob commented 10 months ago

I'm not sure if this is a good place to make suggestions, but any chance you can create a table of system lambdas and aggregate in a manner consistent with demand aggregations (demand-weighted averages, ideally)?

zaneselvans commented 10 months ago

Hey @mikejrob this issue is about cleaning up a couple of existing tables, but if you're interested in the system lambda table that's reported in FERC-714, would you be interested in creating a new issue describing your needs? We haven't worked with data particular data yet, though we do extract the raw info from the 2006-2020 CSVs, and have the 2021-2022 data in the SQLite DB we create from the newer XBRL data.

If you'd be at all interested in helping get a cleaned up version of the system lambda table working we'd also be very happy to work with you to get a PR merged in. You might take a look at the current pudl.transform.ferc714 module to see what the code for the existing hourly demand table looks like, and adapt that.

mikejrob commented 10 months ago

The structure and format of the system lambda data are very similar to the demand data, so it might be easy to adapt what you've already done. I'll see if I can find someone to work on this, and if I can, I'll open a new issue.

zaneselvans commented 10 months ago

That would great!