search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
478 stars 110 forks source link

Scope 2023 FERC 1 data integration #3544

Closed aesharpe closed 1 month ago

aesharpe commented 7 months ago

Not all of the FERC data is in, but there's enough to test the extractor and see if there are any major changes to the data format that would beget a refactor

Keep this exploration cursory. Don't try and fix any major errors, just SCOPE them. Check-in after attempt to extract new year of data and then proceed to transformation if fixes are strait forward / RMI gives explicit permission to address them.

Success criteria

### Tasks
- [ ] https://github.com/catalyst-cooperative/pudl-archiver/issues/318
- [ ] https://github.com/catalyst-cooperative/pudl/issues/3575
- [ ] https://github.com/catalyst-cooperative/pudl-archiver/issues/349
- [ ] https://github.com/catalyst-cooperative/pudl/issues/3650
- [ ] https://github.com/catalyst-cooperative/pudl/issues/3698
jdangerx commented 6 months ago

FYI @aesharpe

@zschira and I just spent some time looking into these issues - broadly we see three areas of changes: archival, extraction, and downstream transforms.

archival

These changes enable the extractor changes below, which then will fix the issues found in #3575.

We expect this to take roughly 10h.

extraction

These changes are a little more entangled with the rest of the system - we expect them to take about 15h. Combined with the archiver changes above, this should address all the issues found in #3575 .

transformation

This is the first time we are encountering a new schema for the FERC1 data - which means the transformations will likely need various fixes to get them running again, in addition to the normal utility mapping, validation tests/data investigation, etc.

Last year, according to Harvest, it took us 240 hours to handle all of the 2022 data - probably about half of that dedicated to FERC (looks like I put 90h or so into it).

It was easier, since we didn't have a new schema to deal with, but also harder, in that it was the first time dealing with multiple years of XBRL data. I would be comfortable budgeting for 120 hours of time to integrate the new FERC data, though I know that sounds like a huge number and I really hope it takes less time.

Overall that means 145 hours of work.

I think we should aim to complete this in June or July - May is already quite booked. As of right now, we see 221 filings for 2023 in the latest archive, which is 85% of the number of filings for 2022. So I do also expect some more filings to trickle in over the next month or so.

cmgosnell commented 6 months ago

hey @jdangerx when i was trying to compile the general pudl maintenance budget (an internal doc for now!) i looked into the 2022 update and the ferc hours a bit. I pulled out all of the time entries that had either "ferc" or "xbrl" in there and it was just under 60 hours. It included the saving of a bunch of recrds we had previously been dropping during the archiving... which was hopefully a one-off but actually seems pretty representative for FERC weirdness. So I think this estimate is high!

jdangerx commented 6 months ago

I went through and looked at each of the time entries under software development or data wrangling for the FERC 2022 project - there was a bunch of stuff that didn't explicitly have the words FERC or XBRL that was related to one of those things. Fortunately most of the work was done by me so I understand the cryptic notes I left 😅

It all sums to:

Dealing with utility mapping, FERC transforms, investigating why validation tests were failing and whether that's a real problem: 64h

Dealing with the duplicate data issue from having multiple filings from different years representing the same data: 19h

Additionally Austen looks like she spent about 3h on FERC stuff. So that all comes out to 86h.

Intuitively, a new data schema seems like a big source of uncertainty, I think we should have some buffer above "what we did last year." 120 does still sound high but not outrageous.

To get a sense of how much the schemas really changed, I could probably pull like an hour to:

That would probably be a good investment of our scoping time.

jdangerx commented 5 months ago

There are a total of 10 tables which appear to have some sort of schema change between the 2022-01-01, 2023-04-01, and 2023-11-01 FERC 1 taxonomies.

None of these tables appear to be actively used in our current FERC1 ETL - I pulled the list of used tables from pudl.extract.ferc1.TABLE_NAME_MAP_FERC1. That means I don't expect to have to make big changes to the transform code to handle the new year's data.

We will, however, still have to investigate validation errors for the new data, do the utility/plant mapping, etc. Which will probably take 30-40h depending on how thoroughly we want to investigate the validation errors.

Here's the rundown of the schema changes:

Changes from adding energy storage operations data for small plants

Removing electricity sales by rate schedules in C&I in 2023-04

zaneselvans commented 3 months ago

Looks like this issue can be closed?

Maybe we want to ask @arengel or @jrea-rmi whether the brand new energy storage tables are of interest.

arengel commented 3 months ago

@zaneselvans, this data looks to be useful but it's not critical for any near-term work I am aware of. So I think we are likely to want to get the storage tables incorporated in the future, though not now. But I do have 2 questions related to this data specifically and decisions like this generally:

  1. On the energy storage tables, do they have capacity in MW and duration / capacity in MWh? If not does another table have that data in a way that doesn't require a a matching exercise to pull in?

  2. Do you see incorporating this table as fitting under PUDL maintenance / updates or would this be additional, at least as our arrangements stands now? I assume that a future PUDL patron setup would define how decisions about and payment for this sort of work would... work.

zaneselvans commented 3 months ago

@arengel taking a quick look at the tables in Datasette:

I note that:

If you want those attributes, I suspect you should look at the EIA generator + energy storage data instead:

Currently the generic generator fields like capacity_mw and (somewhat oddly) energy_storage_capacity_mwh lives in the generic generator tables like core_eia860__scd_generators, and only the storage, wind, and solar specific generator fields fall into the specific storage, wind, and solar generator tables so you would need to join them together to combine energy_storage_capacity_mwh and max_charge_rate_mw and max_discharge_rate_mw to be able to calculate the duration of storage available (presumably energy_storage_capacity_mwh / max_discharge_rate_mw?)

I could definitely see creating a dedicated energy storage output table so that this kind of information is available off the shelf without introducing a huge number of additional columns into the big generator output table that only pertain to storage, wind, or solar.

I think that adding these tables would probably fall into the "stretch goals" part of what we're envisioning. It's an incremental improvement beyond the minimum viable PUDL maintenance, but should be low-risk and of general utility. So, if there are enough supporters beyond the baseline requirements to keep the PUDL data flowing, adding these tables would be on the menu of improvements those beyond-baseline funds could be allocated to.