search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
465 stars 107 forks source link

Double check ferc1 row maps #1287

Closed bendnorman closed 2 years ago

bendnorman commented 2 years ago

Update the FERC 1 row maps for individual tables, based on the row_num associated with each named field in the f1_row_lit_tbl -- if the row_chg_yr column hasn’t changed, you should be able to copy the previous year’s column over to the new year.

zaneselvans commented 2 years ago

I bet could write a little script / function that did this update so long as nothing had changed, and flagged it for human attention when something has changed.

bendnorman commented 2 years ago

I think you already did! Just found this notebook that compares consecutive years in the f1_row_lit_tbl table. I reran it and it looks like nothing changed between 2019 and 2020.

Would you like to include the logic in the notebook in the ferc1 extract step? I think keeping it in the notebook is sufficient given we only do this once a year.

zaneselvans commented 2 years ago

Ah, right I vaguely remember that notebook. Probably not worth changing anything, since they're switching to XBRL next year anyway. So we get to build a whole brand new ETL process!

bendnorman commented 2 years ago

The ferc1 rows were checked and are in the 2020 branch.