search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
468 stars 107 forks source link

Integrate minor cleaning fixes from spot-checks of XBRL explosion #2599

Closed e-belfer closed 1 year ago

e-belfer commented 1 year ago

This issue compiles a few small fixes identified in discussions in #2016.

- [x] 1.) `utility_plant_summary_ferc1`: Where `depreciation_utility_plant_in_service` is negative and exactly opposite the sign of its calculation, flip it. See details here: https://github.com/catalyst-cooperative/pudl/issues/2016#issuecomment-1553191877
- [x] 2.) `plant_in_service_ferc1`: Fix negatives in the `electric_plant_sold` column. See details here: https://github.com/catalyst-cooperative/pudl/issues/2016#issuecomment-1559672614
- [x] 3.) `electric_operating_revenues_ferc1`: figure out why `small_or_commercial`and `large_or_industrial` values are getting dropped from the cleaned table @e-belfer
e-belfer commented 1 year ago

With electric_operating_revenues, it looks like the column names are getting split by the wide_to_tidy function, with revenue attached to small_or_commercial and the other two fields attached to small_or_commercial_sales. image

I fixed this in the FERC transform parameters.

e-belfer commented 1 year ago

Branch off of check_calcs_in_transformers

aesharpe commented 1 year ago

@e-belfer and I decided that Task 1 makes sense to do as a one-off spot fix so that we can run the reconcile_table_calculations functions after the records are fixed

aesharpe commented 1 year ago

Values pertaining to Task 1 spot fix:

report_year utility_id_ferc1 ending_balance calculated_amount abs_diff
52661 2002 170 -8732 8732 17464
75784 2006 211 2.0784e+09 -2.0784e+09 4.1568e+09
75797 2006 211 2.0784e+09 -2.0784e+09 4.1568e+09
85062 2007 393 3.67392e+08 -3.67392e+08 7.34784e+08
85073 2007 393 3.67392e+08 -3.67392e+08 7.34784e+08
93853 2009 211 2.46878e+09 -2.46878e+09 4.93756e+09
93866 2009 211 2.46878e+09 -2.46878e+09 4.93756e+09
99498 2010 211 2.55997e+09 -2.55997e+09 5.11995e+09
99511 2010 211 2.55997e+09 -2.55997e+09 5.11995e+09
106272 2011 211 2.68028e+09 -2.68028e+09 5.36056e+09
106285 2011 211 2.68028e+09 -2.68028e+09 5.36056e+09
112580 2012 156 3.36681e+08 -3.36681e+08 6.73362e+08
112591 2012 156 3.36681e+08 -3.36681e+08 6.73362e+08
112791 2012 211 2.62733e+09 -2.62733e+09 5.25466e+09
112804 2012 211 2.62733e+09 -2.62733e+09 5.25466e+09
118471 2013 170 1.28336e+10 -1.28336e+10 2.56671e+10
118486 2013 170 1.28336e+10 -1.28336e+10 2.56671e+10
119717 2013 211 2.56201e+09 -2.56201e+09 5.12401e+09
119730 2013 211 2.56201e+09 -2.56201e+09 5.12401e+09
125655 2014 211 2.61497e+09 -2.61497e+09 5.22994e+09
125667 2014 211 2.61497e+09 -2.61497e+09 5.22994e+09
131996 2015 211 2.81513e+09 -2.81513e+09 5.63025e+09
132008 2015 211 2.81513e+09 -2.81513e+09 5.63025e+09
137551 2016 211 4.26923e+09 -4.26923e+09 8.53845e+09
137563 2016 211 4.26923e+09 -4.26923e+09 8.53845e+09
143526 2017 211 1.17593e+09 -1.17593e+09 2.35186e+09
143538 2017 211 1.17593e+09 -1.17593e+09 2.35186e+09
149711 2018 211 1.17073e+09 -1.17073e+09 2.34146e+09
149723 2018 211 1.17073e+09 -1.17073e+09 2.34146e+09
aesharpe commented 1 year ago

@e-belfer

Image

This is an example of on the of spot-fix records. Can you explain how depreciation_utility_plant_in_service is the issue? I can't see how it's involved in the calculation of accumulated_provision... Just want to clarify that I'm flipping the signs of the depreciation_utility_plant_in_service records, not the accumulated_provision... records.