Transform `f1_purchased_power` xbrl + dbf

[aesharpe]

conversions from old transform to new

• [x] _clean_cols ->
  • [x] Ferc1AbstractTableTransformer.assign_record_id deployed in Ferc1AbstractTableTransformer.process_dbf and Ferc1AbstractTableTransformer.process_xbrl
  • [x] AbstractTableTransformer.enforce_schema deployed in Ferc1AbstractTableTransformer.transform_end
• [x] rename -> rename_cols
• [x] fillna w/ 0's + drop data cols == 0 -> `drop_invalid_rows
• [x] drop_duplicates? This was somehow not required. We are checking the PK uniqueness in assign_record_id.
• [x] convert to numeric -> use strip_non_numeric_values maybe?
• [x] run the encoder
  • I added this into transform_main

added

• [x] there are 7 dbf and 2 xbrl duplicates in the record_id col... need to go investigate that. maybe make a record_id nullable: bool param?
  • I added a drop_duplicates into transform_main

missing plant/util ids

• [-] run pytest test/integration/glue_test.py --live-dbs --save-unmapped-ids (not needed!/tests all pass)
• [-] add any ids into appropriate id maps (not needed!)

settings files

• [x] update fast w/ table name
• [x] update full w/ table name

[cmgosnell]

current task:

• determine what to do with the to_numeric data cols.

Here is my little check-er: Setup

import pudl
from pudl.transform.ferc1 import *

ferc1_settings = pudl.settings.Ferc1Settings(tables=["purchased_power_ferc1"])
# Extract FERC form 1
ferc1_dbf_raw_dfs = pudl.extract.ferc1.extract_dbf(
    ferc1_settings=ferc1_settings, pudl_settings=pudl_settings
)
# Extract FERC form 1 XBRL data
ferc1_xbrl_raw_dfs = pudl.extract.ferc1.extract_xbrl(
    ferc1_settings=ferc1_settings, pudl_settings=pudl_settings
)
pp = PurchasedPowerTableTransformer(cache_dfs=True, clear_cached_dfs=False)
start = pp.transform_start(
    raw_dbf=ferc1_dbf_raw_dfs["purchased_power_ferc1"],
    raw_xbrl_instant=ferc1_xbrl_raw_dfs["purchased_power_ferc1"]["instant"],
    raw_xbrl_duration=ferc1_xbrl_raw_dfs["purchased_power_ferc1"]["duration"]
)

Bad data finder:

num_cols = ["billing_demand_mw", "non_coincident_peak_demand_mw", "coincident_peak_demand_mw"]
col = num_cols[0]
out_col =pd.to_numeric(start[col], errors="coerce")
start.loc[out_col[out_col.isnull()].index][col].unique()

[zaneselvans]

Closed by PR #2011