Integrate reshaped XBRL metadata into existing FERC1 table transforms

[zaneselvans]

@zschira has updated the XBRL taxonomy extraction to group the metadata by table, which will simplify it's usage in table transformers. This new structure appears in v0.7.0 of catalyst-cooperative/ferc-xbrl-extractor#38

• Currently every table transformer gets a copy of the whole XBRL metadata structure.
• With per-table lookup, we'll only give each transformer the metadata that pertains to it.
  • Metadata could be passed in as an argument like the dataframes.
  • Could also be stored as a class attribute (as it is now).
• Should the extract_xbrl_metadata() function also take a Ferc1Settings object, and return a dictionary of dataframes using the same structure as the XBRL instant/duration table extraction?
• Like in the XBRL data, there will be instant & duration metadata tables.
• Should we keep instant & duration separate, or should we merge them together and use them for both tables?
• If there are rows with duplicate names in the instant/duration tables, they should also be identical in the other columns.
• Combining the instant & duration metadata would be concatenating additional rows, not columns.
• Where does the metadata come into things now in the transform?
• Simplest thing to do immediately would be to (as before) put a copy of the entire metadata structure into the transformer class, and grab just the appropriate table information when it's needed.
• Currently we combine the instant & duration column names and store a single dataframe with metadata for all of them.

Plan

• Pass Ferc1Settings to the XBRL metadata extract function. Extract only the relevant metadata objects.
• Use a nested dictionary structure identical to what we do with the XBRL instant/duration tables to return the extracted metadata and key it by the PUDL table name.
• Pass in table-specific dictionary of instant/duration metadata when instantiating the TableTransformer class.
• Create a Ferc1AbstractTableTransformer class method or helper function that's called by __init__()
  • normalizes JSON to dataframes
  • concatenates instant/duration metadataframes
  • renames the xbrl_fact_name as appropriate for the class (store new name as a parameter?)
  • saves the result in an instance attribute for later use?

Notes / Questions

• ElectricEnergySourcesFerc1TableTransformer.drop_invalid_rows() was dropping all rows because its parameters had not been updated to use energy_mwh rather than energy_source_mwh. We need validations that will catch this kind of thing and fail loudly if we mess up.
• Why do we need to drop the disposition columns in the sources process_dbf() but we don't need to drop the sources columns in the dispositions process_dbf()? Shouldn't these transformers be symmetric?
• No record_id field in the electric energy sources / dispositions tables.
• What is the thinking behind renaming xbrl_factoid within merge_xbrl_metadata()? Does the rename need to be separately parameterized? Based on current examples it seems like we can always rename the xbrl_factoid column to whatever column is named in MergeXbrlMetadata.on.

[zaneselvans]

• We can parameterize the transformation of the metadata too.
• self.params has already been set when we're normalizing the metadata.
• At the very least we need to rename the name column to reflect whatever it means in the table at hand.
• This basic normalization can be done by the Ferc1AbstractTableTransformer.normalize_xbrl_metadata() method,
• Subsequent specialized metadata processing can be done by the child class, which calls the super() method and then applies its own changes on top of them.
• Maybe this is really process_xbrl_metdata() and not normalize_xbrl_metadata()
• Need to check-in with Christina about why we're doing this "staging" thing with column renames instead of just passing in parameters. Seems like an unnecessary complication but maybe I forgot something.