zaneselvans commented 4 years ago

In many of our older EIA transformation functions, we preemptively drop columns from the tables that are being processed, in order to produce normalized tables. However, many of these columns contain information about the entities (plants, generators, utilities) that should be integrated into the entity harvesting and resolution process, which happens after the transform step.

Discarded Columns

Check whether the column name is defined in pudl.metadata.fields
If it is not defined but does correspond to an existing column, change the name in the appropriate column_map.csv under src/pudl/package_data/{data source}/ so that it matches the DB schema.
If the column does not correspond to any existing defined field, it may be appropriate to discard it. E.g. total_fuel_consumption_mmbtu is an annual total of monthly values that are retained, and so we don't need it.
If the column corresponds to a defined field (either before or after the name has been fixed) then retain it and debug any issues that keeping it around results in later on in the transform process. It should make it to the harvesting step and go into the process of informing plant/generator/boiler/utility attributes.

EIA-860

`pudl.transform.eia860.ownership()`

None

`pudl.transform.eia860.generators()`

None

`pudl.transform.eia860.plants()`

None

`pudl.transform.eia860.utilities()`

None

EIA-923

`pudl.transform.eia923.plants()`

None

`pudl.transform.eia923.generation_fuel()`

[ ] combined_heat_power
[ ] plant_name_eia
[ ] operator_name (probably utility_name_eia)
[ ] operator_id (probably utility_id_eia)
[ ] plant_state
[ ] census_region
[ ] nerc_region
[ ] naics_code
[ ] fuel_unit (should probably be dropped, since unit is implied by fuel type)
[ ] total_fuel_consumption_quantity (annual total?)
[ ] electric_fuel_consumption_quantity (annual total?)
[ ] total_fuel_consumption_mmbtu (annual total?)
[ ] elec_fuel_consumption_mmbtu (annual total?)
[ ] net_generation_megawatthours (annual total?)
[ ] early_release

`pudl.transform.eia923.boiler_fuel()`

This one may give you trouble. See #1847 and #1836.

[ ] combined_heat_power
[ ] plant_name_eia
[ ] operator_name (probably utility_name_eia)
[ ] operator_id (probably utility_id_eia)
[ ] plant_state
[ ] census_region
[ ] nerc_region
[ ] naics_code
[ ] fuel_unit (should probably be dropped, since unit is implied by fuel type)
[ ] total_fuel_consumption_quantity (annual total?)
[ ] balancing_authority_code_eia
[ ] early_release
[ ] reporting_frequency_code
[ ] data_maturity (WE add this field in the extraction... getting dropped b/c of aggregations. See #1847)

`pudl.transform.eia923.generation()`

[ ] combined_heat_power
[ ] plant_name_eia
[ ] operator_name (probably utility_name_eia)
[ ] operator_id (probably utility_id_eia)
[ ] plant_state
[ ] census_region
[ ] nerc_region
[ ] naics_code
[ ] early_release

`pudl.transform.eia923.coalmine()`

None -- we really do just want the very small set of columns retained here, as we're stripping them out to create a new table, normalizing the Fuel Receipts & Costs table.

`pudl.transform.eia923.fuel_receipts_costs()`

[ ] plant_name_eia
[ ] plant_state
[ ] operator_name (probably utility_name_eia)
[ ] operator_id (probably utility_id_eia)
[ ] mine_id_msha (should be dropped)
[ ] mine_type_code (should be dropped)
[ ] state (of the mine?)
[ ] county_id_fips (of the mine?)
[ ] state_id_fips (of the mine?)
[ ] mine_name (should be dropped)
[ ] regulated (mine or plant?)
[ ] early_release

zaneselvans commented 2 years ago

@cmgosnell and I are going to help get @knordback working on this issue as a way to become more familiar with the harvesting process, working with our code, Jupyter, etc.

zaneselvans commented 1 year ago

@cmgosnell while talking over some of these fields with @knordback yesterday, I noticed that the associated_combined_heat_power field is part of the generators_entity_eia table, but there's another combined_heat_power field being reported in e.g. the generation_fuel_eia923 table, and looking at the spreadsheets, it seems like that field pertains to the plant (which makes some sense given that generation_fuel_eia923 is reported on a date, plant, prime-mover, fuel basis).

Are these different attributes? Should there be a CHP field at both the generator and the plant level? Should this really be a permanent attribute, or is it another one that changes slowly? Does the generator field really just indicate that the generator is part of a plant that does CHP? Or that it's part of a generation unit that does CHP? Could the plant or plant-prime-fuel level CHP status be inferred from the generator-level CHP attributes?

Right now we're discarding the CHP column reported in generation_fuel_eia923.

@grgmiller or @gschivley do either of you have more context on the relationship between these two different CHP fields?

cmgosnell commented 1 year ago

I don't know exactly. associated_combined_heat_power originates in the generator table. I would not be surprised if there were plants that had some units contributing to a CHP and some that just generated power. I don't think it's generally a good idea to base any logic about the workings of a plant based off of the reporting structure of the generation_fuel_eia923 table. I personally would check whether this value is actually consistent across all generators within a plant before thinking about moving it. But also i could definitely imagine this changing over time (albeit very rarely!).

zaneselvans commented 1 year ago

It seems like we should probably do an exhaustive check of all the currently "permanent" generator attributes on the pre-harvested dataframes... and see how permanent they actually are.

grgmiller commented 1 year ago

I do not have any context on these two fields.

knordback commented 1 year ago

I'll hold off on this one for now.