Fix bug with EIA-923 Generation and Fuel Allocation

[grgmiller]

This PR closes https://github.com/singularity-energy/open-grid-emissions/issues/271, fixing an issue where our data pipeline was dropping a substantial amount of data due to mismatches in the reported energy source codes used in EIA-860 and EIA-923, and our failure to validate the outputs of the allocation process more carefully. While fixing this issue, we also came across several other issues that were causing anomalous emission factor outputs.

Fixes the generation and fuel allocation process

• Fix bugs in pudl allocate_net_gen module, as described in this PR (https://github.com/catalyst-cooperative/pudl/pull/2235):

  • Adds a new function add_missing_energy_source_codes_to_gens() that adds energy_source_codes that appear in the gf table but not gens to gens.
  • In some cases, non-zero fuel consumption and net generation is reported in the EIA-923 generation and fuel table that is associated with an energy_source_code that is not associated with that plant-prime mover in the gens table, which would cause these data to get dropped when these two tables are merged. To fix this, for each plant-pm, this function identifies such esc, and adds them to the gens_at_freq table as new energy_source_code columns.
  • The sub-function identify_missing_gf_escs_in_gens() identifies when there are fuels reported in the gf table for that plant-pm that are not listed in the gens table for that plant pm.
  • Adds the MISSING_SENTINEL value to the net_generation_mwh_g_tbl column. For some reason, this column had been commented out, which was leading to NaNs appearing in the data when dividing by this column when the value was zero. I un-commented this line.
  • In allocate_net_gen_by_gen_esc(), we no longer allow frac_from_g_tbl to be greater than 100%. This was previously happening when the mwh reported in the g table were greater than the mwh reported in the gf table. However, numbers greater than 100% was causing the frac_missing_from_g_tbl to become negative, which was resulting in nonsensical allocations. We implement the same cap on frac_from_bf_tbl in allocate_fuel_by_gen_esc().
  • In allocate_fuel_by_gen_esc(), when calculating frac_cap, the code had been dividing capacity_mw by capacity_mw_unit_fuel. However, this was resulting in some nonsensical allocations because fuel is being allocated by PM-fuel, not by unit. We changed this to divide by capacity_mw_pm_fuel instead. This is consistent with how frac_cap is calculated in the allocate_net_gen_by_gen_esc() function
  • Rename adjust_energy_source_codes() to adjust_msw_energy_source_codes() to more precisely describe what the function does

• Adds new entries to the manual emissions factor tables for NOx and SO2 to represent fuel-boiler combinations that had previously been getting dropped from the data due to this bug.
• Changes the pudl version that we use in our environment from catalyst-cooperative/pudl@main to grgmiller/pudl@oge_release. This will give us more control over performing fixes like this in the future.
• Adds a new validation check to the EIA-923 data cleaning process to verify that for each plant, the total allocated fuel and generation matches the total fuel and generation reported in the input generation and fuel table (basically that the allocation process is not dropping or inflating the data).

Updates plant primary fuel identification

• When assigning the plant primary fuel based on the most consumed fuel, we were previously assigning this based on the fuel with the highest fuel_consumed_mmbtu. However, we should be using fuel_consumed_for_electricity_mmbtu since we want to assign the primary fuel used for electricity generation.
• Sometimes nuclear generators report 0 fuel consumption in EIA-923. Since we were assigning a plant's primary fuel first based on fuel consumption, this meant that sometimes if a nuclear plant had a backup fossil generator, the plant was being assigned the fuel code of that backup generator. To fix this, we now assign the primary fuel of any plant that contains a nuclear unit based on the nameplate capacity of the unit.

Updates to gross to net generation conversions

• Previously, when converting CEMS gross generation to net generation, we had filtered out any ratios that were greater than 1.5 or less than 0.2. However, these values were somewhat arbitrary and turning out to be too wide of a range. For example, when there was a large discrepancy between CEMS gross generation and EIA-923 net generation, we were scaling the CEMS generation to match, even though the fuel consumption and emissions reported in CEMS also disagreed and were not being scaled. This was leading to instances where a plant was using CEMS CO2 totals but EIA-923 net generation totals, resulting in the plant having abnormally high emission rates. To be consistent, if we are going to use CEMS data at all, we want to make sure that the net generation values are reasonable given the reported net generation. After analyzing three years (2019-2021) of annual gross to net ratios, both at the plant and subplant levels, it appears that generally the interquartile range of GTN ratios is between 0.75 and 1.00, with an upper bound around 1.25. Thus, we are now using 0.75 as the lower bound for filtering out ratios, and 1.25 as the upper bound.
• Previously, the backstop gross to net generation approach if all other conversion factors were not available was to assume that gross generation equaled net generation (i.e. a GTN ratio of 1). However, as identified in https://github.com/singularity-energy/open-grid-emissions/issues/177, the EIA has default gross to net conversion factors for each prime mover that they use. This PR introduces these PM-specific gross to net conversion factors as the default backstop option now. As noted in the issue, there are still improvements that need to be made before #177 can be closed, but this is a step in the right direction.

Flags potentially anomalous generated emission factors

• When outputting annual plant level data, we add a new validation check that calculates a generated co2 rate, and flags any rates that appear to be anomalous, so that we can manually inspect these plants to see if there are any unexpected results. For this test, we define anomalous values two ways. On the high end, the check flags the plant if the co2 rate is higher than 15,000 lb/MWh. On the low end, the check flags any plants that have rate lower than 10 lb/MWh but higher than 0 MWh.

Other

• Fixes futurewarnings from pandas related to grouping non-numeric columns when using pd.groupby(). This involved only summing numeric columns and sometimes adding the numeric_only=True argument to .sum().

TODO:

• [x] remove WIP notebooks
• [ ] Update documentation