determine granularity match of depreciation records to steam records (thru MUL)

[cmgosnell]

Well... ~9% of the depreciation records line up perfectly with their corresponding ferc1 steam record :-/

connects_deprish_ferc1 = pd.merge(connects_deprish_eia, connects_ferc1_eia, 
                                  on=['record_id_eia'],
                                  suffixes=('_eia','_ferc1'),
                                  how='outer', indicator=True
                                 )

connects_clean = connects_deprish_ferc1[connects_deprish_ferc1['_merge'] == 'both']
f"Clean Depreciation to EIA Connections: {len(connects_clean)/len(connects_deprish_eia):.02%}"

Produces: 'Clean Depreciation to EIA Connections: 9.16%'

Next step is to determine how many of the depreciation records are at a bigger granularity than their corresponding ferc1 steam record. And then those that are smaller than their corresponding ferc record.

[cmgosnell]

My current musings tell me that most of the depreciation records are at a lower level than the records in the steam table:

Portion of depreciation records compared to FERC Steam are:
    Smaller:    40.78%
    Bigger:     1.95%
    Same level: 11.52%
    No link:    45.74%

I still haven't investigated the records without links. I expected there to be some because there are small plants and hydro plants in here that won't show up in the steam table, but I still need to track this down.

[cmgosnell]

Notes for fine tuning....

Restrict columns by:

• plant_id_pudl
• utility_id_pudl
• report_date

Things to figure out:

• true gran (should I change everything to it's "true gran" before attempting to merge?)
• ownership slices (solution unclear)

[cmgosnell]

On ownership. I'm struggling thinking through how to deal with different ownership between the depreciation records and the steam records.

If the steam record is an owned record and the deprish record is a total record.... how should we connect them? My first thought is that we could grab the ownership levels from the steam table and only do the fuzzy matching between depreciation and the MUL on ownership. Then the connections between the tables would be on the same grounds. If we are just trying to pull over the non-fuel opex costs/MWh, then this should be too bad, but I would have to be very careful to know that the owners the records could be different - and thus the data about them could be different. I'll keep mulling this one over to see if there are other options.

[cmgosnell]

We need to develop methods to choose the right record(s) from FERC out of the various options when connecting depreciation records and FERC1 steam. We are generating all of our options between these data sets by merging on these columns: ['plant_id_pudl', 'utility_id_pudl', 'report_date'].

Here are some options for methods. This is almost certainly not inclusive.

Option 1:

• [x] When the records are at the same level, this is easy. We just check whether or not the record_id_eia is the same. Easy peasy.

Option 2:

• [x] If there is only one record option from the FERC steam record, can we assume this record should be the matching record? We'll need to be careful because we may be aggregating the data on different slices of the plant.

Option 3:

• [x] ownership....uugh. If the records from FERC and depreciation have nearly the same records, but one is labeled as "owned" and one is labeled as "total", then can we assume they should be associated? We'd have to be careful in assigning values across these records because they would represent different slices of plants. This issue will run through all of these options.

__

• [x] The following options require including all qualifying/identifying records in the master unit list. This should be straightforward because this is already in place with grab_qualifiers. It should be as easy as adding all of the id columns into QUAL_RECORD_TABLES.

Option 3:

• Example:
  • Depreciation records: multiple generator level records (some have GT prime movers, some have CC prime movers)
  • FERC records: two prime mover records (GT and CC).
• [x] I thiiiink that we can assume that all of the GT generators should be associated with the GT prime mover record. These records will need a disaggregation method applied.

Option 4:

• [x] The opposite of option 3. These records will need an aggregation method applied.

Option x:

• [x] need to investigate the 22 (!) remaining matches... it looks like most of them require some level of overrides - often within the eia to ferc connection.

We also need to incorporate the primary line flag into the depreciation records so we can remove them in this matching process.