Open zaneselvans opened 7 years ago
It seems like this is a very similar problem to the one that @cmgosnell has been working on with backfilling constant values (like location) in the EIA 860 data. Maybe the same tool can be applied?
:-/ sort of. what I've been doing is not backfilling. I've been attempting to find the canonical annual or static values from multiple tables and store that info in one table as opposed to all over the place. But for the static info, this can be used to backfill. Nonetheless, the process is pretty generalizable and could certainly be applied to FERC.
There are some cases in which we get good clean category information about a plant and its fuel for some years, but not for others. Right now this means that we're perhaps unnecessarily dumping data that we could keep -- if we were willing to assign the categories for the gapping years.
For instance, in 2014-2015, (respondent_id=2, plant_name=Barry) has nothing in the fuel or fuel_unit fields in the FERC Form 1 database clone, but for 2007-2012 we see that it's fuel=coal and fuel_unit=tons, and there are good numerical values in the fuel_quantity, fuel_avg_heat, fuel_cost_btu, and other fields.
Once we have generated time series for each FERC plant and inserted the new Plant ID values into the fuel_ferc1 table, we can go ahead and fill in many of these missing categorical values. This will require addressing issue #39 as well -- since we'll need to retain many imperfect records that are currently getting dropped in order to fix them.