search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
456 stars 106 forks source link

Test all FERC1 tables and see which can be automatically imported to postgres #68

Closed zaneselvans closed 5 years ago

zaneselvans commented 7 years ago

We have functions for automatically slurping database tables from the old DBF format into postgres, but we haven't tried to use them on all of the old DBF tables. We might as well give all of the old tables a shot -- so we at least know which ones can be imported automatically for which years, in case we need to access data from some of them in the future.

Once a table has been tested for the entire 2004-2015 date range, it should be added to the ferc1_working_tables list in constants.py.

zaneselvans commented 5 years ago

I tried to clone every FERC Form 1 table into postgres, for every year using 2017 as the template database, and surprisingly, a huge number of earlier years for all the tables work -- it doesn't seem to be that older years have different structures. Rather, there seems to be poor enforcement of database constraints in the FoxPro database, and so there are a bunch of little errors that cause a table-year to fail to come into the more stringently enforced postgres. So it might be possible to come up with a long list of tedious work arounds for all of those little issues.

There's also a set of tables that work from early years but only up to the early 2000s, and another set that appears in the early 2000s, and I'm guessing they just don't exist in those other years (though the error they give is about not having primary key fields, which they wouldn't if there were no fields).

There are some tables that don't work for any years, including f1_row_lit_tbl which is too bad, since it contains descriptions of all of the form rows. IIRC this is because it has some kind of uninterpretable character in one of the entries, that breaks dbfread, but maybe we could try/except around that.

Then there's a mysterious class of tables that work up to some year, and then not only stop working, but prevent anything after them from working. I think this may be a result of too many database connections getting opened and then not closed -- I was only able to do batches of this exploration because of this issue... and tried a little bit to track down where the stale connections are coming from, but failed to find the source. Anyway, the working table-years appear to be...

# A dictionary listing all the years for each FERC Form 1 table that could be
# cloned from FoxPro/DBF with our current process and a refyear of 2017.
ferc1_working_table_years = {
    # Special table, always works.
    'f1_respondent_id': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    # First group: no big irregularities, though some years didn't work.
    'f1_acb_epda': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_accumdepr_prvsn': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_accumdfrrdtaxcr': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_adit_190_detail': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_adit_190_notes': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_adit_amrt_prop': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_adit_other': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_adit_other_prop': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015, 2016, 2017],
    'f1_allowances': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_capital_stock': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_cash_flow': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2007, 2009, 2010, 2011, 2012, 2014, 2016, 2017],
    'f1_cmmn_utlty_p_e': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_control_respdnt': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015, 2016, 2017, ],
    'f1_co_directors': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_cptl_stk_expns': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_dacs_epda': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_edcfu_epda': [1994, 1995, 1996, 1997, 1999, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_elctrc_erg_acct': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_elctrc_oper_rev': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2007, 2009, 2010, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_elc_oper_rev_nb': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_elc_op_mnt_expn': [1994, 1995, 1996, 1997, 1998, 1999, 2001, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2016, 2017, ],
    'f1_fuel': [1996, 1997, 1999, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_general_info': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015, 2016, 2017, ],
    'f1_gnrt_plant': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_miscgen_expnelc': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_misc_dfrrd_dr': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_mthly_peak_otpt': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_mtrl_spply': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2017, ],
    'f1_nuclear_fuel': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_officers_co': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_othr_dfrrd_cr': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_othr_pd_in_cptl': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_othr_reg_assets': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2016, 2017, ],
    'f1_plant_in_srvce': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_pumped_storage': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_purchased_pwr': [1994, 1995, 1996, 1997, 1999, 2001, 2002, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_reconrpt_netinc': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_reg_comm_expn': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_respdnt_control': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_retained_erng': [1994, 1995, 1996, 1997, 1999, 2001, 2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2016, 2017, ],
    'f1_r_d_demo_actvty': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2016, 2017, ],
    'f1_sales_by_sched': [1994, 1995, 1996, 1997, 1999, 2000, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_sale_for_resale': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_sbsdry_totals': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_slry_wg_dstrbtn': [1994, 1995, 1996, 1997, 1999, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_substations': [1994, 1995, 1996, 1997, 2002, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2015, 2016, 2017, ],
    'f1_taxacc_ppchrgyr': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_unrcvrd_cost': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_work': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_xmssn_adds': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_xmssn_line': [1994, 1995, 1996, 1997, 1998, 1999, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2016, 2017, ],
    'f1_xtraordnry_loss': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_codes_val': [],
    'f1_sched_lit_tbl': [2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_audit_log': [],
    'f1_col_lit_tbl': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_load_file_names': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_privilege': [],
    'f1_sys_error_log': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_unique_num_val': [],
    'f1_row_lit_tbl': [],
    'f1_footnote_data': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_hydro': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_footnote_tbl': [],
    'f1_ident_attsttn': [2000, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_steam': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_leased': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_sbsdry_detail': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_plant': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_long_term_debt': [1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2016, 2017, ],
    'f1_email': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_freeze': [],
    'f1_pins': [],
    'f1_s0_checks': [],
    'f1_s0_filing_log': [2000, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    'f1_security': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, ],
    # Tables that only exist for part of the time series. This seems to have
    # created some primary key errors in the testing.
    'f1_construction': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002],
    'f1_csscslc_pcsircs': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002],
    'f1_dscnt_cptl_stk': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002],
    'f1_electric': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002],
    'f1_envrnmntl_expns': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002],
    'f1_envrnmntl_fclty': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002],
    'f1_nbr_elc_deptemp': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001],
    'f1_nonutility_prop': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002],
    'f1_overhead': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002],
    'f1_pccidica': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001],
    'f1_security_holder': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002],
    'f1_106_2009': [2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_106a_2009': [2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_106b_2009': [2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    # dead ender
    'f1_208_elc_dep': [2007, 2008, 2009, 2010, 2011, 2012, 2014, 2015, 2016],
    'f1_231_trn_stdycst': [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_324_elc_expns': [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_325_elc_cust': [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2015, 2016, 2017],
    'f1_331_transiso': [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    # dead ender.
    'f1_338_dep_depl': [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2015, 2016],
    'f1_397_isorto_stl': [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_398_ancl_ps': [2004, 2005, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_399_mth_peak': [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_400_sys_peak': [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_400a_iso_peak': [2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_429_trans_aff': [2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_allowances_nox': [2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    # dead ender.
    'f1_cmpinc_hedge_a': [2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2016],
    'f1_cmpinc_hedge': [2002, 2003, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    'f1_rg_trn_srv_rev': [2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017],
    # "Dead enders" These groups did something weird after the last year.
    # See also a few of the above truncated time series.
    'f1_bal_sheet_cr': [1994, 1995, 1996, 1997, 1998, 1999, 2003, 2007, 2009, 2010, 2011, 2012, 2016],
    'f1_comp_balance_db': [1994, 1995, 1996, 1997, 1998, 2001, 2002, 2003, 2007, 2009, 2010, 2011, 2016],
    'f1_important_chg': [1994, 1995, 1996, 1997],
    'f1_incm_stmnt_2': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013],
    'f1_income_stmnt': [1994, 1995, 1996, 1997, 1998, 1999, 2001, 2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2016],
    'f1_note_fin_stmnt': [1994, 1995, 1997],
    'f1_othr_reg_liab': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016],
    'f1_schedules_list': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2004, 2007, 2008, 2009, 2010, 2011, 2013, 2014, 2015, 2016],
    'f1_utltyplnt_smmry': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016],
    'f1_xmssn_elc_bothr': [1994, 1995, 1996, 1997, 1999, 2000, 2001, 2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016],
    'f1_xmssn_elc_fothr': [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016],
}
zaneselvans commented 5 years ago

With a few tweaks to which tables are given composite primary keys vs. auto-incremeting pseudo-keys, I've gotten 109 of the 117 possible tables to load into the cloned FERC database, for at least some subset of the years. Additional, probably tedious, work would be required to figure out the hundreds of minor issues preventing the other table-years from loading directly.