New dframe to sql

[sfsinger19103]

• Efficiency improvements:

  • Doesn't upload jurisdiction more than once
  • Replaces inefficient dframe_to_sql() routine with insert_to_cdf_db() and append_id_to_dframe()

• Better user messaging

• Caveat emptor! Change to data model:

  • removed ExternalIdentifier (and all ExternalIdentifier.txt)
  • In DB, eschew nulls in favor of 0 (for Id fields) and '' (for text fields)

• Moves successfully loaded files to the archive directory

[ericmtsai]

@sfsinger19103 it looks like the data loading works well. I tested with NC and all the data made it in correctly. However, it throws an error a the end as it tries to move the results files that have been processed:

>>> err = mdl.load_all()
Loading jurisdiction from /Users/eric/Desktop/election_anomaly/src/jurisdictions/NC to Engine(postgresql://postgres:***@localhost:5433/NC)
Processing results files ['results.par']

    Results uploaded with munger nc_gen to database Engine(postgresql://postgres:***@localhost:5433/NC)
from file /Users/eric/Desktop/nc_general/results_pct_20181106.txt
assuming jurisdiction /Users/eric/Desktop/election_anomaly/src/jurisdictions/NC
    Note:
{'munge_warning': 'Warning: Results for 768400 rows with unmatched contests will not be loaded to database.'}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/eric/Desktop/election_anomaly/src/election_anomaly/__init__.py", line 117, in load_all
    self.move_loaded_results_file(sdl,load_error)
  File "/Users/eric/Desktop/election_anomaly/src/election_anomaly/__init__.py", line 130, in move_loaded_results_file
    ui.archive_results(f, self.d['results_dir'], self.d['archive_dir'])
NameError: name 'f' is not defined

[sfsinger19103]

@ericmtsai just after your test I pushed a fix for the file-moving error-marking routine. Right now it prints errors to stdout, and if there are no errors but there are warnings it saves the warnings to a file in the archive directory.

[sfsinger19103]

Found bugs. Will reopen when they are fixed.