Create a data handler thread that bulk inserts at some interval
The worker threads now just post their retrieved pages to this
list of pages to be bulk inserted with executemany
Rely on psycopg2s encoding. Don't try to hand format the
insert string.
Remove MongoDB stuff
Remove wierd exceptions
Remove old codez
Re-License to a more liberal license (BSD)
This is far from complete, but this removes a lot of old stuff that
either doesn't work or is super fragile. After this 60% of the time it
should work all of the time.
executemany
psycopg2
s encoding. Don't try to hand format the insert string.This is far from complete, but this removes a lot of old stuff that either doesn't work or is super fragile. After this 60% of the time it should work all of the time.