Open stucka opened 7 years ago
Branch created; performance looks to be about 10x. Still need to: -- Clean up. -- Set file handle flush and disk writes to maybe every 100 rows or so, to decrease I/O lag. Grab both at the same time. -- Fix creds.py to default to current directory. -- Test, test, test, test. 3.6 compatibility fixes may have broken 2.7. Still not sure where this leaves us with similarity of issues on https://github.com/PalmBeachPost/postgeo/issues/7 on parallel project, which I thought was a Mac difference and may really be a Python 3 v. Python 2 difference. -- Testing specifics include: Do we have the same effects in Python 2 as Python 3? Does it work from scratch in Linux and Windows? Can we find someone with a Mac? -- Seek testers from PythonJournos, maybe?
Variables are probably going to need to use locks for safety: http://effbot.org/zone/thread-synchronization.htm
Instead of writing to -geo and cache files directly with csvwriter, we probably ought to add them to a Queue.Queue and dump them periodically, maybe every couple hundred rows, to the CSVs, then flush and disk writes.
Weeeeee.
If we're paying for geocoding, we should be able to get more results faster. Should.
This depends on cache bug to be fixed, among others, in case we have a crash. Robustness of crash protection will need to be tested.