geocoder over limit query being thrown

[eric-jahn]

GeocoderError: Error OVER_QUERY_LIMIT Query: https://maps.google.com/maps/api/geocode/json?region=&latlng=28.243738%2C-80.743672&sensor=false&bounds=&language= [E 141205 21:51:02 web:849] 500 GET /resources?kind=shelter&lat=28.243738&lng=-80.743672&radius=100 (107.145.106.186) 118.02ms

http://stackoverflow.com/questions/9805529/geocoding-api-over-query-limit

[eric-jahn]

Google's geocoding service is the best, still, after comparing. Rate limitation is currently 5 request /sec, even if we pay for the service. They recommend a client side approach, and 100% client side requests would resolve this. Regardless, we need for this limitation to not bring the system down, so that it exits gracefully, and notifies the user that this is what's going on. Geocoder limits of 5/s are being reached, causing the error. There are 4 threads that can call the google API from 6 different areas. Reverting to one thread only for now and setting a watch on the number of calls.

[eric-jahn]

Recent fixes have stabilized the site at homelesshelper.us. On the new testing site at http://75.127.68.15:8888 , we're rolling out more robust code that should put the stake in these geocoder issues for good. The site errors are mainly due to incoming street addresses containing information that the Google geocoder cannot process. Records like these fail, where the "Program Address" is not resolving:

[{u'AgencyName': None, u'ContactName': None, u'ContactPhone': u'5404321812', u'ProgramAddress': u'P.O. Box 1478', u'ProgramAddressFull': None, u'ProgramCity': u'Harrisonburg', u'ProgramKey': u'126603', u'ProgramName': u'Mercy House, Inc.', u'ProgramType': None, u'ProgramZip': u'22803', u'SiteGeocode': None, u'TargetPopAName': None, u'UnitsAvailable': u'0', u'UnitsOccupied': u'0', u'UnitsTotal': u'0', u'UpdateTimeStamp': u'2014-10-08 14:38:45'}]

or

[{u'AgencyName': None, u'ContactName': None, u'ContactPhone': u'7706425500', u'ProgramAddress': u'200 Hannover Park Rd Ste 100', u'ProgramAddressFull': None, u'ProgramCity': u'Atlanta', u'ProgramKey': u'126547', u'ProgramName': u'Mary Hall Freedom House', u'ProgramType': None, u'ProgramZip': u'30350', u'SiteGeocode': None, u'TargetPopAName': None, u'UnitsAvailable': u'0', u'UnitsOccupied': u'0', u'UnitsTotal': u'0', u'UpdateTimeStamp': u'2014-10-08 14:38:40'}]

As a consequence they are not entered in the mongo db and the app keeps querying google over and over, causing the Geocoder to reach its overage limit and err. So we're taking a better approach. First we try once with the address. If that fails, we just try again with the Program Name and ZIP. It works as a good backup.

To demonstrate, if you go http://75.127.68.15:8888/position.html and enter

Shelter Name = "Mercy House, Inc" and ZIP = "22803"

You will see that it can retrieve the business info. It appears that this approach never fails if you enter at least a valid zip code, and it returns the lat/long of the ZIP code. I tried against the +/-200 failing addresses and I got all of them through with the exception of 6 that were test of the db (no shelter name and/or zip). Those 6 are thrown out, and we'll log it. When the address is vague, and it returns more than one result, it just takes the first one (and logs that).

The main improvement is that the entry will be only processed at most twice, not looping and erring. When coupled with only looking up changed information, this should make the app far less "chatty" with Google.