search

easherma / mapextractor

3 stars 1 forks source link

ensure burst limit is not exceeded #10

Open easherma opened 8 years ago

easherma commented 8 years ago

We need to throttle requests + also make sure that we're not continuing to send requests after we've gotten all the expected results. I've played around a bit with incorporating the limiter library but no luck so far.

easherma commented 8 years ago

atm im testing it really slow, at 10 per minute but the query im running is around 2300 results, its taking a long time to run, which id expect but its also getting really close to hitting the daily limit, which should be 10,000 calls im not sure if its re-running calls maybe? i also have no idea how many records I've got so far, since we dont really save the data as we go. and if i stop it, I wont have any output at all

https://github.com/easherma/mapextractor/blob/boxsplit/routes/index.js#L70

easherma commented 8 years ago

@ajredovan just see if you can follow this example: https://github.com/jhurliman/node-rate-limiter and find the right place in the code to nest