Incorporate new API throttling

rdohms / meetup-api-client

Guzzle powered Meetup.com API Client

MIT License

76 stars 24 forks source link

Incorporate new API throttling #3

Closed rdohms closed 9 years ago

rdohms commented 11 years ago

Add a plugin to handle the API rate limiting system used by Meetup.com

rdohms commented 10 years ago

Leaving this for a new version, pending replies from Meetup team about proper response HTTP code.

rdohms commented 10 years ago

API has now implemented code 429.

This may be a good start to researching how AWS does this: http://docs.aws.amazon.com/aws-sdk-php/latest/class-Aws.Common.Client.ThrottlingErrorChecker.html

rdohms commented 10 years ago

Guzzle provides a Backoff Plugin.

Need to look into:

[ ] Adjusting "throttle" identification to Meetup API format.
[ ] Define retry strategy for throttled requests
[ ] Define strategy for blocked requests (needs to wait until end of hour)

stevelacey commented 10 years ago

I am currently running into a bunch of throttling issues, any implementation that allows for pausing between requests would probably be good enough for me, I am essentially grabbing data on mass from Meetup and smashing it into Redis, if that task took a while because of pausing to avoid throttling, I'd live.

I couldn't find any guidelines in the Meetup docs however describing how many requests in what period are throttled, so maybe that's a problem.

rdohms commented 10 years ago

Its not about a specific number, it relates to the other traffic as well and such. So basically you have to listen for the throttle, then slow down before you get a block. You can't really predict it.

shaunybee commented 10 years ago

Friend of mine has a redis throttler if that is what you need? https://github.com/neilbeveridge/shrug

rdohms commented 10 years ago

@shaunybee that's not quite what we are talking about here. Its just a question of sitting down and configuring the backoff plugin to Meetup standards.

shaunybee commented 10 years ago

ah no worries - I am on the lookout for a throttler which led me to your page :) cheers

stevelacey commented 9 years ago

If anyone's interested, I finally implemented an albeit basic solution, by simply wrapping the client class and interogating all responses and sleeping then nearing the limit: https://github.com/phpsw/phpsw/blob/develop/src/Meetup/API/Client.php

rdohms commented 9 years ago

This is also solved by the patch in PR #19

rdohms commented 9 years ago

Implemented in #19 and accepted for v2.