How to handle rate limiting with multiple apps

ttezel / twit

Twitter API Client for node (REST & Streaming API)

4.31k stars 568 forks source link

How to handle rate limiting with multiple apps #501

Open ghost opened 5 years ago

ghost commented 5 years ago

Hi,

I need to fetch large amounts of data. I am fetching tweets from every follower of a specific user, the user has over 1 million followers.

I have multiple twitter apps is it possible to handle rate limiting by using these multiple api tokens. For example when one is about to reach the limit to exchange the next request to use the next api token.

Is there an example of this?

netik commented 4 years ago

Collect all of the API requests into manageable chunks of pagination or whatever.

Feed those requests into a queue (like, Bull / Redis). Use the queue ratelimiter to ratelimit requests going to Twitter. Write a worker to process the requests as they come back.