search

geduldig / TwitterAPI

Minimal python wrapper for Twitter's REST and Streaming APIs
936 stars 263 forks source link

too many requests #180

Closed caimeng2 closed 3 years ago

caimeng2 commented 3 years ago

I am using TwitterPager for 2/tweets/search/all and it runs smoothly until about 10,000 tweets. Then I keep getting this "too many requests" error. Any suggestion how to work around it? Traceback (most recent call last): File "get_full.py", line 33, in for item in r: File "/anaconda3/lib/python3.8/site-packages/TwitterAPI/TwitterPager.py", line 53, in get_iterator it = r.get_iterator() File "/anaconda3/lib/python3.8/site-packages/TwitterAPI/TwitterAPI.py", line 257, in get_iterator raise TwitterRequestError(self.response.status_code, msg=self.response.text) TwitterAPI.TwitterError.TwitterRequestError: ('{"title":"Too Many Requests","type":"about:blank","status":429,"detail":"Too Many Requests"}',) (429): {"title":"Too Many Requests","type":"about:blank","status":429,"detail":"Too Many Requests"}

geduldig commented 3 years ago

You must respect Twitter's rate limits. TwitterPager.get_iterator takes an optional wait parameter described here. You can find the rate limit in Twitter's doc for the request.

The default wait between requests used by TwitterPager is 5 seconds, which should be enough for this2/tweets/search/all. Assuming you are making this request in only one place in your code, I'm not sure why you are exceeding the rate limit, but try increasing the wait and see it that solves your problem.

caimeng2 commented 3 years ago

Not sure why I exceeded the limit but wait works. Thank you!