search

osome-iu / botometer-python

A Python API for Botometer by OSoMe
https://botometer.osome.iu.edu
MIT License
371 stars 59 forks source link

Rate limit of "ultra" API access #69

Closed JanaLasser closed 2 years ago

JanaLasser commented 2 years ago

I am trying to classify a large number (250k) accounts using the "ultra" subscription to the API. Since I need all sub-scores, I am not using the "lite" endpoint.

I have tried to parallelize calls but API responses are very slow and stop occasionally, for some period of time. This looks to me like I have run into some hidden rate limit, even though the rate limit of "one request/second" should only apply to the "basic" subscription. Is there another rate limit that I should be aware of?

filmenczer commented 2 years ago

The average latency for the v4 endpoint is currently around 1.4 s. This includes communication between RapidAPI and our server, calculation of the 1k+ features, and classification. On top of that, the botometer-python library has to fetch the data from Twitter, therefore is also subject to the Twitter API rate limits.

@yangkcatiu may be able to provide further details.

JanaLasser commented 2 years ago

Thanks for the quick reply! I think what I am observing might be due to the Twitter API rate limit then.

If I understood correctly, I should be able to reduce the 1.4s latency of the v4 endpoint by parallelizing requests (given that I am not constrained by the Twitter API rate limit), correct?

yang3kc commented 2 years ago

To add to what @filmenczer mentioned, we don't have any explicit rate limit for the endpoints under the ULTRA plan. But RapidAPI might have some hidden constraints on concurrency. We are not aware of any, but it's logical. You can actually contact their support team and ask about it.

Also, please be advised that we only have one server hosting Botometer with limited threads handling the requests. Parallelizing your requests should help you get your results faster up until a certain point.

JanaLasser commented 2 years ago

Thank you for the additional info! I will go ahead and ask RapidAPI about their internal constraints.