search

medialab / minet

A webmining CLI tool & library for python.
GNU General Public License v3.0
290 stars 26 forks source link

potential changes in rate limit of twitter public API #976

Closed taniki closed 2 months ago

taniki commented 5 months ago

I am trying to scrape tweets from a list of users with the minet tw scrape tweets command. It works despise X chaotic changes. I run randomly into a rate limit that is escalading quickly. I don't know if there is something to do avoid it.

Screenshot from 2024-06-17 23-46-18

Yomguithereal commented 5 months ago

Usually this happens because you are using your Twitter account on some browser or phone app while scraping, since the rate limit is shared. Could this be the issue here?

taniki commented 5 months ago

That's unlikely. I hit the limit last night while my browser was closed (I will double check tonight). I am also using a nitter instance to browse twitter anonymously and don't have any open tab on twitter. I am not using the app either, the limit also appears with two distinct accounts used on two computers on two networks.

Yomguithereal commented 4 months ago

@taniki I have updated the rate limit of the scraper to be more conservative from v2.0.7. Tell me if this fixes your issue.