vladkens / twscrape

2024! X / Twitter API scrapper with authorization support. Allows you to scrape search results, User's profiles (followers/following), Tweets (favoriters/retweeters) and more.
https://pypi.org/project/twscrape/
MIT License
1.12k stars 133 forks source link

Concurrency Issues #217

Open anonymous-sushi opened 1 month ago

anonymous-sushi commented 1 month ago

Trying to async.gather user_tweets_and_replies gets bottlenecked when doing 10+ accounts at once, is there any way to properly scale this (with many accounts) with less delay in between calls?

vladkens commented 1 month ago

@anonymous-sushi Hi. What exactly are you calling a bottleneck? I've run it with hundreds of accounts and don't remember having any problems with it.

anonymous-sushi commented 1 month ago

@anonymous-sushi Hi. What exactly are you calling a bottleneck? I've run it with hundreds of accounts and don't remember having any problems with it.

if you try running 100 accounts calling user_tweets concurrently, it will bottleneck and take 3-5 seconds to async gather them all versus a normal 0.01 second call, ideally with enough accounts we should be able to fetch these all simutaneously in roughly the same amout of time 0.01~