search

bisguzar / twitter-scraper

Scrape the Twitter Frontend API without authentication.
MIT License
3.88k stars 600 forks source link

Proxy usage #137

Open SomebodyOnceToldMeI opened 4 years ago

SomebodyOnceToldMeI commented 4 years ago

At now I have one request processed for ~1 second. I need it to be faster, is there a way to use proxies? I tried to add them, but requests doesn't support https proxies. I tried to run curl commands using proxy, but they run even slower, maybe because they don't work in parallel. Maybe there is a solution?

bisguzar commented 4 years ago

I don't have any suggestion. I'm adding label to your issue. I hope someone knows the solution and suggest it.

dgnsrekt commented 4 years ago

@SomebodyOnceToldMeI Requests documentation shows it does support https proxies. https://2.python-requests.org/en/master/user/advanced/#id10

It may be possible to reimplement the get_tweets function with a proxy parameter which will pass the proxy to the gen_tweets function then into the session.get(). https://github.com/bisguzar/twitter-scraper/blob/master/twitter_scraper/modules/tweets.py#L32

I'm currently running into a similar issue so I'll be experimenting.

bisguzar commented 4 years ago

Is there any update guys? We'll implement your solution into package @dgnsrekt but I want to keep everything simple. Even args of functions.