taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 579 forks source link

proxy ip was blocked and video url was not found #256

Open gwontaeyong opened 4 years ago

gwontaeyong commented 4 years ago

i'm crawling about 3000query for one day. after get 10000 tweets, my query response code is 429. it's means 'too many request' after 15minutes around crawling work again but i wonder i using many proxy from 'https://free-proxy-list.net/' but ip was blocked. so i think code for change ip in proxy list has bug../?

to fix this error i update version but the respose code still get 429. and found other errror video_url was not catched because this line (https://github.com/taspinar/twitterscraper/blob/master/twitterscraper/tweet.py#L81) raise TypeError.