taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 579 forks source link

NO RESULTS: unable to collect any tweets. #239

Open has-ctrl opened 4 years ago

has-ctrl commented 4 years ago

Just a few days ago, everything was working fine. Unfortunately since yesterday, almost all of my requests are left unanswered. This is extremely annoying because there is time pressure on my project.

This is the result for today (this is the only way it is working): INFO: queries: ['Trump since:2019-11-08 until:2019-11-09'] INFO: {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; rv:2.2) Gecko/20110201'} INFO: Querying Trump since:2019-11-08 until:2019-11-09 INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-11-08%20until%3A2019-11-09&l= INFO: Using proxy 138.197.222.35:8080 INFO: Scraping tweets from https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=TWEET-1192954944879550466-1192954955084304389&q=Trump%20since%3A2019-11-08%20until%3A2019-11-09&l= INFO: Using proxy 151.253.165.70:8080 INFO: Scraping tweets from https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=thGAVUV0VFVBaEgLup9YydjiEWisC8rcGNnY4hEjUAFQAlAFUAFQAA&q=Trump%20since%3A2019-11-08%20until%3A2019-11-09&l= INFO: Using proxy 119.192.195.83:443 INFO: Got 56 tweets for Trump%20since%3A2019-11-08%20until%3A2019-11-09. INFO: Got 56 tweets (56 new).

While this is the result for last month: INFO: queries: ['Trump since:2019-05-08 until:2019-05-09'] INFO: {'User-Agent': 'Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16'} INFO: Querying Trump since:2019-05-08 until:2019-05-09 INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-05-08%20until%3A2019-05-09&l= INFO: Using proxy 138.197.222.35:8080 INFO: Retrying... (Attempts left: 50) INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-05-08%20until%3A2019-05-09&l= INFO: Using proxy 151.253.165.70:8080 INFO: Retrying... (Attempts left: 49) INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-05-08%20until%3A2019-05-09&l= INFO: Using proxy 119.192.195.83:443 INFO: Retrying... (Attempts left: 48) INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-05-08%20until%3A2019-05-09&l= INFO: Using proxy 188.226.141.127:3128 INFO: Retrying... (Attempts left: 47) INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-05-08%20until%3A2019-05-09&l= INFO: Using proxy 139.59.64.9:8080 INFO: Retrying... (Attempts left: 46) ...

INFO: Retrying... (Attempts left: 3) INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-05-08%20until%3A2019-05-09&l= INFO: Using proxy 200.89.178.186:80 INFO: Retrying... (Attempts left: 2) INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-05-08%20until%3A2019-05-09&l= INFO: Using proxy 207.154.231.216:3128 INFO: Retrying... (Attempts left: 1) INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=Trump%20since%3A2019-05-08%20until%3A2019-05-09&l= INFO: Using proxy 200.255.122.174:8080 INFO: Got 0 tweets for Trump%20since%3A2019-05-08%20until%3A2019-05-09. INFO: Got 0 tweets (0 new).

The strangest thing is that the links actually do link to pages that have tweets (but the page returns no tweets: see below). twitter

Does anyone have any idea how to fix this?

I have already tried:

F1282231 commented 4 years ago

I have the exactly same situation. Sometimes it works sometimes not, still trying to figure out why.

fimenten commented 4 years ago

I have the exactly same situation since yesterday. I have confirmed html which was fetched and it said 'There are no results', nevertheless There are tweets if I manually searched with same query.