search

taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 581 forks source link

Twitterscraper stuck scraping, not gathering tweets #323

Closed erb13020 closed 3 years ago

erb13020 commented 4 years ago

When I try to query tweets, it keeps looping and doesn't scrape. My code was working last week but stopped working today. Here is my code.

def scrape(y, m, d, query):

    begin_date = dt.date(y, m, d)
    end_date = begin_date + dt.timedelta(days=1)
    tweets = query_tweets(query, begindate=begin_date, enddate=end_date, poolsize=30)

    df = pd.DataFrame(t.__dict__ for t in tweets)

    return df

Here is my output when I try to run my query.

INFO: {'User-Agent': 'Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16'}
Scraping tweets for 27/2/2020
INFO: queries: ['tesla since:2020-02-27 until:2020-02-28']
INFO: Querying tesla since:2020-02-27 until:2020-02-28
INFO: Scraping tweets from https://twitter.com/search?f=tweets&vertical=default&q=tesla%20since%3A2020-02-27%20until%3A2020-02-28&l=
INFO: Using proxy 1.20.101.114:34979
INFO: Scraping tweets from https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=TWEET-1233179367884361731-1233179901437628417&q=tesla%20since%3A2020-02-27%20until%3A2020-02-28&l=
INFO: Using proxy 45.32.155.143:8080
INFO: Scraping tweets from https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=thGAVUV0VFVBaGwLTxvoaRnSIWgsC6lcalkZ0iEjUAFQAlAFUAFQAVARUAFQAA&q=tesla%20since%3A2020-02-27%20until%3A2020-02-28&l=
INFO: Using proxy 157.119.207.10:6666
INFO: Scraping tweets from https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=thGAVUV0VFVBaAgLXxxeiQnSIWgsC6lcalkZ0iEjUAFQAlAFUAFQAVARUAFQAA&q=tesla%20since%3A2020-02-27%20until%3A2020-02-28&l=
INFO: Using proxy 13.85.25.59:8080
INFO: Scraping tweets from https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=thGAVUV0VFVBaAwLz5sciQnSIWgsC6lcalkZ0iEjUAFQAlAFUAFQAVARUAFQAA&q=tesla%20since%3A2020-02-27%20until%3A2020-02-28&l=
INFO: Using proxy 181.118.167.104:80
INFO: Scraping tweets from https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=thGAVUV0VFVBaAwL7Vmq2QnSIWgsC6lcalkZ0iEjUAFQAlAFUAFQAVARUAFQAA&q=tesla%20since%3A2020-02-27%20until%3A2020-02-28&l=
INFO: Using proxy 178.128.216.13:1080
INFO: Scraping tweets from https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=thGAVUV0VFVBaAwKKxpouQnSIWgsC6lcalkZ0iEjUAFQAlAFUAFQAVARUAFQAA&q=tesla%20since%3A2020-02-27%20until%3A2020-02-28&l=
INFO: Using proxy 124.41.211.196:58563

Would implementing the pull request in https://github.com/taspinar/twitterscraper/pull/302 fix this issue?

Adi2K commented 4 years ago

stuck with the same thing

danyaljj commented 3 years ago

Having the same issue.

GalDayan commented 3 years ago

Having the same issue

erb13020 commented 3 years ago

Has anyone had any success by using https://github.com/taspinar/twitterscraper/pull/302 ?

taspinar commented 3 years ago

@erb13020 @Adi2K @danyaljj @GalDayan Do you still have this issue with version 1.5.0? (which has the updated header file)

erb13020 commented 3 years ago

@taspinar Yes, it works using version 1.5.0

I was having this problem using version 1.4.0. I also upgraded to version 1.6.0 but that did not work. 1.5.0 works fine for this issue.