JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.31k stars 698 forks source link

Cannot get tweets #1029

Closed gridikono closed 9 months ago

gridikono commented 9 months ago

Describe the bug

I am trying to extract tweets for a given profile, but I got the error below: here is my code:

def scrape_tweets(self, username):
        for tweet in sntwitter.TwitterSearchScraper(f'from:{username}').get_items():
            dal = DAL()
            dal.insert_tweet(tweet=tweet)
            self.scrape_replies(tweet=tweet)

It worked until at the mid of April 2023. Below is my error that I got:

Error retrieving https://api.twitter.com/1.1/guest/activate.json: non-200 response (401)
    return self._request('POST', *args, **kwargs)
  File "C:\Users\Gridi\Desktop\nlp\osfa2022\Source code\ASSA.Scraper.Twitter\snscrape\base.py", line 247, in _request
    raise ScraperException(msg)
snscrape.base.ScraperException: 4 requests to https://api.twitter.com/1.1/guest/activate.json failed, giving up.

I know that twitter has applied restriction in its public APIs, but there is any fix for it ?

How to reproduce

def scrape_tweets(self, username):
        for tweet in sntwitter.TwitterSearchScraper(f'from:{username}').get_items():
            dal = DAL()
            dal.insert_tweet(tweet=tweet)
            self.scrape_replies(tweet=tweet)

Expected behaviour

To get the tweets

Screenshots and recordings

No response

Operating system

Windows 10

Python version: output of python3 --version

3.9

snscrape version: output of snscrape --version

latest version installed from pip

Scraper

TwitterSearchScraper

How are you using snscrape?

Module (import snscrape.modules.something in Python code)

Backtrace

No response

Log output

No response

Dump of locals

No response

Additional context

No response

MazenTayseer commented 9 months ago

996