MartinKBeck / TwitterScraper

Repository containing all files relevant to my basic and advanced tweet scraping articles.
196 stars 117 forks source link

Geocode and since for extracting tweets #9

Closed KimShan1 closed 2 years ago

KimShan1 commented 2 years ago

Hi, I'm trying to extract tweets combining the geocode filter and the since filter but every time I run it I end up having this error: 'Unable to find guest token'. I've run this same search using Tweepy and I do get a lot of tweets but because of the time constraint I'm very interested in making it run with this scraper. Do you know why could this be happening?

for i,tweet in enumerate(sntwitter.TwitterSearchScraper('covid geocode:"34.052235,-118.243683,10km" since:2021-12-24').get_items()):
    if i>maxTweets:
        break
    tweets_list1.append([tweet.url,tweet.date, tweet.id, tweet.content,
                         tweet.user.username,tweet.replyCount,tweet.retweetCount,
                         tweet.likeCount,tweet.quoteCount,tweet.source,tweet.media,
                         tweet.retweetedTweet,tweet.mentionedUsers])
print('Complete')

Also, if I want to append the coordinates to the dataframe or the country/city, what attribute of tweet. should I use?

Thanks a lot!

MartinBeckUT commented 2 years ago

For location you can use tweet.user.location, however since you're already doing a geocode search you know what city you're searching in. I'd just do an apply or set the dataframe column to the city you're searching in.

In regards to the guest token I was running into this problem with the dev version of snscrape. I would uninstall the dev version and install the released version. You can just do a pip uninstall snscrape and pip install snscrape. This resolved the problem on my end.

KimShan1 commented 2 years ago

Thanks a lot! It worked. Regarding the location, I wanted to get the exact tweet's coordinates because I want to create a heat map of sentiment for these tweets. However, I'm not sure how to extract them as the tweet.user.location is not working.

Thanks a lot!