JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.43k stars 706 forks source link

Twitter user scrapes do not include retweets #8

Closed JustAnotherArchivist closed 4 years ago

JustAnotherArchivist commented 6 years ago

Because of how snscrape discovers tweets (through the search page rather than the user profile, because the latter is limited to 3200 results while the former is not), it can't discover retweets. Or at least I haven't found any way to do that. The search term suggestions I found online are all several years old and no longer work.

I believe this is unfixable, but as a workaround, snscrape could at least retrieve the retweets contained among the user's 3200 most recent tweets by scraping the user profile page if requested. See #5.

JustAnotherArchivist commented 4 years ago

The include:nativeretweets search filter works but only for retweets from the past 7 days. It can be used with snscrape twitter-search 'from:user include:nativeretweets'. This returns the retweet URL (not the URL of the reposted tweet).

According to my research, there is currently no way to find retweets from more than 7 days ago through the search and not even through the API.

Closing this as it's impossible to implement. Scraping the profile page is tracked in #5.