search

JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.31k stars 698 forks source link

"TwitterSearchScraper" function very unstable on scraping top trends #978

Closed 4yR1l1k closed 1 year ago

4yR1l1k commented 1 year ago

I've been scraping tweets based on top trends, however some of them are receiving the exact number of tweets that I've addressed inside my code, and also some trends are giving me only 10 or 200 something tweets. As these trends are actually the real trends, there should be much more tweets my code can get for each trend. Do you know why such different numbers are being gathered for top trends? Is there a better way to scrape top trends, such as scraping just like another keyword phrase? Thank you.

JustAnotherArchivist commented 1 year ago

I'm not entirely sure what you're doing exactly. If you're using top tweet search, then that's always been a bit weird (returning only a couple hundred results). In general, the new search endpoint is less reliable and doesn't always return all matches, cf. #886 #891 etc. But that's an issue on Twitter's side.