search

JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.39k stars 702 forks source link

Support for top tweets on twitter-search #109

Closed NowBothWhiteAndRed closed 3 years ago

NowBothWhiteAndRed commented 3 years ago

Hi,

Is there any way to only get back top tweets from snscraper? Presently it looks like it's getting all of them.

Thanks!

JustAnotherArchivist commented 3 years ago

This isn't currently supported but probably shouldn't be too difficult to implement in principle. I'll have to see whether it will work well though. It will definitely break --since because the posts are returned in 'random' order (chronologically speaking).

Fatima-Haouari commented 3 years ago

This isn't currently supported but probably shouldn't be too difficult to implement in principle. I'll have to see whether it will work well though. It will definitely break --since because the posts are returned in 'random' order (chronologically speaking).

Hi, I am interested in getting the top tweets. I modified the code as follows: 1) Removed 'f':'live' in line 405: super().init(baseUrl = 'https://twitter.com/search?' + urllib.parse.urlencode({'lang': 'en', 'q': query, 'src': 'spelling_expansion_revert_click'}), **kwargs) 2) commented line 443 'tweet_search_mode': 'live' Please advice if what I am doing it the right way because I am getting many duplicates for the same query. Am I really getting top tweets?

Fatima-Haouari commented 3 years ago

The issue with my updated code is that it keeps collecting tweets without an end. Any idea why is that?

JustAnotherArchivist commented 3 years ago

I have not looked yet into what changes are needed, and I can't provide support for modified versions of snscrape. I do intend to eventually implement this, but I'm currently busy with other things.