search

JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.33k stars 699 forks source link

TwitterProfileScraper doesn't scrape tweets more than ~3200 #194

Closed sufyanhamid closed 3 years ago

sufyanhamid commented 3 years ago

hi @JustAnotherArchivist I was try to fetch the data of some twitter user profiles than I analyze that TwitterProfileScraper function doesn't scrape ~3200 tweets of the profile although the profile have thousands of tweets. E.g: @DonaldJTrumpJr have 41k tweets in his account but the script return just 3214 tweets. So please help me to fetch all the available data from this profile. Thanks :)

JustAnotherArchivist commented 3 years ago

That is a limitation by Twitter on profile pages. You probably want to use the search-based TwitterUserScraper instead. See https://github.com/JustAnotherArchivist/snscrape/issues/85#issuecomment-698998263 for more details on the difference between TwitterUserScraper (aka twitter-user) and TwitterProfileScraper (aka twitter-profile).