search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.75k stars 2.72k forks source link

Twitter does not return more data, scrape stops here. #636

Closed deepalidhaka closed 4 years ago

deepalidhaka commented 4 years ago

Command Ran

c = twint.Config() c.Username = username c.Output=c.Username.csv c.Limit=1000 c.Lang='en' c.Store_csv=True c.Retweets=True twint.run.Profile(c)

Description of Issue

I AM GETTING THIS ON THE IPYTHON CONSOLE- [!] Twitter does not return more data, scrape stops here. [!] Twitter does not return more data, scrape stops here.

But when i run "twint -u username" through command line, i get the tweets of the same user. I couldn't understand what wrong i am doing in code.

Environment Details

Using Windows 8.1 pro, Running this in Anaconda 3, spyder.

pielco11 commented 4 years ago

Idk what's wrong, you specified the same username in both cases. It's expected to get the same tweets

deepalidhaka commented 4 years ago

I forgot to mention that ,I am running this code in a for loop to fetch tweets of multiple users. loop runs n time for n number of users.

pielco11 commented 4 years ago

twint -u username uses twint.run.Search scraping function, not twint.run.Profile

The difference is that the first uses the Twitter Advanced Search, the second scrapes the user's timeline

deepalidhaka commented 4 years ago

Thanks a lot for highlighting the difference, I have changed twint.run.Profile to twint.run.Search in the above code and I have not got that issue again.

shunyamaya commented 4 years ago

Hello,

Thank you for developing the package. I'm getting the same error and I do want to scrape a given user's timeline.

For example with this code sample:

c = twint.Config()
c.Username = "AOC"
c.Retweets = True

tweets = twint.run.Profile(c)

I get 

[!] Twitter does not return more data, scrape stops here.

Do you have any recommendations?

pielco11 commented 4 years ago

@Shooony how many tweets did you get? That error message is expected since Twitter stops returning data when you get ~3200 tweets (value might vary)

shunyamaya commented 4 years ago

@pielco11 Ah I see. I ran it once which gave me around 3000 tweets, but the second time it gave me none (I tried about an hour later).

Is the 3200 limit per profile, or just in general?

pielco11 commented 4 years ago

3200 is the limit for profile and favorites. 3000 is an expected result, compatible with the 3200 limit (proven by experience)