search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.7k stars 2.72k forks source link

[!] No more data! Scraping will stop now. #1106

Open noelmathews opened 3 years ago

noelmathews commented 3 years ago

Command Ran

twint -s "AUD/USD" --since 2016-01-01 --json -o AUDUSD.json

Description of Issue

Hi, I managed to get around 296946 tweets from the above command from Jan 2021 till August 2019 then got the prompt [!] No more data! Scraping will stop now. Surely there are more tweets beyond this point of date as well. Does anyone know what's causing the script to stop?

Environment Details

Windows | Command Prompt

mayankrichu commented 3 years ago

If it says no more data then there won't be more than what you already fetched.

noelmathews commented 3 years ago

Hey, thanks for getting back.

I had also ran the command twint -s "AUD/USD" --year 2019 to fetch all tweets from 2018 and it scrapped more data which means there were more tweets. I scrapped all the way heading back to 2015. I think the program stops when it's not able to find tweets around that month perhaps? Not sure whats happening..

mayankrichu commented 3 years ago

Hi, yeah that could happen, maybe you can write a script with predefined month and year. You can fetch the data using for loop. If there is any month which doesn't contain the data you can just raise an exception/or just continue with the loop.

noelmathews commented 3 years ago

Yeahh, that does sound like a possibility to try to get the script working. Thanks!

KamelGaanoun commented 3 years ago

Hi, Can I ask again the same question. But for me it stops and says no more data, then I put it in a loop like this : 

st = datetime.strptime('2021-09-13','%Y-%m-%d')
for i in range(356):
    c= twint.Config()
    c.Search= my keywords here

    stdt=st-timedelta(days=i+1)
    c.Until=stdt.strftime('%Y-%m-%d')
    c.Store_csv= True
    #c.Hide_output= True
    c.Filter_retweets= True
    c.Output="C:\dabaList.csv"
    c.Resume="C:\Resume_dabaList.csv"
    twint.run.Search(c)`

It scapes but with a lot of "No more data! Scraping will stop now" messages.

What I did is going to Twitter advanced research and there is existing tweets on the days twint says no more data. ANd even for days I get data, not all tweets apearing in advanced search are there from twint.

Can you help with this please?