twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

MIT License

15.66k stars 2.72k forks source link

twint -u username return small number of tweets #566

Closed Arwa200 closed 4 years ago

Arwa200 commented 4 years ago

264 Initial Check

twint -u username return small number of tweets [QUESTION] how to get all tweets

Make sure you've checked the following:

[] Python version is 3.6;
[] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
[] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran

twint -u username

Environment Details

Using mac terminal

Qriist commented 4 years ago

try the --profile-full option

or try formatting a manual search: -s "from:username"

see what you get

pielco11 commented 4 years ago

I also suggest you to do your search via browser and see how far back it goes

Arwa200 commented 4 years ago

--profile-full option: it's gather approximately 3000 tweets the user had 52 K of tweets I'm trying now with -s

Arwa200 commented 4 years ago

-s return same small number.. also tried with dates but its not working, still give me newest tweets furthermore its small number

pielco11 commented 4 years ago

What do you see if you scroll manually with your browser?

Arwa200 commented 4 years ago

sorry, lost the track.. why do i have to scroll the browser?

pielco11 commented 4 years ago

So that we know if it's Twitter is acting in a strange way, or just because "that's it" the situation

pielco11 commented 4 years ago

Scrolling down with the browser is the same thing as running Twint until it stops by itself, we'd be cool to know if the results are the same

Arwa200 commented 4 years ago

just want to update you about an automatic scrolling for both tweets and likes from twitter.com and mobile.twitter.com and goth the following: - tweets: result in terminal: it returns until 16/07/2017
twitter.com result from auto scrolling: returns until 26/10/2017
mobile.twitter.com: returns until 26/10/2017

likes: result in terminal: it returns until 29/04/2013 twitter.com result from auto scrolling: returns until Mar 2012
mobile.twitter.com: returns until November 2012 and both mobile and normal web are crashed while it is not while running auto scrolling in tweets

Screen Shot 2019-11-04 at 7 04 54 PM Screen Shot 2019-11-04 at 8 39 35 PM

pielco11 commented 4 years ago

tweets

So you got more results than via browser, awesome

likes

you should consider that scraping followers/following/favorites is a bit more limited than just searching for tweets. Those are not intended to get all the awaited results

Arwa200 commented 4 years ago

yes, I got the idea.. Do you think would be possible to get get all the awaited results somehow I mean by scraping not "Paid API"? or is even Paid API provide this feature or its limited?

pielco11 commented 4 years ago

I don't know, there limits with every tool (more or less)

Arwa200 commented 4 years ago

Thank you