search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.69k stars 2.71k forks source link

Twint returning exceptions and timing out after --follower list pull #612

Closed PaulWitry closed 4 years ago

PaulWitry commented 4 years ago

Very new to Twint! Employing it for a specific project. (I apologize for my general lack of knowledge and if things are formatted incorrectly)

twint -u harrispolicy -followers

Running this through OSX, verified that my Python is up to date, etc.

The command starts running fine but returns an initial exception and later on has a timeout exception related to calling for this data. The account only features approx. 10k followers. I noted in another issue post that there was a limit to how many it could pull down, but is that relative to follows or overall?

Thanks!

rlleshi commented 4 years ago

You might want to take a look at this issue.

pielco11 commented 4 years ago

Timeout exception means that Twitter stopped giving you data

Also I think that you used --followers and not -followers

IMO, scraping followers/following reached a dead end. I added some timeouts around, but Twitter still works good

PaulWitry commented 4 years ago

Thanks @rlleshi I appreciate the advice! I looked there originally and saw that Pielco11 had said that Twitter was getting more effective at blocking Twint data requests but just wanted an update :)

PaulWitry commented 4 years ago

@pielco11 I had a feeling that was the case, I appreciate it. Is it the timing between each data request that they are blocking off of or the size of the request overall?

(I missed the extra tack on the --followers when making the initial post haha sorry, noob mistake)

tripelix commented 4 years ago

I am tracking a bot army on twitter it currently has over 549 accounts posting spam and phishing links (6584 tweets 2019-12-18) and using --followers or --following I get sporadic: twint.feed:Follow:IndexError on some accounts, from what I can tell there is nothing different just some won't give back any data, this is also true occasionally on tweets. i see them on the platform fine. i would be happy to share account to look at just don't click on any of the phishing links you will get an FBI visit dm me on twitter for data @trip_elix