Does Twint respect robots.txt [QUESTION]

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

MIT License

15.77k stars 2.73k forks source link

Does Twint respect robots.txt [QUESTION] #898

Open ChristianZX opened 4 years ago

ChristianZX commented 4 years ago

Hi everybody, first of all I want to say what a great tool Twint is. While scraping is is respecting the robots.txt?

If not, would some features be in accordance in with robots.txt? For example mere tweet scraping.

Regards, Christian

pielco11 commented 4 years ago

Nope