search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.8k stars 2.72k forks source link

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting #587

Closed stormstricker closed 4 years ago

stormstricker commented 4 years ago

I keep getting this error! It has something to do with Twitter ratelimiting the scraper, I presume? Was it really that hard to build in a queue of requests and only allow N requests per minute?

pielco11 commented 4 years ago

Maybe you add your query or at least describe what you're trying to achieve, please?

azraelmt commented 4 years ago

On a similar note, i keep getting this error on my linux/debian 10 distribution when im using twint to collect tweets. In contrast, this error have never appeared in my Windows 10 machine, even when looking for the same hashtag/user/keyword/etc. Is there a reason why this keep happening?

p.s. excuse the broken english, not a native speaker

pielco11 commented 4 years ago

I think that's related to #567

o7n commented 4 years ago

When fixating the user agent to

Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2224.3 Safari/537.36

or

Mozilla/5.0 (Windows NT 6.4; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36

this error seems to go away.

This looks related to https://github.com/twintproject/twint/issues/426 which is essentially the same error (Currently it prints noData without a space before "Expecting", back then it said "json.decoder.JSONDecodeError: Expecting"). That issue was also fixed by changing the user agent.

pielco11 commented 4 years ago

@o7n thank you for sharing your findings, I'll give it a run asap!

AldebaranNapoli commented 4 years ago

I tried 4 installation with

Debian Mint e Kali

always the same error of critical root

I did installations with

Pip3 Python3 build and install setup.py Pipenv

On the older isntallations no problems, it works. Why this?

I tried with both user and root commands

pielco11 commented 4 years ago

How much old?

AldebaranNapoli commented 4 years ago

september!

Works on every virtual machine i installed, on my native debian laptop

But not in the last 4 installations i did saturday. I format my pc many times and the error is always present.

sup97 commented 4 years ago

I have the same problem. I ran twint -s "children's home volunteer" --year 2020 -o -children_home_volunteer.csv --csv It stopped once in 2018-09-27 and again in 2017-02-25. It seems to be a problem with a certain number of blank pages?

Screen Shot 2019-12-16 at 9 51 25 AM
pielco11 commented 4 years ago

604