search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.65k stars 2.72k forks source link

[QUESTION] Why does search fail with Feed:noData error? #1023

Open packetrat opened 3 years ago

packetrat commented 3 years ago

Using Docker build 2.1.8., but had same error with latest:

Command Ran

docker-compose run -v $PWD/twint:/opt/app/data twint --since 2020-11-3 --search "#hammer"

Description of Issue

Output: Creating 218_twint_run ... done CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) CRITICAL:root:twint.run:Twint:Feed:Tweets_known_error:Expecting value: line 1 column 1 (char 0) Expecting value: line 1 column 1 (char 0) [x] run.Feed [!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it! rodan:2.1.8 admin$

Environment Details

Using docker images (Ubuntu) 2.1.8 and Latest

WaWaKing commented 3 years ago

Twitter updated the protocol! The api that uesed by twint isn't working!

haccer commented 3 years ago

Yeah, looks like Twitter changed the way Twitter search works. It looks like we'll have to modify the tool to make requests to https://api.twitter.com/2/search/adaptive.json instead.

azraelmt commented 3 years ago

this command worked for me in order to make it work

pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

also, updating the aiohttp_socks library

pip3 install --upgrade aiohttp_socks

himanshudabas commented 3 years ago

@packetrat you must be installing from pypi using pip install twint.

Newer version of twint hasn't been released on pypi yet. After twitter migrated to the newer endpoints (v2), older version of twint stopped working (as it was designed to work with the older twitter website).

You need to install directly from GitHub using the command : 

pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

like mentioned above by @azraelmt It contains the latest implementation of twint. Though, I must warn you that It can be unstable and can contain several bugs as it's been constantly modified.

irwandareza commented 3 years ago

i was try doing like @himanshudabas and @azraelmt said, but the output still same as @packetrat. and while try to scrape data with CLI basic "twint -u username", the output is command not found.

any wrong with my step?

himanshudabas commented 3 years ago

@irwandareza Have you tried uninstalling your previous twint installation and then reinstall it using the command provided above? I think that should fix the issue for you unless twitter changed something yet again. If you still experience the same issue after following these steps, then provide me some extra information like your Python version, Environment, Machine etc.

irwandareza commented 3 years ago

Ahh i see thankyou @himanshudabas, i'll try it soon

irwandareza commented 3 years ago

@himanshudabas can i have your email or something for asking some question about this?