search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.7k stars 2.71k forks source link

zsh: command not found: twint #1071

Closed ghost closed 3 years ago

ghost commented 3 years ago

Command Ran

twint -s "python" --since 2020-12-10

Description of Issue

zsh: command not found: twint**

Environment Details

Have same issue in:

Arch Linux: Terminal VMWare (Kali): Terminal

rachmadaniHaryono commented 3 years ago

https://github.com/rachmadaniHaryono/twint/tree/feature/main

can you try that and run twint help command like this python -m twint --help?

ghost commented 3 years ago

https://github.com/rachmadaniHaryono/twint/tree/feature/main

can you try that and run twint help command like this python -m twint --help?

I get these error, VMWare - Kali: /usr/bin/python: No module named twint

WSL-Ubuntu: /usr/bin/python3: No module named twint.main; 'twint' is a package and cannot be directly executed

rachmadaniHaryono commented 3 years ago

can you install it with pip3 install --user https://github.com/rachmadaniHaryono/twint/archive/feature/main.zip

after install, try pip3 show -f twint

example of the output

pip show output ``` Name: twint Version: 2.1.21 Summary: An advanced Twitter scraping & OSINT tool. Home-page: https://github.com/twintproject/twint Author: Cody Zacharias Author-email: codyzacharias@pm.me License: MIT Location: /home/r3r/envs/temp/lib/python3.8/site-packages Requires: googletransx, aiodns, aiohttp-socks, pandas, schedule, beautifulsoup4, pysocks, geopy, cchardet, fake-useragent, dataclasses, aiohttp, elasticsearch Required-by: Files: ../../../bin/twint twint-2.1.21.dist-info/INSTALLER twint-2.1.21.dist-info/LICENSE twint-2.1.21.dist-info/METADATA twint-2.1.21.dist-info/RECORD twint-2.1.21.dist-info/REQUESTED twint-2.1.21.dist-info/WHEEL twint-2.1.21.dist-info/dependency_links.txt twint-2.1.21.dist-info/direct_url.json twint-2.1.21.dist-info/entry_points.txt twint-2.1.21.dist-info/top_level.txt twint/__init__.py twint/__main__.py twint/__pycache__/__init__.cpython-38.pyc twint/__pycache__/__version__.cpython-38.pyc twint/__pycache__/cli.cpython-38.pyc twint/__pycache__/config.cpython-38.pyc twint/__pycache__/datelock.cpython-38.pyc twint/__pycache__/feed.cpython-38.pyc twint/__pycache__/format.cpython-38.pyc twint/__pycache__/get.cpython-38.pyc twint/__pycache__/output.cpython-38.pyc twint/__pycache__/run.cpython-38.pyc twint/__pycache__/token.cpython-38.pyc twint/__pycache__/tweet.cpython-38.pyc twint/__pycache__/url.cpython-38.pyc twint/__pycache__/user.cpython-38.pyc twint/__pycache__/verbose.cpython-38.pyc twint/__version__.py twint/cli.py twint/config.py twint/datelock.py twint/feed.py twint/format.py twint/get.py twint/output.py twint/run.py twint/storage/__init__.py twint/storage/__pycache__/__init__.cpython-38.pyc twint/storage/__pycache__/db.cpython-38.pyc twint/storage/__pycache__/elasticsearch.cpython-38.pyc twint/storage/__pycache__/panda.cpython-38.pyc twint/storage/__pycache__/write.cpython-38.pyc twint/storage/__pycache__/write_meta.cpython-38.pyc twint/storage/db.py twint/storage/elasticsearch.py twint/storage/panda.py twint/storage/write.py twint/storage/write_meta.py twint/token.py twint/tweet.py twint/url.py twint/user.py twint/verbose.py ```

notice that there is twint/__main__.py on list of files

can you try that and run twint help command like this python -m twint --help?

kali /usr/bin/python: No module named twint

it seems your env have python3 instead python so i will change the instruction based on python3 from now on

ghost commented 3 years ago
In WSL-Ubuntu: Name: twint Version: 2.1.21 Summary: An advanced Twitter scraping & OSINT tool. Home-page: [https://github.com/twintproject/twint](https://github.com/twintproject/twint) Author: Cody Zacharias Author-email: [codyzacharias@pm.me](mailto:codyzacharias@pm.me) License: MIT Location: /home/neo/.local/lib/python3.8/site-packages Requires: aiohttp-socks, pysocks, googletransx, pandas, aiohttp, dataclasses, schedule, elasticsearch, fake-useragent, beautifulsoup4, cchardet, aiodns, geopy Required-by: Files: ../../../bin/twint twint-2.1.21.dist-info/INSTALLER twint-2.1.21.dist-info/LICENSE twint-2.1.21.dist-info/METADATA twint-2.1.21.dist-info/RECORD twint-2.1.21.dist-info/WHEEL twint-2.1.21.dist-info/dependency\_links.txt twint-2.1.21.dist-info/entry\_points.txt twint-2.1.21.dist-info/top\_level.txt twint/\_\_init\_\_.py twint/\_\_main\_\_.py twint/\_\_pycache\_\_/\_\_init\_\_.cpython-38.pyc twint/\_\_pycache\_\_/\_\_main\_\_.cpython-38.pyc twint/\_\_pycache\_\_/\_\_version\_\_.cpython-38.pyc twint/\_\_pycache\_\_/cli.cpython-38.pyc twint/\_\_pycache\_\_/config.cpython-38.pyc twint/\_\_pycache\_\_/datelock.cpython-38.pyc twint/\_\_pycache\_\_/feed.cpython-38.pyc twint/\_\_pycache\_\_/format.cpython-38.pyc twint/\_\_pycache\_\_/get.cpython-38.pyc twint/\_\_pycache\_\_/output.cpython-38.pyc twint/\_\_pycache\_\_/run.cpython-38.pyc twint/\_\_pycache\_\_/token.cpython-38.pyc twint/\_\_pycache\_\_/tweet.cpython-38.pyc twint/\_\_pycache\_\_/url.cpython-38.pyc twint/\_\_pycache\_\_/user.cpython-38.pyc twint/\_\_pycache\_\_/verbose.cpython-38.pyc twint/\_\_version\_\_.py twint/cli.py twint/config.py twint/datelock.py twint/feed.py twint/format.py twint/get.py twint/output.py twint/run.py twint/storage/\_\_init\_\_.py twint/storage/\_\_pycache\_\_/\_\_init\_\_.cpython-38.pyc twint/storage/\_\_pycache\_\_/db.cpython-38.pyc twint/storage/\_\_pycache\_\_/elasticsearch.cpython-38.pyc twint/storage/\_\_pycache\_\_/panda.cpython-38.pyc twint/storage/\_\_pycache\_\_/write.cpython-38.pyc twint/storage/\_\_pycache\_\_/write\_meta.cpython-38.pyc twint/storage/db.py twint/storage/elasticsearch.py twint/storage/panda.py twint/storage/write.py twint/storage/write\_meta.py twint/token.py twint/tweet.py twint/url.py twint/user.py twint/verbose.py
In VMWare-Kali: Name: twint Version: 2.1.21 Summary: An advanced Twitter scraping & OSINT tool. Home-page: https://github.com/twintproject/twint Author: Cody Zacharias Author-email: codyzacharias@pm.me License: MIT Location: /home/kali/src/twint Requires: aiodns, aiohttp, aiohttp-socks, beautifulsoup4, cchardet, dataclasses, elasticsearch, fake-useragent, geopy, googletransx, pandas, pysocks, schedule Required-by: Files: Cannot locate installed-files.txt
rachmadaniHaryono commented 3 years ago

check your PATH env var on wsl ubuntu, for example with this command echo $PATH | tr -s ":" "\n" | sort i got following result

path ``` /bin /home/r3r/.cargo/bin /home/r3r/.fzf/bin /home/r3r/.gem/ruby/2.7.0/bin /home/r3r/.go/bin /home/r3r/.local/bin /home/r3r/.npm-global/bin /sbin /snap/bin /usr/bin /usr/games /usr/local/bin /usr/local/games /usr/local/go/bin /usr/local/sbin /usr/sbin ```

notice there is /home/r3r/.local/bin

you can run twint with /home/neo/.local/bin/twint

for kali, try list files on installed location with ls /home/kali/src/twint and see what is in there

ghost commented 3 years ago

for echo $PATH | tr -s ":" "\n" | sort

path: /bin /mnt/c/MinGW/bin /mnt/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL /mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common /mnt/c/Program Files (x86)/VMware/VMware Workstation/bin/ /mnt/c/Program Files/Git/cmd /mnt/c/Program Files/Intel/Intel(R) Management Engine Components/DAL /mnt/c/Program Files/JetBrains/IntelliJ IDEA Community Edition 2020.2/bin /mnt/c/Program Files/Microsoft VS Code/bin /mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR /mnt/c/Users/r46j/AppData/Local/Microsoft/WindowsApps /mnt/c/Users/r46j/AppData/Local/Programs/Python/Python39/ /mnt/c/Users/r46j/AppData/Local/Programs/Python/Python39/Scripts/ /mnt/c/Windows /mnt/c/Windows/System32/OpenSSH/ /mnt/c/Windows/System32/Wbem /mnt/c/Windows/System32/WindowsPowerShell/v1.0/ /mnt/c/Windows/system32 /mnt/c/adb_fastboot/platform-tools_r30.0.5-windows/platform-tools /sbin /usr/bin /usr/games /usr/local/bin /usr/local/games /usr/local/sbin /usr/sbin

for /home/neo/.local/bin/twint I got this output: [-] Error: Please use at least -u, -s, -g or --near.

On Kali, for ls /home/kali/src/twint Output: automate.py elasticsearch LICENSE MANIFEST.in README.md requirements.txt setup.py test.py twint twint.egg-info

rachmadaniHaryono commented 3 years ago

wsl ubuntu

as expected there is no /home/neo/.local/bin.

for example i put these lines on .zshrc

export PYTHON_BIN_PATH="$(python3 -m site --user-base)/bin"
export PATH="$PATH:$PYTHON_BIN_PATH"

which will add python bin path to PATH.

for /home/neo/.local/bin/twint I got this output: [-] Error: Please use at least -u, -s, -g or --near.

this already meant you can already run twint. run /home/neo/.local/bin/twint --help will print help section.

you can put alias to this or change PATH as stated above

kali linux

kali is a weird one. i got the same pip show result only if i install from source after clone the repo

files on the location mean it is either the entire repo or files from the package

the next step maybe check python bin path ls $(python3 -m site --user-base)/bin if there is twint and also check PATH env var just as above

ghost commented 3 years ago

wsl ubuntu

as expected there is no /home/neo/.local/bin.

for example i put these lines on .zshrc

export PYTHON_BIN_PATH="$(python3 -m site --user-base)/bin" export PATH="$PATH:$PYTHON_BIN_PATH"

Thanks, now its working in WSL-Ubuntu

In Kali, ls $(python3 -m site --user-base)/bin twint is listed.

rachmadaniHaryono commented 3 years ago

twint is listed.

maybe it is the same problem with PATH env var try doing the same check just like you did on ubuntu and add the same config to .zshrc on kali

ghost commented 3 years ago

twint is listed.

maybe it is the same problem with PATH env var try doing the same check just like you did on ubuntu and add the same config to .zshrc on kali

I already tried it, it's not working.

rachmadaniHaryono commented 3 years ago

on kali

ghost commented 3 years ago

echo $PATH | tr -s ":" "\n" | sort

output: /bin /sbin /usr/bin /usr/games /usr/local/bin /usr/local/games /usr/local/sbin /usr/sbin

python3 -m site --user-base output: /home/kali/.local

rachmadaniHaryono commented 3 years ago

I already tried it, it's not working.

based on that and your last post, it mean the path is not changed

you can try /home/kali/.local/bin/twint --help to try to print help section

other thing you can do is to set up alias, for example alias twint=/home/kali/.local/bin/twint

ghost commented 3 years ago

/home/kali/.local/bin/twint --help

output: :219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject :219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject :219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject :219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject usage: python3 twint [options] TWINT - An Advanced Twitter Scraping Tool. optional arguments: -h, --help show this help message and exit -u USERNAME, --username USERNAME User's Tweets you want to scrape. -s SEARCH, --search SEARCH Search for Tweets containing this word or phrase. -g GEO, --geo GEO Search for geocoded Tweets. --near NEAR Near a specified city. --location Show user's location (Experimental). -l LANG, --lang LANG Search for Tweets in a specific language. -o OUTPUT, --output OUTPUT Save output to a file. -es ELASTICSEARCH, --elasticsearch ELASTICSEARCH Index to Elasticsearch. --year YEAR Filter Tweets before specified year. --since DATE Filter Tweets sent since date (Example: "2017-12-27 20:30:15" or 2017-12-27). --until DATE Filter Tweets sent until date (Example: "2017-12-27 20:30:15" or 2017-12-27). --email Filter Tweets that might have email addresses --phone Filter Tweets that might have phone numbers --verified Display Tweets only from verified users (Use with -s). --csv Write as .csv file. --tabs Separate CSV fields with tab characters, not commas. --json Write as .json file --hashtags Output hashtags in seperate column. --cashtags Output cashtags in seperate column. --userid USERID Twitter user id. --limit LIMIT Number of Tweets to pull (Increments of 20). --count Display number of Tweets scraped at the end of session. --stats Show number of replies, retweets, and likes. -db DATABASE, --database DATABASE Store Tweets in a sqlite3 database. --to USERNAME Search Tweets to a user. --all USERNAME Search all Tweets associated with a user. --followers Scrape a person's followers. --following Scrape a person's follows --favorites Scrape Tweets a user has liked. --proxy-type PROXY_TYPE Socks5, HTTP, etc. --proxy-host PROXY_HOST Proxy hostname or IP. --proxy-port PROXY_PORT The port of the proxy server. --tor-control-port TOR_CONTROL_PORT If proxy-host is set to tor, this is the control port --tor-control-password TOR_CONTROL_PASSWORD If proxy-host is set to tor, this is the password for the control port --essid [ESSID] Elasticsearch Session ID, use this to differentiate scraping sessions. --userlist USERLIST Userlist from list or file. --retweets Include user's Retweets (Warning: limited). --format FORMAT Custom output format (See wiki for details). --user-full Collect all user information (Use with followers or following only). -tl, --timeline Collects every tweet from a User's Timeline. (Tweets, RTs & Replies) --translate Get tweets translated by Google Translate. --translate-dest TRANSLATE_DEST Translate tweet to language (ISO2). --store-pandas STORE_PANDAS Save Tweets in a DataFrame (Pandas) file. --pandas-type [PANDAS_TYPE] Specify HDF5 or Pickle (HDF5 as default) -it [INDEX_TWEETS], --index-tweets [INDEX_TWEETS] Custom Elasticsearch Index name for Tweets. -if [INDEX_FOLLOW], --index-follow [INDEX_FOLLOW] Custom Elasticsearch Index name for Follows. -iu [INDEX_USERS], --index-users [INDEX_USERS] Custom Elasticsearch Index name for Users. --debug Store information in debug logs --resume TWEET_ID Resume from Tweet ID. --videos Display only Tweets with videos. --images Display only Tweets with images. --media Display Tweets with only images or videos. --replies Display replies to a subject. -pc PANDAS_CLEAN, --pandas-clean PANDAS_CLEAN Automatically clean Pandas dataframe at every scrape. -cq CUSTOM_QUERY, --custom-query CUSTOM_QUERY Custom search query. -pt, --popular-tweets Scrape popular tweets instead of recent ones. -sc, --skip-certs Skip certs verification, useful for SSC. -ho, --hide-output Hide output, no tweets will be displayed. -nr, --native-retweets Filter the results for retweets only. --min-likes MIN_LIKES Filter the tweets by minimum number of likes. --min-retweets MIN_RETWEETS Filter the tweets by minimum number of retweets. --min-replies MIN_REPLIES Filter the tweets by minimum number of replies. --links LINKS Include or exclude tweets containing one o more links. If not specified you will get both tweets that might contain links or not. --source SOURCE Filter the tweets for specific source client. --members-list MEMBERS_LIST Filter the tweets sent by users in a given list. -fr, --filter-retweets Exclude retweets from the results. --backoff-exponent BACKOFF_EXPONENT Specify a exponent for the polynomial backoff in case of errors. --min-wait-time MIN_WAIT_TIME specifiy a minimum wait time in case of scraping limit error. This value will be adjusted by twint if the value provided does not satisfy the limits constraints

Still same error in Kali, after add the alias twint=/home/kali/.local/bin/twint

twint -s "python" --since 2020-12-10 Error: zsh: command not found: twint

ghost commented 3 years ago

It is working in WSL-Ubuntu, that's enough for now, I'm still learning Linux & Python. Knew about twint from the recent video by @cybermentor

Thank you.

Edit: When I install Arch again, I'll add same .zshrc config & will check it.

rachmadaniHaryono commented 3 years ago

sorry that it still doesn't work on kali

last word after issue closed

/home/kali/.local/bin/twint --help actually working but i have no idea what happen with the error

i also have no idea why alias twint=/home/kali/.local/bin/twint not working. if it work, running which twint should give result twint: aliased to /home/kali/.local/bin/twint

ghost commented 3 years ago

Today Installed Arch, after adding export PYTHON_BIN_PATH="$(python3 -m site --user-base)/bin" export PATH="$PATH:$PYTHON_BIN_PATH" to zshrc like Ubuntu-WSL, twint is working.

Thank you