search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.64k stars 2.72k forks source link

TWINT missingimport related to cchardet, cannot build wheel #1407

Open haftnerb opened 2 years ago

haftnerb commented 2 years ago

Issue Template

Please use this template!

Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'twint' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'twint'. Thanks.

Make sure you've checked the following:

Command Ran

py -m pip install cchardet AND py -m pip install twing

Description of Issue

cchardet has not been updated for a long time so it appears to be working for python 2, cannot build wheel

Environment Details

Windows, inside command prompt

Command prompt error details are as follows...

C:\Users>py -m pip install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint Obtaining twint from git+https://github.com/twintproject/twint.git@origin/master#egg=twint Cloning https://github.com/twintproject/twint.git (to revision origin/master) to c:\users\src\twint Running command git clone --filter=blob:none --quiet https://github.com/twintproject/twint.git 'C:\Users\src\twint' WARNING: Did not find branch or tag 'origin/master', assuming revision or ref. Running command git checkout -q origin/master Resolved https://github.com/twintproject/twint.git to commit origin/master Preparing metadata (setup.py) ... done Collecting aiohttp Using cached aiohttp-3.8.1-cp310-cp310-win_amd64.whl (555 kB) Collecting aiodns Using cached aiodns-3.0.0-py3-none-any.whl (5.0 kB) Requirement already satisfied: beautifulsoup4 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from twint) (4.10.0) Collecting cchardet Using cached cchardet-2.1.7.tar.gz (653 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: dataclasses in c:\users\appdata\local\programs\python\python310\lib\site-packages (from twint) (0.6) Collecting elasticsearch Using cached elasticsearch-8.1.3-py3-none-any.whl (373 kB) Requirement already satisfied: pysocks in c:\users\appdata\local\programs\python\python310\lib\site-packages (from twint) (1.7.1) Requirement already satisfied: pandas in c:\users\appdata\local\programs\python\python310\lib\site-packages (from twint) (1.3.5) Collecting aiohttp_socks Using cached aiohttp_socks-0.7.1-py3-none-any.whl (9.3 kB) Collecting schedule Using cached schedule-1.1.0-py2.py3-none-any.whl (10 kB) Collecting geopy Using cached geopy-2.2.0-py3-none-any.whl (118 kB) Requirement already satisfied: fake-useragent in c:\users\appdata\local\programs\python\python310\lib\site-packages (from twint) (0.1.11) Collecting googletransx Using cached googletransx-2.4.2-py3-none-any.whl Collecting pycares>=4.0.0 Using cached pycares-4.1.2-cp310-cp310-win_amd64.whl (83 kB) Requirement already satisfied: attrs>=17.3.0 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from aiohttp->twint) (21.4.0) Collecting multidict<7.0,>=4.5 Using cached multidict-6.0.2-cp310-cp310-win_amd64.whl (27 kB) Collecting frozenlist>=1.1.1 Using cached frozenlist-1.3.0-cp310-cp310-win_amd64.whl (33 kB) Collecting async-timeout<5.0,>=4.0.0a3 Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB) Collecting yarl<2.0,>=1.0 Using cached yarl-1.7.2-cp310-cp310-win_amd64.whl (122 kB) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from aiohttp->twint) (2.0.12) Collecting aiosignal>=1.1.2 Using cached aiosignal-1.2.0-py3-none-any.whl (8.2 kB) Requirement already satisfied: python-socks[asyncio]<3.0.0,>=2.0.0 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from aiohttp_socks->twint) (2.0.3) Requirement already satisfied: soupsieve>1.2 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from beautifulsoup4->twint) (2.3.1) Collecting elastic-transport<9,>=8 Using cached elastic_transport-8.1.2-py3-none-any.whl (59 kB) Requirement already satisfied: geographiclib<2,>=1.49 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from geopy->twint) (1.52) Requirement already satisfied: requests in c:\users\appdata\local\programs\python\python310\lib\site-packages (from googletransx->twint) (2.27.1) Requirement already satisfied: python-dateutil>=2.7.3 in c:\users\appdata\roaming\python\python310\site-packages (from pandas->twint) (2.8.2) Requirement already satisfied: numpy>=1.21.0 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from pandas->twint) (1.21.5) Requirement already satisfied: pytz>=2017.3 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from pandas->twint) (2021.3) Requirement already satisfied: urllib3<2,>=1.26.2 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from elastic-transport<9,>=8->elasticsearch->twint) (1.26.8) Requirement already satisfied: certifi in c:\users\appdata\local\programs\python\python310\lib\site-packages (from elastic-transport<9,>=8->elasticsearch->twint) (2021.10.8) Requirement already satisfied: cffi>=1.5.0 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from pycares>=4.0.0->aiodns->twint) (1.15.0) Requirement already satisfied: six>=1.5 in c:\users\appdata\roaming\python\python310\site-packages (from python-dateutil>=2.7.3->pandas->twint) (1.16.0) Requirement already satisfied: idna>=2.0 in c:\users\appdata\local\programs\python\python310\lib\site-packages (from yarl<2.0,>=1.0->aiohttp->twint) (3.3) Requirement already satisfied: pycparser in c:\users\appdata\local\programs\python\python310\lib\site-packages (from cffi>=1.5.0->pycares>=4.0.0->aiodns->twint) (2.21) Building wheels for collected packages: cchardet Building wheel for cchardet (setup.py) ... error error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [12 lines of output] cythonize: ['src/cchardet\_cchardet.pyx'] running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-3.10 creating build\lib.win-amd64-3.10\cchardet copying src\cchardet\version.py -> build\lib.win-amd64-3.10\cchardet copying src\cchardet__init__.py -> build\lib.win-amd64-3.10\cchardet running build_ext building 'cchardet._cchardet' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for cchardet Running setup.py clean for cchardet Failed to build cchardet Installing collected packages: cchardet, schedule, multidict, geopy, frozenlist, elastic-transport, async-timeout, yarl, pycares, googletransx, elasticsearch, aiosignal, aiohttp, aiodns, aiohttp_socks, twint Running setup.py install for cchardet ... error error: subprocess-exited-with-error

× Running setup.py install for cchardet did not run successfully. │ exit code: 1 ╰─> [14 lines of output] cythonize: ['src/cchardet\_cchardet.pyx'] running install C:\Users\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build\lib.win-amd64-3.10 creating build\lib.win-amd64-3.10\cchardet copying src\cchardet\version.py -> build\lib.win-amd64-3.10\cchardet copying src\cchardet__init__.py -> build\lib.win-amd64-3.10\cchardet running build_ext building 'cchardet._cchardet' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

× Encountered error while trying to install package. ╰─> cchardet

note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure.

WilliamsJack commented 2 years ago

Microsoft Visual C++ 14.0 or greater is required to build cchardet on Windows.

To get it, visit https://visualstudio.microsoft.com/visual-cpp-build-tools/ and download the build tools installer.

Run the installer and select the "Workloads" tab. If you don't see "Workloads", click "Modify" on the already installed Visual Studio Build Tools and then select the tab.

From the list on the left, select "Desktop development with C++".

In the list of individual components on the right, select the following:

You can deselect all others.

image

Then, re-run your Twint installation command! :-)