search

pgh-public-meetings / city-scrapers-pitt

Pittsburgh City Scrapers: sourcing public meetings in Pittsburgh
https://pgh-public-meetings.github.io/events/
MIT License
19 stars 66 forks source link

Twisted Error using 'pipenv install twisted #36

Closed winrg closed 4 years ago

winrg commented 5 years ago

Upon trying to run scrapy, I got an error. reading through it, I discovered it required twisted. I installed it using "sudo pipenv install twisted", I got this terminal out. I am doing this on a chromebook using the native debian VM.

MartinSeibert commented 5 years ago

I ha the same problem in a ubuntu 16.04 crouton on a chromebook. This stackoverflow solution worked for me: https://stackoverflow.com/questions/31457009/error-installing-twisted-for-python

I had to run the following before installing twisted: sudo apt-get install python3-dev

jeslevine commented 5 years ago

Install .whl file from here and make sure the c3x version of python matches https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted

bonfirefan commented 5 years ago

  1. Install C++ Build Tools from here (only select build tools, ~3GB): https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2017

  2. Download wheel files from the link jess posted
    Make sure the filename matches your version of python in the (27, 33, 37) etc. (So if you have Python 3.6 and Windows 64bit, install Twisted‑19.7.0‑cp36‑cp36m‑win_amd64.whl )

  3. Run pip install [wheel.whl] where you downloaded the wheel

biniona commented 4 years ago

I am having a twisted bug on OSX 10.14.6. After entering my forked directory and running pipenv install --dev --ignore-pipfile --three . I try to run scrapy genspider chi_housing "Chicago Housing Authority" http://www.thecha.org and get the error ModuleNotFoundError: No module named 'twisted'. I tried to run

pip install twisted and pip3 install twisted outside of pipenv shell

MESSAGE : alekbinsmacbook:~ AlekB$ pip3 install twisted Requirement already satisfied: twisted in ./miniconda3/lib/python3.6/site-packages (19.2.1) Requirement already satisfied: hyperlink>=17.1.1 in ./miniconda3/lib/python3.6/site-packages (from twisted) (19.0.0) Requirement already satisfied: attrs>=17.4.0 in ./miniconda3/lib/python3.6/site-packages (from twisted) (19.1.0) Requirement already satisfied: Automat>=0.3.0 in ./miniconda3/lib/python3.6/site-packages (from twisted) (0.7.0) Requirement already satisfied: incremental>=16.10.1 in ./miniconda3/lib/python3.6/site-packages (from twisted) (17.5.0) Requirement already satisfied: PyHamcrest>=1.9.0 in ./miniconda3/lib/python3.6/site-packages (from twisted) (1.9.0) Requirement already satisfied: constantly>=15.1 in ./miniconda3/lib/python3.6/site-packages (from twisted) (15.1.0) Requirement already satisfied: zope.interface>=4.4.2 in ./miniconda3/lib/python3.6/site-packages (from twisted) (4.6.0) Requirement already satisfied: idna>=2.5 in ./miniconda3/lib/python3.6/site-packages (from hyperlink>=17.1.1->twisted) (2.6) Requirement already satisfied: six in ./miniconda3/lib/python3.6/site-packages (from Automat>=0.3.0->twisted) (1.12.0) Requirement already satisfied: setuptools in ./miniconda3/lib/python3.6/site-packages (from PyHamcrest>=1.9.0->twisted) (41.0.1)

and pipenv install twisted before going into pipenv.

All seemed to be successful but the error is still happening.

Any ideas what I should try?

Sections of error message when running pip install twisted when in pipenv shell (full error message is unreasonably long)

ERROR: Command errored out with exit status 1: command: /Users/AlekB/.virtualenvs/city-scrapers-pitt-yoEmy647/bin/python3.7m -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/xf/1f4dfks55slcjxl6s7b5dty40000gn/T/pip-install-n1jkz4xl/twisted/setup.py'"'"'; __file__='"'"'/private/var/folders/xf/1f4dfks55slcjxl6s7b5dty40000gn/T/pip-install-n1jkz4xl/twisted/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/xf/1f4dfks55slcjxl6s7b5dty40000gn/T/pip-wheel-y37x3pla --python-tag cp37 cwd: /private/var/folders/xf/1f4dfks55slcjxl6s7b5dty40000gn/T/pip-install-n1jkz4xl/twisted/

...

`running build_ext building 'twisted.test.raiser' extension creating build/temp.macosx-10.14-x86_64-3.7 creating build/temp.macosx-10.14-x86_64-3.7/src creating build/temp.macosx-10.14-x86_64-3.7/src/twisted creating build/temp.macosx-10.14-x86_64-3.7/src/twisted/test /usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/twisted/test/raiser.c -o build/temp.macosx-10.14-x86_64-3.7/src/twisted/test/raiser.o clang: warning: no such sysroot directory: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk' [-Wmissing-sysroot] In file included from src/twisted/test/raiser.c:4: /opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m/Python.h:25:10: fatal error: 'stdio.h' file not found

include 

         ^~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1
----------------------------------------

ERROR: Command errored out with exit status 1: /Users/AlekB/.virtualenvs/city-scrapers-pitt-yoEmy647/bin/python3.7m -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/xf/1f4dfks55slcjxl6s7b5dty40000gn/T/pip-install-n1jkz4xl/twisted/setup.py'"'"'; file='"'"'/private/var/folders/xf/1f4dfks55slcjxl6s7b5dty40000gn/T/pip-install-n1jkz4xl/twisted/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/xf/1f4dfks55slcjxl6s7b5dty40000gn/T/pip-record-edgdrx_1/install-record.txt --single-version-externally-managed --compile --install-headers /Users/AlekB/.virtualenvs/city-scrapers-pitt-yoEmy647/bin/../include/site/python3.7/twisted Check the logs for full command output.`

hope some of this information says something about the error.

biniona commented 4 years ago

Solved the problem for Mac OSX Mojave twisted error! I think the issue had to do with me having too many versions of python on my machine and a bunch of dependencies got jumbled up. I attempted to deinstall all my versions of python and then reinstall python 3.7 by following these two links and it worked.

Also reinstalled pipenv during the process (a licensing error came up). I am down to talk about this more if anyone else gets the same error.

SUMMARY: It had pretty much nothing to do with twisted. Seemed more to be an issue with my having weird linking and dependency stuff and I just had to delete all my python versions and reinstall the one I needed for this project.

ben-nathanson commented 4 years ago

Duplicated by #84