Error when install scrapy in window by using pip install scrapy

[lhkthomas]

Errors exists about the Twisted when installing scrapy in window by using pip install scrapy. Any one know how to fix it.

 copying src\twisted\internet\test\_yieldfromtests.py.3only -> build\lib.win32-2.7\twisted\internet\test
    creating build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\chain.pem -> build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\not-a-certificate -> build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing1.pem -> build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing2-duplicate.pem -> build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing2.pem -> build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\mail\test\rfc822.message -> build\lib.win32-2.7\twisted\mail\test
    copying src\twisted\python\test\_deprecatetests.py.3only -> build\lib.win32-2.7\twisted\python\test
    copying src\twisted\words\im\instancemessenger.glade -> build\lib.win32-2.7\twisted\words\im
    copying src\twisted\words\xish\xpathparser.g -> build\lib.win32-2.7\twisted\words\xish
    running build_ext
    building 'twisted.test.raiser' extension
    error: INCLUDE environment variable is empty

    ----------------------------------------
Command "c:\python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\thomas~1\\appdata\\local\\temp\\pip-build-3oirm6\\Twisted\\setup.py';
f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))"
 install --record c:\users\thomas~1\appdata\local\temp\pip-xvbgd2-record\install-record.txt --single-version-externally-managed --compile" 
failed with error code 1 in c:\users\thomas~1\appdata\local\temp\pip-build-3oirm6\Twisted\

[redapple]

Installing scrapy on Windows using pip is notably tricky. The current recommended way to get Scrapy on Windows is to use Anaconda/Miniconda and the conda-forge channel.

[redapple]

@lhkthomas , did you make it work?

[ghost]

I got it to work by issuing the pip install scrapy command in the Developer Command Prompt for Visual Studio.

[leuthier]

I'm having same issue trying to install Scrapy on Windows using pip...

3.4\twisted\internet\test\fake_CAs
    copying src\twisted\mail\test\rfc822.message -> build\lib.win-amd64-3.4\twisted\mail\test
    copying src\twisted\python\test\_deprecatetests.py.3only -> build\lib.win-amd64-3.4\twisted\python\test
    copying src\twisted\words\im\instancemessenger.glade -> build\lib.win-amd64-3.4\twisted\words\im
    copying src\twisted\words\xish\xpathparser.g -> build\lib.win-amd64-3.4\twisted\words\xish
    running build_ext
    building 'twisted.test.raiser' extension
    error: Microsoft Visual C++ 10.0 is required. Get it with "Microsoft Windows SDK 7.1": www.microsoft.com/download/details.aspx?id=8279

    ----------------------------------------
Command "c:\python34\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\victo\\AppData\\Local\\Temp\\pip-build-hizsl1j0\\Twisted\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\victo\AppData\Local\Temp\pip-1b19gjbt-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\victo\AppData\Local\Temp\pip-build-hizsl1j0\Twisted\

[cpoptic]

@leuthier Ditto here. I've installed Visual Studio C++ 14.0

[ranajit-paul]

I got similar type of error when I tried to install scrapy in cloudera. I used python 2.6.6 .

gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.4m -c src/twisted/test/raiser.c -o build/temp.linux-x86_64-3.4/src/twisted/test/raiser.o src/twisted/test/raiser.c:4:20: error: Python.h: No such file or directory src/twisted/test/raiser.c:6:6: error: #error Python headers needed to compile C extensions, please install development version of Python. error: command 'gcc' failed with exit status

How to resolve the issue???

[ayushmankoul]

Even I got the same issue but I took reference from https://www.youtube.com/watch?v=eEK2kmmvIdw video and was able to pip install scrapy on windows without using anaconda

[eulerlobo]

The video above helped a lot!

But I was struggle to install Cryptography and pyOpenSSL, after install the WinOpenSSL from the Shining Light Productions, even with the Enviroment Variables setted.

The solution I found was set global-options along the instalation, like:

pip install cryptography --global-option=build_ext --global-option="-IC:\OpenSSL-Win64\include" --global-option="-LC:\OpenSSL-Win64\lib"

pip install pyopenssl --global-option=build_ext --global-option="-IC:\OpenSSL-Win64\include" --global-option="-LC:\OpenSSL-Win64\lib"

[Chloe-Y]

win10, python 3.5 manualy download twisted from https://www.lfd.uci.edu/~gohlke/pythonlibs/ choose the version and download pip install then pip install Scrpy hope it help

[flyher]

@Chloe-Y Your answer have solved my question, thank you very much!

pip install <Twisted-18.7.0-cp35-cp35m-win_amd64.whl>
pip install scrapy
pip install pypiwin32

[musawakiliML]

just use conda install scrapy with anaconda. it worked for me.

[MounikaKalahasthi7]

@Chloe-Y while using the below command pip install i got an error like "not a supported wheel on this platform" can you give any suggestions to resolve this issue

[Chloe-Y]

import platform platform.architecture()

then, download the file of that bit irrespective of your system architecture.

[midcenturydude7]

@musawakiliML, your suggestion worked for me! conda install scrapy

I could not get it to work from my terminal (cmder), but, I was able to get it to properly install from the Anaconda prompt. At least I can access it there.

Of note, the conda-forge channel did not work for me, either in the Windows cmd (admin or otherwise), or in the Anaconda prompt, or in a PyCharm virtualenv. But, I am very much new to downloading and installing packages at the command line.

[samhaldia]

If you have conda Use: conda install -c anaconda scrapy # to install

Install Using pip

Or Download https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted at this location pip install Twisted-18.9.0-cp27-cp27m-win_amd64.whl pip install scrapy

[monirrad]

I know the question is asked long time ago, but I post my answer as a reference if anyone else had the same problem.

I had the same problem too and I solved it as follow:

Open the Anaconda Prompt as administrator (For Windows10: open cortana/write Anaconda Prompt/choose "Run as Administrator")

You should go to the path of Anaconda, for me was like:

C:\WINDOWS\system32>cd ..
C:\WINDOWS>cd..
C:\>cd  ProgramData
C:\ProgramData>cd Anaconda3
C:\ProgramData>Anaconda3>

Then you should run the following command

C:\ProgramData>Anaconda3>conda install -c anaconda twisted

At some point it asks

Proceed ([y]/n)?

type y. Now twisted is installed.

To install scrapy, you either install it in Anaconda Prompt (as administrator) by running the following command:

C:\ProgramData>Anaconda3>conda install -c conda-forge scrapy (again y for Proceed ([y]/n)?)

or on jupyter notebook and run the command

!pip install scrapy

[ghost]

@Chloe-Y thank you! win10, python 3.7 manualy download twisted from https://www.lfd.uci.edu/~gohlke/pythonlibs/ choose the version and download Ex: pip install Twisted-18.9.0-cp37-cp37m-win_amd64 Successfully Installed Twisted 18.9.0 Ex: pip install Scrapy Successfully Installed Scrapy 1.9.0

[H-Hooshmand]

@Chloe-Y Tnx very much. It was very helpful.

[vipulgupta2048]

For Ubuntu folks, trying to follow how-to fix this --> Check this

[elhloukhalil]

just use conda install scrapy with anaconda. it worked for me too thank you

[tuanthank]

ERROR: Twisted-19.2.1-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.

[samhaldia]

@tuanthank You need to download Your Python version and Platform specific Twisted-19.2.1-cp37-cp37m-win_amd64.whl.whl file, for ex. cp37 here is for Python 3.7 and win_amd64 is for 64 bit OS. Use the Twisted whl file accordingly

[tuanthank]

Thanks @samhaldia I have installed scrapy 1.7.3 but can not startproject. The error message is "'scrapy' is not recognized as an internal or external command, operable program or batch file"

[FredRF]

Hi @tuanthank

Provided that you followed @samhaldia instructions, if you still get the error you mention above, you may need to add "scrapy" to your environement variables. After doing that, re-open your command line for the changes to make effect.

[ghost]

I had the very same problem the solution for me was to install node.js I installed the last version of node.js and the scrapy (which needs twisted) got downloaded without any problem hope it be useful for you

[selvarajdevaraj]

win10, python 3.5 manualy download twisted from https://www.lfd.uci.edu/~gohlke/pythonlibs/ choose the version and download pip install then pip install Scrapy hope it help

yeah!.. it works

[supercool27]

while installing scrapy using conda . having problem like fatel error

issue resolved i opend command prompt using anaconda and apply

scrapy is working fine for me.

hope it will works for any of my friends. good luck.

[Bishwas-py]

It's super easy to solve. You can install twisted manually. Here's the solution for this: Instant Solution for Scrapy failed building wheel for Twisted

[ZER0-SAMA]

No conda install or microsoft studio. Follow: 1.Download Twisted : https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted 2.In CMD : cd C:\Users\USERNAME\Downloads 3.pip install Twisted-version.whl Enjoy!

[shubham-tyagi]

@tuanthank You need to download Your Python version and Platform specific Twisted-19.2.1-cp37-cp37m-win_amd64.whl.whl file, for ex. cp37 here is for Python 3.7 and win_amd64 is for 64 bit OS. Use the Twisted whl file accordingly

Thank you !! It was very helpful.

[ndamulelonemakh]

win10, python 3.5 manualy download twisted from https://www.lfd.uci.edu/~gohlke/pythonlibs/ choose the version and download pip install then pip install Scrapy hope it help

yeah!.. it works

Worked on windows 10, python3.8.6

[LaPalli]

Installing scrapy on Windows using pip is notably tricky. The current recommended way to get Scrapy on Windows is to use Anaconda/Miniconda and the conda-forge channel.

I'm on Windows trying to install miniconda or anaconda. I can't do either. I have tried to download miniconda from docs.conda.io on two different days. Every time I click on the corresponding Python+Windows download link, it states "Connecting..." then the error, "This site can't be reached. repo.ananconda.com took too long to respond." Alternatively, I have visited https://www.anaconda.com/download/#windows and tried to download the full Anaconda. I get the same error here as well. My internet is working fine on all other websites and my popup blocker is disabled. Any idea why I can't download these?

[AlkisPis]

Forget about pip install! The issue with 'scrapy' is all over the Web. Download 'Scrapy-2.5.0.tar.gz' (or whatever version is current) from https://pypi.org/project/Scrapy/, exract it to a temp folder and run < python setup.py install >

[wRAR]

All of these errors are about Twisted, not Scrapy.