(myvenv) ➜ /Users/fatli/python/test02/scrapy-master > python3 setup.py install
# 省略......
creating dist
creating 'dist/Scrapy-1.6.0-py3.5.egg' and adding 'build/bdist.macosx-10.6-intel/egg' to it
removing 'build/bdist.macosx-10.6-intel/egg' (and everything under it)
Processing Scrapy-1.6.0-py3.5.egg
creating /Users/fatli/python/test02/myvenv/lib/python3.5/site-packages/Scrapy-1.6.0-py3.5.egg
Extracting Scrapy-1.6.0-py3.5.egg to /Users/fatli/python/test02/myvenv/lib/python3.5/site-packages
Adding Scrapy 1.6.0 to easy-install.pth file
Installing scrapy script to /Users/fatli/python/test02/myvenv/bin
Installed /Users/fatli/python/test02/myvenv/lib/python3.5/site-packages/Scrapy-1.6.0-py3.5.egg
Processing dependencies for Scrapy==1.6.0
Searching for service_identity
Reading https://pypi.python.org/simple/service_identity/
Download error on https://pypi.python.org/simple/service_identity/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:645) -- Some packages may not be found!
Reading https://pypi.python.org/simple/service-identity/
Download error on https://pypi.python.org/simple/service-identity/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:645) -- Some packages may not be found!
Couldn't find index page for 'service_identity' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:645) -- Some packages may not be found!
No local packages or download links found for service_identity
error: Could not find suitable distribution for Requirement.parse('service_identity')
# 此时Scrapy已经安装好,但是缺少某些依赖包,例如这里输出显示的”service_identity“包,将导致运行scrapy失败。
# 此时的 pip list:
(myvenv) ➜ /Users/fatli/python/test02/scrapy-master > pip3 list
Package Version
---------- -------
pip 19.0.3
Scrapy 1.6.0
setuptools 20.10.1
wheel 0.33.1
三、安装Scrapy依赖包
(myvenv) ➜ /Users/fatli/python/test02/scrapy-master > pip3 install service_identity
# 省略......
scrapy 1.6.0 requires cssselect>=0.9, which is not installed.
scrapy 1.6.0 requires lxml, which is not installed.
scrapy 1.6.0 requires parsel>=1.5, which is not installed.
scrapy 1.6.0 requires PyDispatcher>=2.0.5, which is not installed.
scrapy 1.6.0 requires pyOpenSSL, which is not installed.
scrapy 1.6.0 requires queuelib, which is not installed.
scrapy 1.6.0 requires Twisted>=13.1.0, which is not installed.
scrapy 1.6.0 requires w3lib>=1.17.0, which is not installed.
Installing collected packages: attrs, pyasn1, pyasn1-modules, six, asn1crypto, pycparser, cffi, cryptography, service-identity
Successfully installed asn1crypto-0.24.0 attrs-19.1.0 cffi-1.12.2 cryptography-2.6.1 pyasn1-0.4.5 pyasn1-modules-0.2.4 pycparser-2.19 service-identity-18.1.0 six-1.12.0
# 此时的 pip list:
(myvenv) ➜ /Users/fatli/python/test02/scrapy-master > pip3 list
Package Version
---------------- -------
asn1crypto 0.24.0
attrs 19.1.0
cffi 1.12.2
cryptography 2.6.1
pip 19.0.3
pyasn1 0.4.5
pyasn1-modules 0.2.4
pycparser 2.19
Scrapy 1.6.0
service-identity 18.1.0
setuptools 20.10.1
six 1.12.0
wheel 0.33.1
# 根据上面输出命令提示,直接使用pip安装lxml等依赖包,除了Twisted包。
(myvenv) ➜ /Users/fatli/python/test02/scrapy-master > pip3 install Twisted
# 省略......
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/bf/cq8wzfms3glgschngtqys5f80000gn/T/pip-install-__uqc53m/Twisted/
# pip安装Twisted失败,解决方法:下载安装:
# 下载Twisted压缩包:https://twistedmatrix.com/trac/wiki/Downloads#SourceTarball
# 解压后进入该包路径进行 $ python3 setup.py install 安装:
(myvenv) ➜ /Users/fatli/python/test02/scrapy-master > cd ../Twisted-19.2.0
(myvenv) ➜ /Users/fatli/python/test02/Twisted-19.2.0 > python3 setup.py install
# 省略......
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('incremental>=16.10.1')
# 根据输出命令,直接pip安装例如incremental等Twisted的依赖包。
# 在Twisted-19.2.0目录下,再次执行python3 setup.py install,成功了
(myvenv) ➜ /Users/fatli/python/test02/Twisted-19.2.0 > python3 setup.py install
# 省略......
Using /Users/fatli/python/test02/myvenv/lib/python3.5/site-packages
Finished processing dependencies for Twisted==19.2.0
# 根据上面的pip list输出,Scrapy已经安装好了。测试此时Scrapy是否能运行。报错:缺少依赖包:service_identity
(myvenv) ➜ /Users/fatli/python/test02/Twisted-19.2.0 > scrapy version
# 省略......
pkg_resources.DistributionNotFound: The 'service_identity' distribution was not found and is required by Scrapy
# 报错:缺少依赖包:service_identity:
(myvenv) ➜ /Users/fatli/python/test02/Twisted-19.2.0 > cd ../scrapy-master
(myvenv) ➜ /Users/fatli/python/test02/scrapy-master > python3 setup.py install
# 省略......
pkg_resources.DistributionNotFound: The 'service_identity' distribution was not found and is required by Scrapy
# 注意:上面“安装Scrapy依赖包”时已经安装该包。根据上面“下载安装Scrapy”输出显示,可能是SSL的问题。
# 此时,暂无法解决。
开发环境:
一、建立虚拟环境并更新pip
此时的 pip list:
二、安装Scrapy
方法一:直接使用pip安装Scrapy(失败):
方法二:下载安装Scrapy(成功):
直接使用pip安装Scrapy失败。建议下载安装,下载解压后,在该包命令行目录下使用setup.py安装: scrapy-master.zip下载地址:https://scrapy.org/download
三、安装Scrapy依赖包
此时的 pip list:
四、测试使用Scrapy