search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 323 forks source link

Can you upgrade scrapy to version 1.5? There is an error in the upgrade #170

Closed zusrsoft closed 6 years ago

zusrsoft commented 6 years ago

Can you upgrade scrapy to version 1.5?

Can there be other components to upgrade the new version?

Thank you!

zusrsoft commented 6 years ago

（Question 1 ） /home/dt/scrapy-cluster-1.2.1/crawler/crawling/distributed_scheduler.py:8: ScrapyDeprecationWarning: Module scrapy.conf is deprecated, use crawler.settings attribute instead from scrapy.conf import settings

（Question 2） /home/dt/scrapy-cluster-1.2.1/crawler/crawling/log_retry_middleware.py:10: ScrapyDeprecationWarning: Importing from scrapy.xlib.tx is deprecated and will no longer be supported in future Scrapy versions. Update your code to import from twisted proper. from scrapy.xlib.tx import ResponseFailed

madisonb commented 6 years ago

Your core question duplicates an existing PR #160.

Your deprecation warnings are fine, the first one I have tried to fix in the past but the recommended import seems to not work. The second deprecation I have not had any issue with either. Maybe that has improved since I last checked.

If you would like to change the imports so the warnings no longer show I would be happy to accept a pull request.

Can I close this?

zusrsoft commented 6 years ago

This upgrade, you can work? Can help test and verify, thank you.

Upgrade package details are as follows:

ConcurrentLogHandler==0.9.1 Flask==0.12.2 # Updated from 0.11/0.12 Jinja2==2.10 # Updated from 2.8/ 2.9.5 MarkupSafe==1.0 # Updated from 0.23 PyDispatcher==2.0.5 PyYAML==3.12 Scrapy==1.5.0 # Updated from 1.3.1 /1.3.3 Twisted==17.9.0 # Updated from 16.4.0 /17.1.0 Werkzeug==0.14.1 # Updated from 0.11.11/0.12.1 attrs==17.4.0 # Updated from 16.1.0/16.3.0 cffi==1.11.4 # Updated from 1.7.0/1.9.1 characteristic==14.3.0 click==6.7 # Updated from 6.6 coverage==4.5 # Updated from 4.0.3/4.3.4 cryptography==2.1.4 # Updated from 1.5/1.8.1 cssselect==1.0.3 # Updated from 0.9.2/1.0.1 enum34==1.1.6 funcsigs==1.0.2 future==0.16.0 # Updated from 0.15.2 idna==2.6 # Updated from 2.1/2.5 ipaddress==1.0.19 # Updated from 1.0.16/1.0.18 itsdangerous==0.24 jsonschema==2.6.0 # Updated from 2.5.1 kafka-python==1.3.5 # Updated from 1.3.2/1.3.3 kazoo==2.4.0 # Updated from /2.2.1 lxml==4.1.1 # Updated from 3.6.4/3.7.3 mock==2.0.0 nose==1.3.7 parsel==1.3.1 # Updated from 1.0.3/1.1.0 pbr==3.1.1 # Updated from 1.10.0/2.0.0 pyOpenSSL==17.5.0 # Updated from 16.1.0/16.2.0 pyasn1-modules==0.2.1 # Updated from /0.0.8 pyasn1==0.4.2 # Updated from 0.1.9/0.2.3 pycparser==2.18 # Updated from 2.14/2.17 python-json-logger==0.1.8 # Updated from 0.1.5/0.1.7 python-redis-lock==3.2.0 # Updated from 3.1.0 queuelib==1.4.2 redis==2.10.6 # Updated from /2.10.5 requests-file==1.4.3 # Updated from 1.4/1.4.1 requests==2.18.4 # Updated from 2.11.1/2.13.0 retrying==1.3.3 scutils==1.2.0 service-identity==16.0.0 six==1.11.0 # Updated from /1.10.0 testfixtures==5.4.0 # Updated from 4.11.0/4.13.5 tldextract==2.2.0 # Updated from 2.0.1/2.0.2 ujson==1.35 w3lib==1.19.0 # Updated from 1.16.0/1.17.0 zope.interface==4.4.3 # Updated from 4.2.0/4.3.3

madisonb commented 6 years ago

Closing in favor of #160.