Scrapy 1.0 Support - Githubissues

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

http://scrapy-cluster.readthedocs.io/

MIT License

1.18k stars 324 forks source link

Scrapy 1.0 Support #1

Closed madisonb closed 8 years ago

madisonb commented 9 years ago

Scrapy Cluster currently supports Scrapy version 0.24.x, and there are many improvements to the new release of Scrapy that we could benefit from using.

http://doc.scrapy.org/en/latest/news.html

madisonb commented 9 years ago

Initial upgrades made via commit 7daef27d90e3d4d30e509544c64e9065ffb0229d. Tested on Scrapy 1.0.3

Still need to fix item dictionaries and ensure we are utilizing everything scrapy 1.0 has to offer, but code now will compile and run against the latest version.

madisonb commented 8 years ago

Closing as I think future enhancements brought via Scrapy should be included in later versions of SC as the biggest changes for SC 1.1 were to go from 0.24 to 1.0