istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 324 forks source link

Sc utils #21

Closed jasonrhaas closed 9 years ago

jasonrhaas commented 9 years ago

All commits to package scutils. Crawler code was modified to use the new package.