search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 323 forks source link

Zookeeper dependency? #37

Closed sibiryakov closed 8 years ago

sibiryakov commented 8 years ago

Hey guys!

We've met at Memex Summer workshop last time. I was looking through this code, and didn't found any dependency on Zookeeper. Am I missed something? If not, probably it makes sense to remove it from docs.

madisonb commented 8 years ago

Zookeeper is a dependency for Kafka, and for controlling the crawlers in version 1.1. So your Kafka hosts needs to connect to ZK as well as all of your Scrapers (if you chose to work off of the dev branch)

madisonb commented 8 years ago

If you have any further questions about the Zookeeper dependency please reopen this issue, otherwise closing for now.