search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 324 forks source link

Add Docker Support #72

Closed madisonb closed 8 years ago

madisonb commented 8 years ago

This PR completes the initial cut of Docker support for scrapy cluster. The 3 core containers are all tagged in dockerhub and it is pretty easy to push new images for the latest dev release that passes all tests once this is merged in.

New documentation is in the Quickstart Guide as well as docs within the Advanced section. Added support for better branching logic within Travis to test everything properly. Docker Compose is used to orchestrate things to give a sample compose file for how to spin up a Scrapy Cluster project via Docker.

coveralls commented 8 years ago

Coverage Status

Coverage remained the same at 65.295% when pulling d58c2deadd9c3a091f4d034b7e99d5270dd19761 on docker into c18a776eec402524d2e34cd5330ca2dac64437ec on dev.

coveralls commented 8 years ago

Coverage Status

Coverage remained the same at 65.295% when pulling d58c2deadd9c3a091f4d034b7e99d5270dd19761 on docker into c18a776eec402524d2e34cd5330ca2dac64437ec on dev.

madisonb commented 8 years ago

Closes #48