istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 323 forks source link

How does it work with CrawlSpider? #212

Closed kevin-ZZZ closed 5 years ago

kevin-ZZZ commented 5 years ago
I did write some code of scrapy ,but with Crawlspider,not spider. So is it necessary to write a base class like RedisSpider in crawler/crawling/spiders/redis_spider.py or you had already done the work?
Appreciate for the answer!
madisonb commented 5 years ago

There is an example of how to create your own spider in the docs. I think the CrawlSpider is a more simplistic approach to the regular Spider class, so you may want to look at how the logic is different inside of each class, but otherwise the high level goal should be the same.

You should be able to add some primitive logic to the spiders of this project to get the same behavior the CrawlSpider gives you, but following the guide above.