search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 323 forks source link

Inter-spider communication #178

Closed shenbakeshkishore closed 6 years ago

shenbakeshkishore commented 6 years ago

Hi,

Please share sample code to perform inter-spider communication.(In reference to https://scrapy-cluster.readthedocs.io/en/latest/topics/crawler/controlling.html#inter-spider-communication)

madisonb commented 6 years ago

If you read that section it gives you an example, but here is a more complete one:

req = Request(some_url, callback=self.parse)
req.meta['spiderid'] = 'myotherspidername'
yield req

Closing.