search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 323 forks source link

Replace Kafka with another pub/sub service? #215

Closed jrmlhermitte closed 5 years ago

jrmlhermitte commented 5 years ago

I'm just curious how hard it would be to replace kafka with another messaging service, say RabbitMQ, google pub sub, or something else. Are there plans to do this?

madisonb commented 5 years ago

There are no plans to do this at this time, however I have certainly thought about adding additional support for other queue frameworks.

You would need to touch every major component, however you would simply need to swap out the code for "reading data from the queue" and "pushing data to the queue".

I am not super familiar with the best practices with RabbitMQ or Redis/Google PubSub, so for now I think I am going to leave it as is.

madisonb commented 5 years ago

Closing this ticket

jrmlhermitte commented 5 years ago

sounds good thanks for the info