search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 323 forks source link

Plans to support latest version of Kafka, ELK & Scrapy? #217

Closed kumaresra closed 5 years ago

kumaresra commented 5 years ago

Any plans to support the latest version of Kafka, ELK & Scrapy?

madisonb commented 5 years ago

I bump things most of the time when it is needed or I get a PR for it.

Current versions in this project under the dev branch Scrapy==1.5.0

Tested against containers wurstmeister/kafka (latest) wurstmeister/zookeeper (latest) redis (latest)

In terms of ELK - this project is not meant to be bleeding edge against the ELK stack, the example in the documentation and located within this codebase is meant to be just an example. However, the logstash config I know works up through the 6.x series, and I am sure would be easy enough to get working against the latest 7.x series. That primary logging interface within Logstash doesn't change much.

If this answers your question, please close this ticket