sopel
commented
10 years ago A few observations upfront:
- Kafka use Apache Zookeeper in turn
- Kafka solves a cross cutting concern, thus wouldn't need to be tied to this project in particular (which might be a pro and a con)
So it is a somewhat more involved approach to the problems it addresses, but also a fairly thorough one. See Announcing Suro: Backbone of Netflix's Data Pipeline for a good example how resulting capabilities and architectures might look like.
- :information_source: notably, Netflix, routes messages from Kafka to various backends, amongst them Elasticsearch
- they route to S3 from the preceding tier rather than Kafka itself, but that depends on their specific use case I think
joekiller
mrdavidlaing
Apache Kafka is publish-subscribe messaging rethought as a distributed commit log - the implications of this design for our use case are so appealing that they are worth repeating here [emphasis mine] :
Not surprisingly, and despite being a version 0.8 currently, Kafka is already used by quite some major companies (the powered by list is almost a who is who in cloud computing based success stories in general and logging/metrics/monitoring ones in particular) - some of these solutions we are either using ourselves, trying to approximate/extend within this project, or both.
It seems reasonable to evaluate whether it might be the proper solution for resilience and scale out for the messaging pipeline (see e.g. #269 and #272).