slifty
commented
4 years ago Samza has no node client so that's easy. Storm also seems like it would be a bit of a hack-and-slash to integrate with, and I'm not convinced it is actually a good fit for our model.
Given that, we're at Kafka
I am strongly leaning to kafkajs at this point because the community seems a bit more active and I suspect that in the long run it will be better maintained.
Lack of stream support is a weakness, but I am also not certain that we want or need kafka-level streams at this stage -- Kafka streams seem nice because it fits in with the decentralized / scaling capabilities of Kafka. That said, those are both features that e honestly are not going to be using yet.
We like Kafka because the paradigms that come with it, such as consumer groups and topics, are attractive, and fit our problem space quite well. The potential future-readiness is a bonus (and so the lack of support for streams-within-kafka is acceptable)
For now I think we would be able to have "streams" be handled by our node code directly, and use kafka as the system for coordination between modular components where messages carry atomic payloads (and where that atomic payload might be a pointer to a node-managed stream).
Discussion
What do you want to talk about?
There are multiple kafka node libraries out there, we should pick which one to use in this project. The two candidates at this point are kafka-node and kafkajs. The former seems a bit more stale, though it has better support for the concept of streams. The latter seems to have a more vibrant community, and a design philosophy that prioritizes simplicity.
As part of our exploration we should actually decide for certain if kafka is the right solution for us here. There are alternatives (E.g. Apache Storm and Apache Samza) which I have no experience with, but that does not mean they wouldn't be a good fit!
Relevant Resources / Research
Kafkajs has a slack group which I've joined