search

SOHU-Co / kafka-node

Node.js client for Apache Kafka 0.8 and later.
MIT License
2.66k stars 628 forks source link

A better design than ConsumerGroupStream you should try! #1077

Open thynson opened 6 years ago

thynson commented 6 years ago

ConsumerGroupStream is hard to (and I believe it's impossible to correctly) use with commit being manually managed. After working with it for several weeks I decided to to made a better design, namely ConsumerGroupPipeline, now available at https://github.com/thynson/kafka-pipeline. Which supports:

  1. Reliable offsets commit, only commit mssages that have actually been consumed.

  2. Concurrently consume. Yes, messages can be concurrently consumed while offset is still correct!

While I maintained it as a separated package now, I'm appreciated that if it could be adopt into the kafka-node.

hyperlink commented 6 years ago

whoa, @thomaslee and @aikar are both trying to solve a similar problem.

Thank you for sharing this @thynson. I hope to take a look at it in detail later.

thomaslee commented 6 years ago

Nice! I'll check this out sometime this week.

In the meantime, #1071 has my own proof of concept and @aikar raised an issue over in #975 with some of his own ideas.