softwaremill / kmq

Kafka-based message queue
https://softwaremill.com/open-source/
Apache License 2.0
330 stars 46 forks source link

Kafka Exactly Once implications #2

Open ralph-tice opened 7 years ago

ralph-tice commented 7 years ago

Hi!

Awesome project! Can some information be added about how Kafka 0.11's exactly once functionality will or won't be incorporated into this work? Are they complementary or orthogonal?

adamw commented 7 years ago

I've been meaning to do that but didn't yet manage to :)

There are two ways in which you could leverage Kafka's new features in the kmq context:

However, keep in mind that you'll still get at-least-once processing when interfacing with external clients - there's no way to help with that (*), and re-processing failed messages is the whole point of kmq

(*) unless your client is transactional and can store current message offset, but then you would just use plain Kafka consumers