search

kafkaex / kafka_ex

Kafka client library for Elixir
MIT License
595 stars 162 forks source link

Exactly once delivery? #287

Closed ghost closed 5 years ago

ghost commented 6 years ago

Hi,

Does kafka_ex currently support exactly-once delivery for producers and consumers?

joshuawscott commented 6 years ago

Kafka claims to provide exactly-once semantics in 0.11 and up: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ but there are a lot of caveats: https://softwaremill.com/what-kafka-exactly-once-really-means/

KafkaEx currently only supports up to the 0.9 API, so the 0.11 features are not available. Additionally, achieving exactly-once depends strongly on the design of the producers and consumers outside of the kafka client.

ghost commented 6 years ago

Thanks, @joshuawscott!

So I'm currently using Erlang and really want a Kafka extension which guarantees exactly-once delivery, is it possible to achieve exactly-once in KafkaEx, say using at-least-once delivery and idempotent operation? If not, do you have any suggestions about how to implement exactly-once feature based on KafkaEx or using any other approaches?

joshuawscott commented 6 years ago

If you have idempotent operation, then you basically get exactly-once just by ensuring you don't commit an offset until the operation has been verified. This is possible now for consumers. You could have the same thing for producers by looking at how Kafka implemented this in 0.11 (basically by creating unique sequential identifiers outside kafka and ensuring you do not produce out of order)