search

zendesk / ruby-kafka

A Ruby client library for Apache Kafka
http://www.rubydoc.info/gems/ruby-kafka
Apache License 2.0
1.28k stars 337 forks source link

Idempotent message production? #600

Closed jwilm closed 6 years ago

jwilm commented 6 years ago

Kafka 0.11 introduced a feature for producers which allows messages to be added to a topic/partition idempotently. Is there support for this in the library already, or is there interest in supporting it?

Reference: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

0x2c7 commented 6 years ago

@jwilm The support of idempotent message production and transaction are not available yet. But it is already in the roadmap. Kafka version 0.11.0 brings a lot of features and changes on the table. It takes a lot of efforts to fully support everything in ruby-kafka. The new Kafka record batch has just been finished (https://github.com/zendesk/ruby-kafka/pull/548) and released in 0.7.x versions. After it is confirmed to be stable on prod, we'll continue with other complicated features like idempotent producing. This progress can be speed up a lot by trying the 0.7.x versions on production. That would be a great help ❤️

0x2c7 commented 6 years ago

It is a little bit late, but you can track the progress of this feature at https://github.com/zendesk/ruby-kafka/pull/608. It is expected to finish within next month.

0x2c7 commented 6 years ago

The idempotent (and transactional production as well) implementation is merged into master and available in newest 0.7.x releases. Please try out this feature, the document is at https://github.com/zendesk/ruby-kafka/pull/608. We'll fulfill the documentation very soon. I'll close this issue now.

jwilm commented 6 years ago

Wow, that was fast! Thanks everyone!

cc @cohedrin