Kafka Exactly Once Semantics Support (Kafka 0.11.0.x)

[vineetgoel]

Is there a plan on when aiokafka will support the newly added exactly once semantics using Idempotent and Transactional Producers?

[tvoinarovskyi]

I will definitely work on this, but aiokafka needs a lot of old features finished, so I would expect kafka-python to have the feature first. Do you have a need for exactly Transactional Producer, or you would be ok with Idempotent delivery. The later is much easier and will probably be the first goal.

[tvoinarovskyi]

Lets add the KIP-98 link here

[vineetgoel]

I think we would need both of them for our use case.

[tvoinarovskyi]

I start development of Transactional producer and plan to finish it till the end of this month, as I will have some spare time.

[tvoinarovskyi]

If there are suggestions on how the API should be structured, would be glad to hear. @ask @vineetgoel

[vineetgoel]

@tvoinarovskyi sorry about the late response. The standard API (in the java clients) should work well for us.

Particularly the things we would need are:

• Idempotent flag on the producer
• Consumer transactional flag
• Producer start transaction
• Producer commit offsets and end transaction;

We will typically be sending a bunch of messages in a transaction and then committing offsets and ending the transaction.

While reading, we would typically only want to read committed transactions.

[tvoinarovskyi]

@vineetgoel The implementation was merged to master if you have time/eagerness I'm open to API suggestions or fixups. Hope will release it as 0.5 this year ;) Thanks for the patience, was a long year with this!

[tvoinarovskyi]

Released as part of v0.5.0