exactly-once delivery in brod

[ghost]

Hi,

Does Brod (or the underlying Kafka protocol) currently support exactly-once delivery for producers and consumers?

[zmstone]

Hi.

It's a work in progress in https://github.com/klarna/kafka_protocol/pull/40.

[ghost]

Thanks for your work, @zmstone! Will Brod then support new functionality like the idempotent producer and then transactional API accordingly?

[zmstone]

Not planing to bump major. Meaning, the current producer/consumer APIs will be kept compatible. New features like transactional produce/commit will be supported via a brod_tx module.

[ghost]

@zmstone Thanks! Again, nice work!

[zmstone]

Hi @arcoviilan The primitive transactional APIs are supported in kafka_protocol but not in brod yet. We had to release 3.6 without brod_tx. This is currently not our priority to implement this new feature. Contributions are welcome.

[ananthakumaran]

Hi @zmstone

I am interested in working on this feature. I have skimmed through the kafka design docs and the transaction apis in kafka_protocol. Given that the transaction api seems like another layer on top of the existing functionality, do you prefer to go with with a new module with completely new set of apis or would you prefer to change the existing implementation to add support for transaction? My use case is similar to kafka stream (transactional consumer group + producer). So, it would be ideal for me if it works with the existing group_subscriber/coordinator implementation.

[zmstone]

Hi. If I understand correctly, the data flow is like: fetch -> callback:process -> produce -> commit offset -> commit transaction I do not think the current implementation will fit. I am not inclined to have a different layer on top, maybe internally or maybe event a higher level application.

This new layer may:

• Implement brod_group_coordinator behaviour for consumer group state management
• Not use brod_consumer (or the current _subscriber modules), to get upstream data, but call brod:fetch directly after assignments are received