search

IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.59k stars 1.76k forks source link

The current state of `read_committed` #1284

Closed FrancoisPoinsot closed 5 years ago

FrancoisPoinsot commented 5 years ago

Even though transaction are not ready yet in sarama, it would still be super useful to be able to read committed messages only. Of course such message would typically be produced by java producer to begin with.

Before spending too much time on this, i would like to confirm a few things first.

1) Sarama does not support yet read_committed on consumer side. Is it correct? 2) Sarama does not support yet transactions on producer side. Is it correct? 3) read_committed is not just about sending the correct flag to the broker on a fetchRequest. The client need to filter the results. Is it correct? 4) It is totally doable to support read_committed without using transactions on producer side. The magic only happen on consumer side. Is it correct? 5) Is there some on-going work regarding the read_committed support?

When looking for information about how to support read_committed i stumbled upon 2 docs that i will note here as reference:

However, even after reading those, the work done by the client is still unclear to me. I would gladly get my hand on any link that would describe accurately how to implement read_committed.

FrancoisPoinsot commented 5 years ago

So i figured that first thing i would need to check if i can handle uncommitted transaction correctly, is to have some uncommitted transaction. So here is a simple producer that output some committed and some uncommitted messages: https://github.com/FrancoisPoinsot/simplest-uncommitted-msg

FrancoisPoinsot commented 5 years ago

Using previous custom producer, here is a setup for testing the read_commited: https://github.com/FrancoisPoinsot/sarama/tree/read-committed/tools/uncomitted-messages

Everything is in the readme, but in short:

With this, i supose next step is to try to actually implement the read_committed.

FrancoisPoinsot commented 5 years ago

I think I found a pretty concise description of how client should handle read_committed: https://github.com/apache/kafka/blob/38a7708a47361929df37ed0f03160bbac5e51e93/clients/src/main/resources/common/message/FetchRequest.json#L58

FrancoisPoinsot commented 5 years ago

https://github.com/Shopify/sarama/pull/1307