search

didier-wenzek / ocaml-kafka

OCaml bindings for Kafka
MIT License
63 stars 10 forks source link

Balanced consumer groups support #29

Open Lupus opened 2 years ago

Lupus commented 2 years ago

librdkafka supports broker based balanced consumer groups (see FAQ), would be awesome to have this API wrapped in ocaml-kafka lib.

didier-wenzek commented 2 years ago

Yes, it would be good to have this. A pull request has been open one year ago in that direction. But, to be honest, I missed the time and energy to work on it.

Lupus commented 2 years ago

Looking at the PR and codebase of Async and Lwt wrappers I'm a bit confused on why Async and Lwt backends are so different. Async backend wraps rd_kafka_consumer_poll while it's not exposed from the C-wrapping async-monad agnostic "core" library. I was expecting the core library to just provide bindings to all (or required) librdkafka API surface, so that it can be used directly in blocking code (say with a bunch of threads, OCaml 5 is coming...), and Lwt/Async wrappers to build on that and provide useful integrations of polling/awaiting of asynchronous operation results.

Leonidas-from-XIV commented 2 years ago

@Lupus Author of the balanced consumer code here. The Async wrappers provide it because they want to expose a high-level interface to balanced consumer groups. As such I was implementing the minimum necessary to support consumer groups using the "modern" API, to be used for Async and not much else. Providing such support for the "core" library was not one of my objectives because for the work that I needed it, there was no need for it (threads are not viable and OCaml 5 is on the horizon but not there yet and even after it is released it will be a few years until it will see widespread usage), and designing the required primitives in a way where both the Lwt bindings and the Async bindings could build upon them was more work. Especially given the Lwt bindings use the "old" rdkafka API.

That said, you could potentially move the C bindings to the "core" part of the library and then rework both concerrency monad bindings to use them. It just wasn't necessary since I don't use Lwt myself.