search

confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client
https://github.com/confluentinc/confluent-kafka-dotnet/wiki
Apache License 2.0
90 stars 868 forks source link

Plans to allow frameworks on top of confluent-kafka-dotnet? #1126

Open ChadJessup opened 4 years ago

ChadJessup commented 4 years ago

I'm currently trying to port Kafka Streams to C# and I'm hitting some issues with the current API approach of this C# kafka library. I believe that the current design is to help reduce support/maintenance burden, and help users fall into the 'pit of success', and I think it does that well.

But, I'm having quite a bit of difficulty doing more advanced scenarios, and I'm not sure I can provide a framework on top of this library without being able to access some of the internals and/or override some of the behavior that's built-in.

What, if any, plans are there for allowing access to some of the internals, or more raw access to the underlying librdkafka interop layer through this library? I feel that opening* it up a little would allow for a vibrant ecosystem to form over this officially supported library. As it'd be a shame to fork just to change some access modifiers. Unless that's the approach that Confluent would like people to go?

By opening up, I've seen a couple good approaches in other officially supported libraries along these lines. NEST - has a Property called InnerApi for accessing the underlying ElasticSearch raw API. Or a `.Core` library, that can be optionally referenced which contains primitives to be consumed by advanced scenarios.

I'd be more than willing to open up a PR exploring something like the above, but I don't want to waste the maintainer's time if that's a direction that isn't desired.

Thanks

mhowlett commented 4 years ago

We're not intentionally providing a dumbed down/simplified API - where a feature isn't exposed it's because we just haven't added it to the binding yet, or it's not implemented in librdkafka yet. the API was designed to be extensible enough for any capability we could foresee.

feel free to provide specific examples of gaps, and we can provide specific guidance on how we envisage that fitting into the API.

we'd be very happy to see a streams layer on top of this library. I've had a go myself (but my motivation really was on playing with some novel ideas, not duplicate kafka streams - i don't have the bandwidth to do a production ready stream processing library).

ChadJessup commented 4 years ago

I'm sorry, I didn't mean to imply that there were problems with the Api as it stands today. I'm sure my ignorance on many aspects is a big factor here.

I'll summarize some of the general issues I've come across below, and open up new issues/PRs on specific problems I can't seem to address on my own.

With much of the implementation marked internal, being required to use the Builder pattern, and not having access to the interop layer, I'm not able to easily (or at all) use a few C# patterns that I'm used to (I might be the only one having these issues, of course):

  1. Deriving from and instantiating custom clients:
    • I'm having to wrap a builder's output in multiple layers of redundant code to use DI, log in a specific place, or set breakpoints, etc.
    • I'd like to be able to create (or have provided) abstract clients, and common clients derived from them. Where I inherit and override where I want. Or, I can build-up from any layer down to the official base interfaces.
    • I wanted to work around some above issues, but due to the internal-only interop layer, I can't implement a consumer from scratch (and copy/paste from this repo) without bringing in the entire code-base of this library.
  2. Using underlying interfaces for functionality and composing:
  3. This might be a misunderstanding on my part in how Kafka works, but I don't see a way to configure a group-leader, or where that's even happening now?

As for my Kafka Streams port, it's very early, and mainly a copy/paste of the Java code for now. But it feels like a Kafka Streams library for C# would fit in quite well, since C# devs are already used to that look and feel of programming due to Linq. I enjoyed your blog posts on exploring streams, thanks for writing those up.

In parallel to the Kafka Streams port, I believe a couple extra libraries can be hit on the way, an AspNetCore library hooking up DI, ILogger, IConfiguration, (Kafka as a ConfigurationProvider, as well as using other ConfigurationProviders to configure this Kafka library), and a Linq Provider. But, we'll see.

Thanks for listening and your help, and I'm looking forward to the future of this library.