search

confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client
https://github.com/confluentinc/confluent-kafka-dotnet/wiki
Apache License 2.0
18 stars 852 forks source link

A Kafka Extensions library #126

Closed zoalasaurusrawr closed 6 years ago

zoalasaurusrawr commented 7 years ago

Hello Confluent!

I wanted to share a project I've been working on. Our company was using RdKafka before but we have switched over to use the Confluent dotnet library because we like how raw and expose the API surface is. As a result, I've been working on an extensions library and I wanted to share it with you! The project page is:

https://github.com/wmccullough/Kafka.Extensions

The goal for me is to provide a series of extensions to help ease implementation and speed adoption of the Confluent.Kafka library. I'm not sure at which level Confluent intends to add features to your own library, but I hope I can help bridge the gap. The library is .NET Standard 1.3, so it should easily cross compile to .NET core or .NET FX.

Just wanted to introduce myself and let you know that you have a fan!

glenthomas commented 7 years ago

I like the fluent API style of configuring your consumers. If you put out a NuGet package I would be interested in integrating with my application that is using Confluent DotNet.

It seems to only support a single topic (and partition and offset) to be configured per consumer instance?

mhowlett commented 7 years ago

cool - very interesting.

There is absolutely scope for a layer on top of the Confluent.Kafka Consumer - the API is verbose and it's pretty easy to abstract a lot of this away (I also do this in my own projects). I hadn't considered System.Reactive, will give that some thought. Note that in Confluent.Kafka we believed it was important to provide an API close to the one provided by librdkafka for full flexibility.

Ultimately we want to have a Kafka Streams implementation across languages (the streams API is currently Java only). I'm particularly excited about C# here as we have some wonderful language constructs to play with.

Please keep posting updates here - we will likely link out to your project from the main readme if it continues looking good.

Also, if there is anyone else out there with other ideas for a higher level API, also please post - i'm excited to see what people come up with.

zoalasaurusrawr commented 7 years ago

@mhowlett Thanks for the nod! I will definitely keep adding to it. I totally get trying to maintain some manner of congruency between Confluent.Kafka and librdkafka. I've seen firsthand how one change in an underlying implementation causes ripple effects that can be hard to reconcile across abstractions. That said, I'm certainly happy to keep building extensions on top when it doesn't make sense for Confluent to adopt a specific design in the Confluent.Kafka library.

I'm thinking that next, I may add support for Microsoft.Logging.Extensions so that dotnet users can easily hook their favorite logging frameworks right on top of the reactive subscriber. I'm also not enabling multiple topics per consumer, and I know that someone will want that, so I need to modify the builder to enable that too.

Keep up the good work!

@MrGlenThomas I'm hoping that this week I can get everything hooked up to AppVeyor so that I can publish out to Nuget. If I don't, I'll just manually push the first version myself.

mhowlett commented 6 years ago

IObservable<T> would change the consumer from a pull to push model, which I don't think is what we want. I think what we probably ideally want is async streams coming in C# 8 (async enumerables).