search

AndyPook / A6k.Kafka

Fully managed dotnet Kafka client
MIT License
2 stars 1 forks source link

How can I help? #2

Open ChadJessup opened 4 years ago

ChadJessup commented 4 years ago

Hey there, Wondering how I can help out?

Some stuff I can think of that might help out with multiple people poking at things:

  1. CI/CD
  2. dotnet core extension stuff (configuration, DI)
  3. Benchmarks
  4. More tests
  5. Test client(s)

I can also just hop in and start on messages or whatnot.

Thanks!

ChadJessup commented 4 years ago

If wanted, I can also spin up Bedrock.Kafka as @AndyPook mentioned, and make Andy and @timjroberts collaborators, if people want to work in one space.

Like I mentioned in another thread, I'm porting Kafka Streams to C#, and that requires a lot of custom implementations that the Confluent library won't support (internal Types, hard-coded partitioners, etc) so I'm hoping for a pure C# client that end-users can replace functionality as they need. In the end, I'd like something like https://github.com/davidfowl/FeatherHttp or http://nancyfx.org/ but for Kafka Consumers/Producers.

https://github.com/ChadJessup/Kafka.Streams.net/tree/master/Kafka.Streams.net https://github.com/ChadJessup/KafkaHelpers <- WIP: General wrappers around confluent-dotnet to allow for DepInjections, the Microsoft.Extensions.Logger, Configuration, typed consumers/producers, etc.

AndyPook commented 4 years ago

Sounds good to me 😄 I started a Slack team for us three to be able to chat. You should have an invite.

I think all three of us have bits that can do the basic messaging, broker rpc thing. One goal is to get to functional (well behaved) Producer/Consumer clients. The semantics are not that hard but handling the various paths robustly is "tricky".

It would be great to figure out a nice DI/config story. One that allows for multiple clients configured differently (to talk to different topics/clusters). Not straight forward without named services. Confluent.Kafka is "easy" as it just needs a bag of config. New'ing clients up is easy. Not so much with the DI requirements of this new stuff. I have an approach for C.K in A6k.Messaging (which I've used in various places). It allows for fairly simple bootstrapping of microservice like things. Kinda tied together using named config. Anyway, long winded way of agreeing with the need for a good DI story 😄

The whole list you give.... yes please to all of it. But I think it would be "a good thing" to focus on getting something "working" so that we can prove that the "framework" can work nicely with the Kafka style (unless you can dream up some cheap wins?)

I'm imagining the we will iterate a bunch on the underpinnings, style etc throw a load of stuff away, rebuild... Don't want to spend too much time on things that will also need redoing 🤔

Maybe, if your streaming thing is ready for it, we could flesh out the message types required and use that to exercise/prove the rest of it??

Having said all that... if there's anything that you'd enjoy doing, then party on! This isn't a commercial project. It's supposed to be fun, a learning thing 😁