Kafka client

[zwubs]

I have a Kafka client I started on called nectar, it's been a while since I worked on it. I might come back to it some day, but for now other projects have taken my interest. But if someone wants to fork and run with it, feel free!

[lpil]

Awesome!

[zwubs]

For this client, would the hope be that it work for both the Erlang & JavaScript targets? In nectar I was planning on pretty heavily making use of actors, and also there's a lot of non-byte aligned decoding & encoding going on.

[lpil]

I suspect there's not much one could share between the two targets here, but if there's any protocol stuff which could be shared that would be great.

[zwubs]

Good call out! I'd stubbed out a project (ambrosia) to work on some codegen for making decoders and encoders. The Kafka protocol gets a bit weird with its versioning, I'm not sure the best way to represent that in Gleam. But, if I never get to it here's some resources I've come across that could help someone else get a protocol library up and running:

• Apache keeps all their protocol messages in (what I think is) a proprietary JSON format in their Kafka Github repo .
• The project kafka-protocol-rs has been a great reference for protocol stuff, and it's where I got the initial idea for codegen.
• And if someone is just getting started, Apache's Kafka protocol documentation is a huge help.