Apache Kafka® Protocol for go and tinygo

This is a fork of the Apache Kafka® protocol implementation from grepplabs/kafka-proxy. This fantastic work is a great donor because it:

• Uses minimal reflection.
• Does not use struct tags.
• Does not use goroutines.
• Does not use channels.

Which results in:

• Being able to compile with tinygo-org/tinygo.
• Because it can be compiled with tinygo, it can be compiled with tetratelabs/proxy-wasm-go-sdk.
  • Which means that it can be used as a foundation for Envoy WebAssembly Layer 7 proxies.

Why forking?

The upstream Kafka Proxy protocol implements sufficient encoding and decoding to support its own functionality, but does not implement everything else required to parse messages beyond request headers, response headers, Metadata, and FindCoordinator. The maintainer of Kafka Proxy isn't keen on supporting anything in the protocol encoder/decoder that does not find a direct use in the upstream project.

The goal of this fork is to add support for all Apache Kafka® messages.

Message support

☑ Headers

General notes

• Compact arrays have been added in Apache Kafka® 2.4.
• Flexible messages support tag buffers. Flexibility flag is declared per message, any structure schema type includes a tag buffer.
• Strings and arrays for flexible messages by default use compact versions of their respective types; reference: https://github.com/apache/kafka/blob/3.2.0/generator/src/main/java/org/apache/kafka/message/SchemaGenerator.java#L224-L322.

License

Apache 2.0.

Run tests

go test -count=1 -v ./...

Generating messages

1. Setup working environment:

export KAFKA_SOURCE_ROOT="${HOME}/dev/my/kafka"

2. Close Kafka sources somewhere on disk and checkout the release you want to use:

mkdir -p "${KAFKA_SOURCE_ROOT}"
cd "${KAFKA_SOURCE_ROOT}"
git clone https://github.com/apache/kafka.git .
git checkout 3.2.0
cd -

3. Run the generator command:

make generate-request-types generate-response-types