segmentio / kafka-go

Kafka library in Go
MIT License
7.61k stars 789 forks source link

Deserialize Protobuf payload based on 4-byte Schema ID from Schema Registry #612

Open ericsun2 opened 3 years ago

ericsun2 commented 3 years ago

Support for Protobuf with Confluent Schema Registry If the Kafka event is serialized in Protobuf and a 4-byte integer schema_id is put in front of the Protobuf binary stream, can kafka-go invoke the REST API of Confluent Schema Registry to fetch the .proto schema text from Schema Registry, and then form a decoder/deserializer on-the-fly to deserialize the Protobuf binary stream?

Supporting documentation

jandyalaphani commented 2 years ago

Any update on this

jordaniversen commented 2 years ago

Poke

achille-roussel commented 2 years ago

Hello, thanks for starting this conversation.

kafka-go exposes the content of messages as an opaque byte array, the package currently does not unmarshal nor interpret the data.

From where we stand as maintainers of kafka-go, this functionality appears to deserve a dedicated package since it does not have a tight dependency to kafka-go itself, any byte array could be decoded from schemas fetched from remote locations.

We would happily provide visibility to such package in kafka-go, or potentially host it as a submodule if we have a convincing case for it. At this time such a change will need to come from a community contributor, the core maintainer team will not be able to allocate development time to build this feature.

Let me know what you think!