search

tchiotludo / akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
https://akhq.io/
Apache License 2.0
3.35k stars 651 forks source link

Support for schema references via header #231

Closed tine2k closed 4 years ago

tine2k commented 4 years ago

Currently, kafkahq supports avro deserialization based on Confluent's wire format (https://docs.confluent.io/current/schema-registry/serializer-formatter.html#wire-format). This format contains 5 bytes in front of the raw avro payload to reference the schema.

Spring Cloud Stream (and maybe other frameworks as well?) uses a different way of referencing the schema. It uses a header with the key "contentType" to reference the schema by subject and version (in the format "application/vnd.\<subject>.v\<version>+avro").

It would be useful to support this way of referencing the schema as well. This should be quite straightforward to do by converting subject+version in the header to the schema id by calling the schema registry beforehand.

I am happy to provide a PR for this, if this feature is of interest to the project.

tchiotludo commented 4 years ago

Hi,

If I correctly understand, the support will be a micro patch in Record class with a if header is present, use another deserializer ? If true, you can send a PR and I will merge it 👍

tine2k commented 4 years ago

I would probably move the schema determination logic from Record to RecordRepository and pass the Schema and id to the Record instance. That way, I can use the same deserializer as before by passing the schema to deserialize().

tchiotludo commented 4 years ago

Ok or maybe a service for that ? Also take care of this PR : https://github.com/tchiotludo/kafkahq/pull/228 That can clash with yours

tine2k commented 4 years ago

Ok or maybe a service for that ?

With service you mean a new layer between controllers and repositories? If yes, then this is quite difficult to do without completely refactoring RecordRepository because the new logic will be quite deep inside the repository. Furthermore, I believe there are some design choices to be made beforehand as I don't see an obvious design what part/responsibility of the repository should be move to the new service.

tchiotludo commented 4 years ago

Ok 😁 Just an idea without looking further the code. Go for RecordRepository.

It's Sunday, I'm not really awake 🥴