magic byte ! : error while trying to use the confluent avro consumer to read messages written with the 'toAvro' method

[ghaithSN]

i used 'toAvro' to write a stream to kafka topic but when i tried to see the content of messages with the confluent avro console consumer i got the error shwon in the capture any idea ?

[felipemmelo]

Hi there. Yes, the issue is that Confluent Avro serializer adds the id of the schema to the payload, thus, its deserializer expects that id to be there.

The library is able to read from Confluent Avro and extract this id, but not to append this id when writing, since that would imply tight coupling to Schema Registry for schema evolution tracking, etc. And although the library supports reading from and writing to Schema Registry, it is not quite clear yet if schema management should be one of the its responsibilities.

On the other hand, we've received some requests for this feature, thus, we'll start to investigate it. Also, the next version will bring a standalone, command line consumer to deal with this situation while the toConfluentAvro method is not added to the API.

Cheers.

[ghaithSN]

thanks for the answer, yah that command line consumer would be useful to verify if the messages are correctly written to the kafka topic

[felipemmelo]

Hi there, ABRiS now has an API that allows you to append the id of your schema to the Avro payload, so that it can be consumed by Confluent tools. Also, the schema is automatically registered if not yet in Schema Registry or updated otherwise.

You will find an example here.