Manually apply avro schema to specific topics/messages

[mrjackbo]

Thanks for the useful tool. This is a feature request.

In my application, I do not serialize the avro schema reference in the message header, but all producers and consumers explicitly know which schema belongs to which topic, and they pull the schema from the registry only once at startup.

This means that kafkahq cannot deserialize the messages, because it cannot choose the schema automatically. It would be useful to have an option in the topic/message view gui, which allows the user to choose a schema from the registry with which the messages should be deserialized.

[tchiotludo]

Hello,

I think I see your feature request ! You use Kafka Schema Registry, but your consumer / producer have a mapping from topic to schema id, right ? In this case, you just serialize only the avro content to Kafka (without magic bytes with schema id like a standard app) ?

If it true ,what you think about configuration (on application.yml) with topic name & schema id in order to decrypt avro by configuration instead of ui ?

[mrjackbo]

You use Kafka Schema Registry, but your consumer / producer have a mapping from topic to schema id, right ? In this case, you just serialize only the avro content to Kafka (without magic bytes with schema id like a standard app) ?

Correct. We have had issues since the Hortonworks registry and the Confluent registry use different magic bytes/headers, and various libraries are developed with different registries in mind, so we just skipped writing the headers altogether.

Configuration via the yaml file would be great.

[tchiotludo]

Ok noted ! It's a little bit specific but not a huge work 👍

[tsavenko]

If it true ,what you think about configuration (on application.yml) with topic name & schema id in order to decrypt avro by configuration instead of ui ?

I am ok doing that, but it doesn't seem to work. There are no errors displayed in the log and the application starts normally. However, it doesn't seem to be serializing the messages (I am trying to post messages via UI and have them go through Avro serializer) Here's the configuration I tried:

# default kafka properties for each clients, available for admin / producer / consumer 
(optional)
  clients-defaults:
    consumer:
      properties:
        isolation.level: read_committed
    producer:
      schema-registry:
        url: "http://localhost:8081"
      schema.id: 61
      kafka.topic.name: testtopic
      properties:
        key.serializer: org.apache.kafka.common.serialization.StringSerializer
        value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer

@tchiotludo I hope I'm just messing up this configuration and that's why it doesn't work :) Thanks in advance for your input!

[tchiotludo]

Hey @tsavenko ,

The issue is not closed meaning that the features is not available 🥴 Are you on the same case ? With a registry ? But without direct connection between producer & SR ?

[tsavenko]

ah, my bad... I thought you were suggesting a solution that would work with the existing code base :) my problem is that kafkahq does not actually serialize messages with avro. The messages get posted alright, but the consumer fails to deserialize them

[tchiotludo]

so your issue is not about deserialize but about serialize :smile: do you use a SR ? if yes there is a selectbox allowing you to choose the schema during produce :

[tsavenko]

What's "SR"? Schema Registry? If so, yes, we are using it. I don't see that checkbox anywhere :(

[tchiotludo]

it's on produce page at the bottom:

If you don't see the button, it must be disabled by configuration

[tsavenko]

it's not seeing any topics :/ no errors anywhere, everything seems connected...

.. and yet everything is peachy with the old version (0.11.0):

Although looks like the old version doesn't have the sweet checkbox I so desperately need :D

[tchiotludo]

You can try with the new ui (maybe the dev version is better) ? The old ui is remove on the dev version and I don't support the old one for now)

[tsavenko]

@tchiotludo after a few sleepless nights, I finally got the configuration right and now I am happily able to produce Avro-serialized messages through UI! YAY!!! Last question though: is it possible to post MULTIPLE messages? or is it only 1 message at a time?

[tchiotludo]

Glad to know it 😊 But for now, there is no way to send multiple messages. If you have some idea about the ui needed for that (or better a PR), I will be thankful 😀

[tchiotludo]

closing since it works for you