Protobuf schemas in apicurio schema registry

[p1ppus]

When using protobuf schemas in our apicurio registry akhq is unable to deserialize the content of the message. Events without schema id can successfully be deserialized with corresponding descriptor files. Only after assigning schema ids akhq is unable to parse the data.

"Error in GUI: Error deserializing Protobuf message for id xx"

The documentation explicitly states for AVRO that messages using schema registry are automatically decoded if the schema registry is configured. But for PROTOBUF there is no such mention, only info about how to create desc files.

Does akhq allow to use (apicurio) schema registry in conjunction with proto schemas or is it not supported?

Parts of our config

akhq:
      connections:
        xxx-kafka:
          deserialization:
            protobuf:
              descriptors-folder: /opt/protodesc
              topics-mapping:
              - descriptor-file: authr_doc.desc
                topic-regex: .*authr_doc
                value-message-type: com.xxxx.xxx.xxx.protobuf.authr_doc.Authr_doc
              #More Proto desc file mappings
          properties:
            bootstrap.servers: SSL://xxx-kafka-bootstrap:9092
            security.protocol: SSL
            #MORE SSL Config
          schema-registry:
            type: confluent
            url: http://apicurio-registry:8008/apis/ccompat/v6

[tchiotludo]

I don't use protobuf for myself, reading the code, only mapping with protobuf file is supported for now.

PR are welcome !

[p1ppus]

Thanks for your reply.

Just to confirm, with mapping you mean the existing use of descriptor files as we already used and like stated here: https://akhq.io/docs/configuration/protobuf.html ?

That would mean for us we can't use our SMT in MM2 (https://github.com/OneCricketeer/schema-registry-transfer-smt) to populate schema IDs (producer has only static proto schemas without registry) and the schema registry pretty much becomes useless when used in akhq with protocol buffers.

Let's see about a PR, have to check about ressources and timeline and if it's even allowed to contribute from our side. Can you provide some implementation hints where and what should be amended in the code?

cheers p1ppus

[tchiotludo]

Just to confirm, with mapping you mean the existing use of descriptor files as we already used and like stated here: https://akhq.io/docs/configuration/protobuf.html ? Yes, exactly

Can you provide some implementation hints where and what should be amended in the code? Seems to be done here: https://github.com/tchiotludo/akhq/blob/7e99277c0766d808ebca1e8f21eadc41f06b7a13/src/main/java/org/akhq/models/Record.java#L170-L188 But reading it right now, it seems that is already supported ... (like JsonSchema also).