[Question] Can you specify serializers per topic?

[rsaltrelli]

Some of my topics have string keys while others have Avro keys. Similarly, most of my topics have Avro values but some have JSON values. The following configuration implies that you can only specify one type of serializer/deserializer per cluster. Is there a way to perhaps override these configurations on a topic-by-topic basis?

clusters:
    default:
        registry: "http://localhost:8081"
        props:
            bootstrap.servers: "localhost:9092"
            key.deserializer: "org.apache.kafka.common.serialization.StringDeserializer"
            value.deserializer: "io.confluent.kafka.serializers.KafkaAvroDeserializer"
            key.serializer: "org.apache.kafka.common.serialization.StringDeserializer"
            value.serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"

Is there a way to perhaps override these configurations on a topic-by-topic basis like this?

clusters:
    default:
        registry: "http://localhost:8081"
        props:
            bootstrap.servers: "localhost:9092"
            key.deserializer: "org.apache.kafka.common.serialization.StringDeserializer"
            value.deserializer: "io.confluent.kafka.serializers.KafkaAvroDeserializer"
            key.serializer: "org.apache.kafka.common.serialization.StringSerializer"
            value.serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
        topics:
            topic1:
                name: topic1-long-name
            topic2:
                name: topic2-long-name
                key.deserializer: "io.confluent.kafka.serializers.KafkaAvroDeserializer"
                key.serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"

So topic1 would use StringDeserializer and StringSerializer for its keys as dictated by the cluster configuration, but topic2 would use KafkaAvroDeserializer and KafkaAvroSerializer instead.

[wlezzar]

Indeed, it's something that would be useful. I will iterate on this and propose a PR soon.

[wlezzar]

@rsaltrelli I created a PR in this regard, can you check it out and tell me if this answers the requirement you had in mind? https://github.com/adevinta/zoe/pull/10