adevinta / zoe

The Kafka CLI for humans
https://adevinta.github.io/zoe
MIT License
287 stars 21 forks source link

Add ability to override topics props #10

Closed wlezzar closed 4 years ago

wlezzar commented 4 years ago

This PR adds the ability to override Kafka clients properties at a topic level.

For an example, checkout: https://github.com/adevinta/zoe/pull/10/files#diff-70b92abb2c7fdc4cedea36503ba7c404R41-R44

rsaltrelli commented 4 years ago

Awesome! So am I correct in assuming that this also lets me set topic configuration in the yml file?

clusters:
    default:
        registry: "http://localhost:8081"
        props:
            bootstrap.servers: "localhost:9092"
            key.deserializer: "org.apache.kafka.common.serialization.StringDeserializer"
            value.deserializer: "io.confluent.kafka.serializers.KafkaAvroDeserializer"
            key.serializer: "org.apache.kafka.common.serialization.StringSerializer"
            value.serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
        topics:
            topic1:
                name: topic1-long-name
            topic2:
                name: topic2-long-name
                propsOverride:
                    key.deserializer: "io.confluent.kafka.serializers.KafkaAvroDeserializer"
                    key.serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
                    retention.ms: -1
                    cleanup.policy: compact        
wlezzar commented 4 years ago

~yes exactly~

Awesome! So am I correct in assuming that this also lets me set topic configuration in the yml file?

@rsaltrelli sorry I misunderstood your question: this override is not about the topic create properties, it's rather the consumer properties. So you cannot specify retention.ms and cleanup.policy which are topic create params. Makes sense?

rsaltrelli commented 4 years ago

OK. That makes sense.