kafka-ops / julie

A solution to help you build automation and gitops in your Apache Kafka deployments. The Kafka gitops!
MIT License
417 stars 113 forks source link

Use different strategies for the key and the value in the same topic #573

Open vascoferraz opened 11 months ago

vascoferraz commented 11 months ago

Hi everyone :)

I read the documentation and I did not find any way to use different strategies, that is, TopicNameStrategy, RecordNameStrategy or TopicRecordNameStrategy, for the key and the value in the same topic.

I also tried the following configurations but them failed while deploying:

First attempt - Add two topics entries (one for the value and another for the key)

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalArgumentException: com.purbon.kafka.topology.exceptions.ValidationException: Missing required value.schema.file on schemas for topic xxx-customer-v1

Second attempt: Duplicate the topic entry and set RecordNameStrategy for the value (first entry) and TopicNameStrategy for the key (second entry).

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalStateException: Duplicate key xxx-customer-v1