[Question] ...Can we create compact topics in strimzi kafka and do they work the way that we can replace the value via key

[ami11singh]

I am using strimzi kafka on kubernetes cluster and want to delete a specific messages from Kafka topic. I read that Compacted topics allow messages to be explicitly deleted or replaced via their key.

[scholzj]

Strimzi support compacted topic as any other Kafka installation (it has real Kafka inside). You can just configure the retention policies and compacting though the Topic Operator or directly in Kafka. It will work exactly the same as any other compacted Kafka topic which AFAIK is that when you send a message with the same key but a null body, it will tell the broker you are deleting the key when compacting. The message will stay there for some time to indicate the deletion to all consumers - how long that is is configurable in configuration.

[ami11singh]

Thanks.

[ami11singh]

I have an additional question, so have re-opened this issue instead of creating a new thread. Hope, that's fine! I put the policies and compaction configurations in topic operator section of Kafka cluster yaml file and created a topic with compaction parameters in config section. It worked fine, when multiple values of a single key are produced on that topic, latest key value is being consumed as expected.

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  namespace: my-kafka
  name: my-cluster
spec:
  kafka:
    version: 2.3.0
    replicas: 3
    listeners:
      plain: {}
      tls: 
        authentication:
          type: tls
      external:
        type: nodeport
        tls: true
        authentication:
           type: tls 
        overrides:
          bootstrap: 
            nodePort: 32100
          brokers:
          - broker: 0
            nodePort: 32000
          - broker: 1
            nodePort: 32001
          - broker: 2
            nodePort: 32002
    authorization:
      type: simple     
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.3"
    storage:
      type: persistent-claim
      class: applications
      size: 10Gi
      deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      class: applications
      size: 10Gi
      deleteClaim: false
  entityOperator:
    topicOperator:
      cleanup.policy: compact
      log.cleanup.policy: compact
      delete.retention.ms: 100 
      segment.ms: 100
      min.cleanable.dirty.ratio: 0.01
      log.cleaner.backoff.ms: 0.01
    userOperator: {}

When I created a topic without compaction parameters, it is working normally and no compaction is happening. I want to confirm if this is how it is meant to to work, because in future I will want some topics with compaction and some without, so if topic operator has these compaction settings, will it work based on the topic having compaction parameters set or not? Please suggest.

[scholzj]

This is not a way how you configure the topic operator. These will be ignored. What do you want to do? Have this úpolicies set as default? If yes, you should use it in the Kafka.spec.kafka.config section.

[ami11singh]

I need a way in which I can create both compacted topics and topics without compaction based on the configuration I provide in the config section of topic yaml file. Is that achievable? And how can I configure Kafka resource to support this?

[scholzj]

Ok. You just configure it in the KafkaTopic resource in the KafkaTopic.spec.config options. The configuration you specify there is valid only for given topic, so whatever you specify there will be created.

[ami11singh]

Thanks! I will try this!!

[ami11singh]

It's working..thanks for the help!