search

segmentio / topicctl

Tool for declarative management of Kafka topics
MIT License
590 stars 55 forks source link

Delete topic settings from broker if they are not present in the topic config #198

Open fpacifici opened 3 months ago

fpacifici commented 3 months ago

Related with issue https://github.com/segmentio/topicctl/issues/197.

The idea of this PR is to make topic apply a bit more declarative by supporting deletion of settings that are not present in config. As this could be a calamity if done by mistake, the behavior is disabled by default. A CLI argument --destructive is supported in the apply subcommand. If that flag is not set, the behavior is the same as it was before.

Example:

topicctl % ./build/topicctl apply --destructive  examples/local-cluster/topics/topic-default.yaml
{{topic-default local-cluster local-region local-env Topic that uses default (any) strategy for assigning partition brokers.
 map[] []} {3 2 0 map[cleanup.policy:delete max.message.bytes:5.54288e+06 message.timestamp.type:LogAppendTime] {in-rack  [] []} <nil>}}
[2024-06-07 16:53:35]  INFO Processing topic topic-default in config examples/local-cluster/topics/topic-default.yaml with cluster config /Users/filippopacifici/code/topicctl/examples/local-cluster/cluster.yaml
[2024-06-07 16:53:35]  INFO Starting apply for topic topic-default in environment local-env, cluster local-cluster
[2024-06-07 16:53:35]  INFO Validating configs...
[2024-06-07 16:53:35]  INFO Checking if topic already exists...
[2024-06-07 16:53:35]  INFO Updating existing topic 'topic-default'
[2024-06-07 16:53:35]  INFO Checking the existing state of the cluster, topic, and throttles...
[2024-06-07 16:53:35]  INFO All replicas are in-sync
[2024-06-07 16:53:35]  INFO Checking topic config settings...
[2024-06-07 16:53:35]  INFO Found 1 key(s) with different values:
-------------------------+----------------------+---------------------
           KEY           | CLUSTER VALUE (CURR) | CONFIG VALUE (NEW)  
-------------------------+----------------------+---------------------
  message.timestamp.type |                      | LogAppendTime       
-------------------------+----------------------+---------------------
[2024-06-07 16:53:35]  INFO Found 1 key(s) set in cluster but missing from config for deletion:
---------------+--------------------
      KEY      |   CLUSTER VALUE    
---------------+--------------------
  retention.ms | 6000000 (100 min)  
---------------+--------------------
OK to update to the new values in the topic config? (yes/no)

Typing yes the retetion.ms setting is deleted. If I run the command again there is no more diff to apply.

Test plan: