segmentio / topicctl

Tool for declarative management of Kafka topics
MIT License
590 stars 55 forks source link

Validation fails for cross-rack topics #138

Closed joewreschnig closed 1 year ago

joewreschnig commented 1 year ago

We use check --validate-only in our CI pipeline for our topicctl configurations. We recently introduced some cross-rack assignments, and it seems like it's incompatible with check in v1.9.1:

$ topicctl check --cluster-config production.cluster.yaml --validate-only production/dataset.yaml 
[2023-05-30 17:10:31]  INFO Check failed for topic dataset (cluster=production, env=production):
-----------------+----+--------------------------------------------------------------------------
       NAME      | OK |                                 DETAILS                                  
-----------------+----+--------------------------------------------------------------------------
  config correct | ✗  | config validation error: 1 error occurred:                               
                 |    |     * Replication factor (3) cannot be larger than the number of racks (-1)  
                 |    |                                                                          
                 |    |                                                                          
-----------------+----+--------------------------------------------------------------------------
[2023-05-30 17:10:31] ERROR Check failed for 1/1 topic configs

Even when not using --validate-only it fails:

$ topicctl check --cluster-config production.cluster.yaml production/dataset.yaml 
[2023-05-30 17:07:46]  INFO Checking cluster ID against version in cluster
[2023-05-30 17:07:46]  INFO Check failed for topic dataset (cluster=production, env=production):
-----------------+----+--------------------------------------------------------------------------
       NAME      | OK |                                 DETAILS                                  
-----------------+----+--------------------------------------------------------------------------
  config correct | ✗  | config validation error: 1 error occurred:                               
                 |    |     * Replication factor (3) cannot be larger than the number of racks (-1)  
                 |    |                                                                          
                 |    |                                                                          
-----------------+----+--------------------------------------------------------------------------
[2023-05-30 17:07:46] ERROR Check failed for 1/1 topic configs

In the case of --validate-only, it should probably skip this check. When not using --validate-only, I expect it should pull the actual number of racks (in this case, 4) from the cluster.

petedannemann commented 1 year ago

Hi @joewreschnig, your proposal sounds good to me. Could you submit a PR for the fix?

joewreschnig commented 1 year ago

I submitted #140 concerning this.

petedannemann commented 1 year ago

Fixed by https://github.com/segmentio/topicctl/pull/140