search

kafka-ops / julie

A solution to help you build automation and gitops in your Apache Kafka deployments. The Kafka gitops!
MIT License
422 stars 115 forks source link

Running KTB with validateOnly option #165

Closed akselh closed 3 years ago

akselh commented 3 years ago

It would be very useful to have an option to validate the topology files, and doing so without requiring a connection to the cluster. As it is now validation can only be done by using --dryRun, however this requires a connection towards the Kafka cluster which might not be possible in all cases.

One important use case is to automatically validate changes on a feature branch before PR approval. For security reasons you should not let the credentials for the KTB GitOps user (Kafka user that is) be available on an unprotected branch. So --dryRun is not even an option in this scenario.

Suggestion: Add new option --validateOnly that only loads the topology and runs validations.

Running with this option obviously validates that the topology is valid per KTB requirements/"schema"; topology can be parsed that is. In addition users can provide their own validators with the topology.validations option.

It would however be great if KTB provided at least one validation out-of-the-box:

akselh commented 3 years ago

Update: After this was posted we figured out that we only need dryRun, the way we fixed it was to have a GitOps Kafka user with read-only permissions. Then dryRun on feature branch is ok.

Also with regards to other issues with comments on improvements on dryRun-logging it might be an advantage to have a connection to the cluster during dryRun.