search

devshawn / kafka-gitops

🚀Manage Apache Kafka topics and generate ACLs through a desired state file.
https://devshawn.github.io/kafka-gitops
Apache License 2.0
317 stars 71 forks source link

Idempotency issue #49

Closed swapnilborade closed 3 years ago

swapnilborade commented 3 years ago

For some topics , we are facing idempotency issue. It creates topic first time but during second run we get below issue.

I am not able to reproduce similar issue on other kafka cluster.

./kafka-gitops -v -f new.yml --no-delete apply Executing apply...

03:39:06.846 [main] INFO com.devshawn.kafka.gitops.config.KafkaGitopsConfigLoader - Kafka Config: {bootstrap.servers=localhost:9092, client.id=kafka-gitops} 03:39:06.853 [main] INFO com.devshawn.kafka.gitops.service.ParserService - Parsing desired state file... 03:39:07.606 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic test-OWN-0 does not exist; it will be created. Applying: [CREATE]

[ERROR] Error thrown when attempting to create a Kafka topic: org.apache.kafka.common.errors.TopicExistsException: Topic 'test-OWN-0' already exists.

[ERROR] An error has occurred during the apply process. [ERROR] The apply process has stopped in place. There is no rollback. [ERROR] Fix the error, re-create a plan, and apply the new plan to continue.

devshawn commented 3 years ago

Hi @swapnilborade,

Without more information, I'm not sure I have much advice to offer. Do you have security on this cluster? My first guess would be the permissions for the user running kafka-gitops does not have the correct ACLs. This could make it so when we fetch the list of topics for the cluster, we do not get all of them, thus, kafka-gitops would try to re-create a topic that already exists.

See the documentation for the required ACLs for kafka-gitops here: https://devshawn.github.io/kafka-gitops/#/permissions?id=state-file-definition

If it's not permissions related, I think we'd need to know more about your setup and try to reproduce it.

devshawn commented 3 years ago

It may help to run plan and apply separately and output the plan as a file to investigate further.

danielprinz commented 3 years ago

Thanks. Had a similar issue. Applying the correct permissions of the git-ops user solved it 🎉 => doc

devshawn commented 3 years ago

@swapnilborade Let me know if you are still seeing issues after looking into the permissions for your gitops user. Thanks!