search

conduktor / kafka-stack-docker-compose

docker compose files to create a fully working kafka stack
Apache License 2.0
3.28k stars 1.27k forks source link

InconsistentClusterIdException: The Cluster ID doesn't match stored clusterId #105

Closed AdrienPoupa closed 2 years ago

AdrienPoupa commented 2 years ago

Hello,

I am using the single kafka, single Zookeeper configuration here: https://github.com/conduktor/kafka-stack-docker-compose/blob/v6.2.1/zk-single-kafka-single.yml

For the most part it works fine, except sometimes when I restart the containers I am getting this:

kafka.common.InconsistentClusterIdException: The Cluster ID xxx doesn't match stored clusterId Some(yyy) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.

I am guessing the old cluster ID yyy remained there on docker compose down/stop and was not flushed.

The proper fix advised on StackOverflow is supposedly to persist Zookeeper/Kafka data, but I think it's already done with the volumes.

My only workaround at the moment is to prune all the volumes to start over from a clean state, but surely there is a better way to resolve this?

Edit: it seems that the latest version removed volumes support. Was it removed because of this issue, or because of something else?

Thank you!

twobeeb commented 2 years ago

Hello @AdrienPoupa, check out this new cool feature : https://www.conduktor.io/the-fastest-way-to-start-kafka-ever/

With Conduktor Jumpstart, your data is persisted between restarts

AdrienPoupa commented 2 years ago

I ended up removing the Kafka and Zookeeper volumes, since I did not need to persist them. This effectively resets them every time I run the project but it's not an issue for me and it solved this issue.