Re-Installing Kafka fails - Cluster id doesn't match

[paulocabrita-ionos]

Affected version

23.4.1

Current and expected behavior

Current: the reinstallation of the Kafka fails with the error "The Cluster ID 4a...... doesn't match stored clusterid Some(Qgt.....) in meta.properties. Expected: reinstallation of a Kafka cluster without issue.

Steps done:

• created an IONOS K8S cluster;
• installed stackable's operators (version 23.4.1);
• created 2 yaml files (zookeeper and kafka);
• applied both of them and everything worked perfectly;
• deleted zookeeper and kafka using the yaml files;
• trying to install again and it does not work (Cluster ID error);

Possible solution

No response

Additional context

(With @adwk67 )

Some measures applied:

• deleted CRD's;
• uninstall/installed the operators; The problem remains.

Related:

• https://amalgjose.com/2021/05/28/kafka-error-cluster-id-doesnt-match-stored-clusterid-in-meta-properties/

Environment

Client Version: v1.25.3 Kustomize Version: v4.5.7 Server Version: v1.25.6

Would you like to work on fixing this bug?

None

[adwk67]

Reproducable in a local kind cluster (using zookeeper 23.4.0 and bumping kafka from 23.4.0 to 23.4.1):

kind delete clusters --all
kind create cluster
stackablectl op in zookeeper=23.4.0 secret=23.4.0 commons=23.4.0 kafka=23.4.0

# 23.4.0 versions
kubectl apply -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  image:
    productVersion: 3.8.0
    stackableVersion: 23.4.0
  servers:
    roleGroups:
      primary:
        replicas: 1
        config:
          myidOffset: 10
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-zk-znode
spec:
  clusterRef:
    name: simple-zk
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-kafka-znode
spec:
  clusterRef:
    name: simple-zk
---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
spec:
  image:
    productVersion: 3.3.1
    stackableVersion: 23.4.0
  clusterConfig:
    zookeeperConfigMapName: simple-kafka-znode
  brokers:
    roleGroups:
      default:
        replicas: 1
EOF

# wait until all pods are ready...

kubectl delete -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-kafka-znode
---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
EOF

# wait until all pods are terminated...

stackablectl op un kafka
# update kafka CRD
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.4.1/deploy/helm/kafka-operator/crds/crds.yaml

stackablectl op in kafka=23.4.1

# 23.4.1 version
kubectl apply -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-kafka-znode
spec:
  clusterRef:
    name: simple-zk
---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
spec:
  image:
    productVersion: 3.3.1
    stackableVersion: 23.4.1
  clusterConfig:
    zookeeperConfigMapName: simple-kafka-znode
  brokers:
    roleGroups:
      default:
        replicas: 1
EOF

# end of reproduceable example
# kafka.common.InconsistentClusterIdException: The Cluster ID LrWO5WmGSLKWuKoIyJhvew doesn't match stored clusterId Some(KuPjNmJjQXqWyfA20Ovj2A) in meta.properties.

kubectl delete -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-kafka-znode
---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-zk-znode
EOF

# now delete the pvcs too:
# data-simple-zk-server-primary-0
# log-dirs-simple-kafka-broker-default-0

kubectl delete pvc data-simple-zk-server-primary-0
kubectl delete pvc log-dirs-simple-kafka-broker-default-0

kubectl apply -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  image:
    productVersion: 3.8.0
    stackableVersion: 23.4.0
  servers:
    roleGroups:
      primary:
        replicas: 1
        config:
          myidOffset: 10
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-zk-znode
spec:
  clusterRef:
    name: simple-zk
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-kafka-znode
spec:
  clusterRef:
    name: simple-zk
---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
spec:
  image:
    productVersion: 3.3.1
    stackableVersion: 23.4.1
  clusterConfig:
    zookeeperConfigMapName: simple-kafka-znode
  brokers:
    roleGroups:
      default:
        replicas: 1
EOF

[adwk67]

The fix above (i.e. everything after # end of reproduceable example) can be simplified to just deleting the kafka cluster and znode, deleting the logs pvc and re-deploying the kafka cluster and znode:

kubectl delete -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-kafka-znode
---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
EOF

# now delete the pvc too: log-dirs-simple-kafka-broker-default-0
kubectl delete pvc log-dirs-simple-kafka-broker-default-0

kubectl apply -f - <<EOF
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-kafka-znode
spec:
  clusterRef:
    name: simple-zk
---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
spec:
  image:
    productVersion: 3.3.1
    stackableVersion: 23.4.1
  clusterConfig:
    zookeeperConfigMapName: simple-kafka-znode
  brokers:
    roleGroups:
      default:
        replicas: 1
EOF

[paulocabrita-ionos]

I think the PVC is the issue. I deleted the PVC as you said, installed again and... it worked like a charm.

[adwk67]

/stackable/config/server.properties includes log.dirs=/stackable/data/topicdata and /stackable/data/topicdata contains the /stackable/data/topicdata/meta.properties file with contents such as cluster.id=DN1ekhLQQm68zv7-MsMriw, so it makes sense that deleting the logs pvc would clear the value of cluster.id.

[adwk67]

To summarise:

• the observation can re reproduced (see above)
• there is a workaround (delete logs pvc) that "makes sense" (in that the clusterId is indeed in a file in the logs pvc)

Still to do:

• determine if it is expected behaviour that re-deploying the kafka operator and cluster should result in a "stale" clusterId
• if not, investigate why this is and open a separate issue to fix this

[adwk67]

This issue can be closed as deleting the ZNode (simple-kafka-znode) removes the cluster co-ordination state: if the ZNode is not deleted, the other steps (removing the kafka cluster, upgrading the operator and then deploying the kafka cluster) do not require the logs pvcs to be removed.