search

strimzi / strimzi-kafka-operator

Apache Kafka® running on Kubernetes
https://strimzi.io/
Apache License 2.0
4.81k stars 1.28k forks source link

[Bug]: Invalid cluster.id error when initializing a new KRaft cluster with existing volumes #10722

Open fvaleri opened 15 hours ago

fvaleri commented 15 hours ago

Bug Description

When Kraft disks are already formatted, we have to reuse the previous cluster.id in addition to the --ignore-formatted flag. This is required by the Kafka storage tool in order to check that cluster.id is the same across all broker disks.

When Kafka pods are restarted, the operator reuses the existing cluster.id from Kafka CR status and skips disk formatting. Instead, when initializing a new Kafka cluster, there is no status, so it always generates a new cluster.id and runs disk formatting.

This is a problem when rebinding existing volumes from a previous Kafka cluster instance because they already have a cluster.id stored in meta.properties. In this case, Kafka pods start crash looping with the following error:

Exception in thread "main" java.lang.RuntimeException: Invalid cluster.id in: /var/lib/kafka/data/kafka-log0/meta.properties. Expected kQHv733NQIew9aw9uCXnDA, but read 2DLef4_8TqOVdBjzCORB1Q

Steps to reproduce

See https://github.com/strimzi/strimzi-kafka-operator/pull/10637#discussion_r1780956446.

Expected behavior

Assuming the binding is correct, the Kafka pods should be able to work with volumes from a previous Kafka cluster instance.

Strimzi version

main

Kubernetes version

1.30

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

scholzj commented 14 hours ago

The expectation is that users set the cluster ID in the Kafka CR status as part of the recovery. There is no expectation to recover it from the disks. This might need to be documented, but it should not be fixed by reading any cluster IDs from the volumes.

fvaleri commented 14 hours ago

This might need to be documented, but it should not be fixed by reading any cluster IDs from the volumes.

Definitely, because the current documented procedure is broken: https://strimzi.io/docs/operators/latest/deploying#proc-cluster-recovery-volume-str.

scholzj commented 14 hours ago

Definitely, because the current documented procedure is broken: https://strimzi.io/docs/operators/latest/deploying#proc-cluster-recovery-volume-str.

It is not broken, just old. Let's keep the issue to track the docs improvement.