Kafka creates a PVC which contains a reference to a ZNode: this information is not updated automatically when the ZNode is deleted. Thus, when terminating and re-creating a Kafka cluster there are at least 3 use-cases which should be considered and clearly documented:-
retain an existing ZNode (same SDP release)
in this case, just remove and re-create a kafka cluster, ensuring that the kafka ZNode and PVCs are also retained
depending on the scope of any breaking changes, it may be possible to upgrade SDP and re-create clusters without having to touch the PVCs
in cases where deleting the aforementioned PVC is unavoidable, it should be made clear that this will also result in the loss of topic offset metadata.
This should be included in the next release (23.11).
As a user, I want to have documentation to make it clear what should be done to a) retain, or b) replace the way in which Kafka references ZooKeeper.
Background (see issue here)
Kafka creates a PVC which contains a reference to a ZNode: this information is not updated automatically when the ZNode is deleted. Thus, when terminating and re-creating a Kafka cluster there are at least 3 use-cases which should be considered and clearly documented:-
cluster.id(see https://github.com/stackabletech/kafka-operator/issues/609#issuecomment-1658232792) , so that a new value can be created that points to the newly created ZNodeThis should be included in the next release (23.11).