ClickHouseKeeper example does not work

[janeklb]

I'm unable to use the https://github.com/Altinity/clickhouse-operator/blob/master/docs/chk-examples/02-extended-3-nodes.yaml example to produce a working ClickHouse Keeper cluster.

The issue's I've bumped into:

• ~no dataVolumeClaimTemplate specified~ fixed here https://github.com/Altinity/clickhouse-operator/commit/037483707c63366ddd0bc264dc0c03b657713bcd
• when clickhouse keeper starts on each of the pods, it expects to find an SSL cert

  Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-keeper/server.key: error:80000002:system library::No such file or directory (version 24.10.1.2812 (official build))

• When the ClickhouseKeeperInstallation resource is created, two (of the three) stateful sets are created at the same time. Once they are both "ready", the 3rd is created. When the 3rd is created, the PVC bound / attached to the 1st stateful set of the keeper goes into a "Terminating" state (and stays that way indefinitely)
  • I can see a Delete PVC: ... log (from worker-deleter.go:114 of operator v0.24.0) indicating the the operator is trying to delete the PVC
  • my volumeClaimTemplate has reclaimPolicy: Retain
• If the ClickhouseKeeperInstallation is deleted, the PVC that was in a "Terminating" state is also deleted; however, the other two remain in place (and stay "Bound")

Also note:

• podAntiAffinity is set up with app label, but the pod template doesn't include such a label (doesn't prevent the keeper from working, but it's incorrect)

[janeklb]

BTW I found that the issue with the PVC getting set to Terminating can be fixed by using provisioner: Operator.

Do you have any documentation around why the StatefulSet is the default provisioner, and when it should / should not be used?