Open Hubbitus opened 1 month ago
@Hubbitus , have you used latest 0.23.6 or earlier release?
@alex-zaitsev, thank you for the response.
That was in older version. Now we have updated operator. What is a correct way to re-init node? Is it enough to just delete PVC of failed node and delete POD?
@Hubbitus , if you want to re-init the existing node, delete STS, PVC, PV and start re-concile. Do you have multiple replicas?
@alex-zaitsev, thank you for the reply.
I understand how to delete objects. But what you are meant under "start re-concile"?
I have two replicas chi-gid-gid-0-0-0
and chi-gid-gid-0-1-0
. And now chi-gid-gid-0-0-0
is misfunction. I want to re-init it from the data in chi-gid-gid-0-1-0
. And that should include sync all:
@Hubbitus , we have released 0.23.7 that is more aggressive re-creating the schema. So you may try to delete PVC/PV completely, and let it to re-create the objects.
@alex-zaitsev, thank you very much! Eventually I get it updated for our cluster:
kub_dev get pods --all-namespaces -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}" -l app=clickhouse-operator
altinity/clickhouse-operator:0.23.7 altinity/metrics-exporter:0.23.7
And doing in ArgoCD:
PVC
default-volume-claim-chi-gid-gid-0-0-0
pod
chi-gid-gid-0-0-0
Then PVC
had been re-created.
I see pod is up and running.
2024.09.04 23:50:34.382651 [ 712 ] {} <Error> Access(user directories): from: 10.42.9.104, user: data_quality: Authentication failed: Code: 192. DB::Exception: There is no user
data_qualityin local_directory. (UNKNOWN_USER)...
. So, users are not copiedSELECT hostname() as node, COUNT(*)
FROM clusterAllReplicas('{cluster}', system.tables)
WHERE database NOT IN ('INFORMATION_SCHEMA', 'information_schema', 'system')
GROUP BY node
node | count() |
---|---|
chi-gid-gid-0-1-0 | 620 |
And also error in log like: 2024.09.04 23:52:49.039132 [ 714 ] {bb628508-db8e-4cf9-8307-a13133a185c9} <Error> PredefinedQueryHandler: Code: 60. DB::Exception: Table system.operator_compatible_metrics does not exist. (UNKNOWN_TABLE)
- so even in system database some tables missing...
So, I see only tables in information_schema
for the 1-st node.
We use your Operator to manage Clickhouse cluster. Thank you.
After some hardware failure we reset PVC (and zookeeper namespace) to re-create one clickhouse node.
Most of metadata like views, materialized views and tables with most engines (
MergeTree
,ReplicatedMergeTree
etc.) was successfully re-created on the node and replication was started.Meantime none of Postgres and Kafka based engines tables was recreated. Is it a bug, or we need to use some commands or hacks to sync all metadata across the cluster?