Open mbrancato opened 1 month ago
Did you upgrade CRDs separatelly as described in https://github.com/Altinity/clickhouse-operator/blob/master/deploy/helm/clickhouse-operator/README.md?
@Slach I did not update the CRD. I have done so now, and it still is happening. Do I need to manually set a status.hostsUnchanged
value in the CHI status?
% kubectl -n clickhouse get deploy chop-altinity-clickhouse-operator -o yaml | grep "image:"
image: altinity/clickhouse-operator:0.23.6
image: altinity/metrics-exporter:0.23.6
% kubectl get crd clickhouseinstallations.clickhouse.altinity.com -o yaml | grep "clickhouse.altinity.com/chop"
clickhouse.altinity.com/chop: 0.23.6
I0710 17:16:03.682328 1 service.go:86] CreateServiceCluster():foo/events/e425f007-f928-4649-92ed-3fb1598f1915:foo/clickhouse-events
I0710 17:16:03.683552 1 worker-chi-reconciler.go:907] reconcileService():foo/events/e425f007-f928-4649-92ed-3fb1598f1915:Service: foo/clickhouse-events not found. err: service "clickhouse-events" not found
I0710 17:16:03.850464 1 deleter.go:322] deleteServiceIfExists():foo/clickhouse-events:Not Found Service: foo/clickhouse-events err: services "clickhouse-events" not found
I0710 17:16:04.074642 1 worker.go:1480] createService():foo/events/e425f007-f928-4649-92ed-3fb1598f1915:OK Create Service: foo/clickhouse-events
I0710 17:16:04.882621 1 worker-chi-reconciler.go:922] reconcileService():foo/events/e425f007-f928-4649-92ed-3fb1598f1915:Service reconcile successful: foo/clickhouse-events
I0710 17:16:17.088227 1 worker-chi-reconciler.go:900] reconcileService():foo/events/e425f007-f928-4649-92ed-3fb1598f1915:Service found: foo/clickhouse-events. Will try to update
E0710 17:16:17.088280 1 worker-chi-reconciler.go:914] reconcileService():foo/events/e425f007-f928-4649-92ed-3fb1598f1915:Update Service: foo/clickhouse-events failed with error: just recreate the service in case of service type change 'LoadBalancer'=>'ClusterIP'
I0710 17:16:18.254178 1 deleter.go:329] deleteServiceIfExists():foo/clickhouse-events:OK delete Service: foo/clickhouse-events
E0710 17:16:18.453646 1 worker.go:1486] createService():foo/events/e425f007-f928-4649-92ed-3fb1598f1915:FAILED Create Service: foo/clickhouse-events err: object is being deleted: services "clickhouse-events" already exists
E0710 17:16:19.295985 1 worker-chi-reconciler.go:928] reconcileService():foo/events/e425f007-f928-4649-92ed-3fb1598f1915:FAILED to reconcile Service: foo/clickhouse-events CHI: events
Service: foo/clickhouse-events
Service: foo/clickhouse-events
Service: foo/clickhouse-events
I tried adding a value into status.hostsUnchanged
(that was the only change compared the the old CRD installed), and it made no difference. The CHOP is still constantly deleting the cluster service.
--- deploy/operatorhub/0.23.2/clickhouseinstallations.clickhouse.altinity.com.crd.yaml 2024-07-10 14:26:54
+++ deploy/operatorhub/0.23.6/clickhouseinstallations.clickhouse.altinity.com.crd.yaml 2024-07-10 14:26:54
@@ -4,14 +4,14 @@
# SINGULAR=clickhouseinstallation
# PLURAL=clickhouseinstallations
# SHORT=chi
-# OPERATOR_VERSION=0.23.2
+# OPERATOR_VERSION=0.23.6
#
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: clickhouseinstallations.clickhouse.altinity.com
labels:
- clickhouse.altinity.com/chop: 0.23.2
+ clickhouse.altinity.com/chop: 0.23.6
spec:
group: clickhouse.altinity.com
scope: Namespaced
@@ -53,6 +53,11 @@
type: string
description: CHI status
jsonPath: .status.status
+ - name: hosts-unchanged
+ type: integer
+ description: Unchanged hosts count
+ priority: 1 # show in wide view
+ jsonPath: .status.hostsUnchanged
- name: hosts-updated
type: integer
description: Updated hosts count
@@ -172,6 +177,10 @@
nullable: true
items:
type: string
+ hostsUnchanged:
+ type: integer
+ minimum: 0
+ description: "Unchanged Hosts count"
hostsUpdated:
type: integer
minimum: 0
While performing an upgrade via Helm from 0.23.2 to 0.23.6, I ran across a problem where the cluster service disappeared. I also included a minor upgrade of the
altinitystable
image, but I don't think that is related.The important bits in my CHI resource:
When the operator upgraded, it appeared to get stuck attempting to convert
clickhouse-events
from aLoadBalancer
to aClusterIP
. I believe this is somehow related to this commit that changes the default fromLoadBalancer
toClusterIP
. However, this CHI has always explicitly set the template to useLoadBalancer
.On startup, I saw this in the logs:
It now appears to be recreated on a forced restart of the operator, and then a minute or so later, is deleted again. It won't be recreated until the operator restarts again.
Note: When it is creates, it is created correctly as a
LoadBalancer
, but then the second resource reconciliation attempts to make it aClusterIP
again.