search

Orange-OpenSource / casskop

This Kubernetes operator automates the Cassandra operations such as deploying a new rack aware cluster, adding/removing nodes, configuring the C* and JVM parameters, upgrading JVM and C* versions, and many more...
https://orange-opensource.github.io/casskop/
Apache License 2.0
183 stars 54 forks source link

Failed update for cassandracluster #364

Closed AKamyshnikova closed 3 years ago

AKamyshnikova commented 3 years ago

Bug Report

What did you do? Update casskop operator from v0.5.3 to latest version. Since v1.0.0. statefulset for CassandraCluster has two containers "cassandra" and "backrest-sidecar".

What did you expect to see? In the setup there are 2 cassandraclusters in one kubernetes cluster with same structure, it is was expected that both got updated.

What did you see instead? Under which circumstances? One of the cassandraclusters got updated - structure changed 2 containers come up. Update for second didn't start. Error observed in casskop logs:

time="2021-09-14T12:53:54Z" level=info msg="Template is different, the number of containers are not the same: len(sts1.Spec.Template.Spec.Containers) = 1, len(sts2.Sp
ec.Template.Spec.Containers) = 2" namespace=tf statefulset=tf-cassandra-config-dc1-rack1                                                                              
time="2021-09-14T12:53:54Z" level=error msg="ensureCassandraStatefulSet Error: failed to create cassandra statefulset: statefulset already exists: &{{{%c(string=)} {%!c(string=) %!c(string=) %!c(string=) %!c(*int64=<nil>)} %!c(string=Failure) %!c(string=StatefulSet.apps \"tf-cassandra-config-dc1-rack1\" is invalid: s
pec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden) %!c(v1.StatusReason=Invalid) %!c(*v1.Sta
tusDetails=&{tf-cassandra-config-dc1-rack1 apps StatefulSet  [{FieldValueForbidden Forbidden: updates to statefulset spec for fields other than 'replicas', 'template'
, and 'updateStrategy' are forbidden spec}] 0}) Ʀ}}c" cluster=tf-cassandra-config dc-rack=dc1-rack1

Environment

Possible Solution Update BackRestSidecar struct with enabled field to allow disable of it.

Additional context As one of clusters got properly updated, this could be some limitation in logic, that second cluster was not processed as expected.

AKamyshnikova commented 3 years ago

I did more debugging of this issue today, the error is not on casskop side. ResourceVersion of cassandracluster(which cannot be updated later) was updated externally.