Cassandra cannot become ready after configuration change

What happened?

We tried to change the configuration of an existing Cassandra cluster, by changing the cassandra-yaml.num_tokens from 16 to 8. The operator proceeds to update the StatefulSets of the racks, by changing the arguments of the Cassandra Pods. However, the restarted Pods are stuck at Unready state. The readiness probe keeps returning 500 errors.

What did you expect to happen?

The operator should be able to update the Cassandra configuration correctly. A restart of the Pod is not the right procedure for changing the num_tokens of Cassandra. To properly change the num_tokens, the operator needs to decommission the node and let the node to rejoin the cluster with the updated num_tokens configuration.

How can we reproduce it (as minimally and precisely as possible)?

Deploy the cass-operator
Deploy CassandraDB with the following CR yaml

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: test-cluster
spec:
  clusterName: development
  serverType: cassandra
  serverVersion: "4.1.2"
  managementApiAuth:
    insecure: {}
  size: 3
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: standard
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
  racks:
    - name: rack1
  config:
    jvm-server-options:
      initial_heap_size: "1G"
      max_heap_size: "1G"
    cassandra-yaml:
      num_tokens: 16
      authenticator: PasswordAuthenticator
      authorizer: CassandraAuthorizer
      role_manager: CassandraRoleManager

Change the cassandra-yaml.num_tokens from 16 to 8 in config:

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: test-cluster
spec:
  clusterName: development
  serverType: cassandra
  serverVersion: "4.1.2"
  managementApiAuth:
    insecure: {}
  size: 3
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: standard
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
  racks:
    - name: rack1
  config:
    jvm-server-options:
      initial_heap_size: "1G"
      max_heap_size: "1G"
    cassandra-yaml:
      num_tokens: 8
      authenticator: PasswordAuthenticator
      authorizer: CassandraAuthorizer
      role_manager: CassandraRoleManager
  users:
  - secretName: demo-secret
    superuser: true

Observe the restarted Pods keep returning 500 errors to readiness probe

cass-operator version

1.22.0

Kubernetes version

1.29.1

Method of installation

Helm

Anything else we need to know?

No response

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: CASS-2

k8ssandra / cass-operator