Replace ScyllaDB nodes without affecting the availablility (create a new node first)

[tnozicka]

ScyllaDB support a graceful replace node procedure where a new node is spun up before the old one is decommissioned.

https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/replace-running-node.html#add-a-new-node-to-the-cluster-and-then-decommission-the-old-node

This is incompatible with using StatefulSets where each replica has identity / ordinal and there can always be just one instance of the ordinal.

[scylla-operator-bot[bot]]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale