search

OT-CONTAINER-KIT / redis-operator

A golang based redis operator that will make/oversee Redis standalone/cluster/replication/sentinel mode setup on top of the Kubernetes.
https://ot-redis-operator.netlify.app/
Apache License 2.0
802 stars 219 forks source link

rediscluster failing to recover when scaling up from zero nodes #1100

Open Nashluffy opened 3 days ago

Nashluffy commented 3 days ago

What version of redis operator are you using?

redis-operator version: master

Does this issue reproduce with the latest release? Yes

What operating system and processor architecture are you using (kubectl version)?

nash:~/code/redis-operator$ k version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.0

What did you do?

Context: We scale down all workloads every night, including all corresponding rediscluster-owned statefulsets & the operator itself. When trying to bring the statefulset back up, redis-operator fails as it naively tries to issue a redis-cli --cluster create instead of rejoining the nodes. This fails as the nodes already have data on them, with the error (mimicking what the operator does)

nash:~/code/redis-operator$ k exec redis-cluster-leader-2 -it -- redis-cli --cluster create redis-cluster-leader-0.redis-cluster-leader-headless.default:6379 redis-cluster-leader-1.redis-cluster-leader-headless.default:6379 redis-cluster-leader-2.redis-cluster-leader-headless.default:6379
[ERR] Node redis-cluster-leader-0.redis-cluster-leader-headless.default:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

OR

The operator will start failover, which includes flushing the masters and then creating a cluster, which also isn't desirable as I lose data.

Reproduce via

  1. Create a minimal redis cluster & throw some keys in
nash:~/code/redis-operator$ kubectl apply -f example/v1beta2/redis-cluster.yaml
...
nash:~/code/redis-operator$ k exec redis-cluster-leader-0 -- redis-cli -c set k1 v1
OK
  1. Scale down both statefulset and operator
nash:~/code/redis-operator$ k scale -n redis-operator-system deploy -l control-plane=redis-operator --replicas=0
deployment.apps/redis-operator-redis-operator scaled
nash:~/code/redis-operator$ k scale sts redis-cluster-leader --replicas=0
statefulset.apps/redis-cluster-leader scaled
  1. Scale statefulset & operator back up, attempt to get keys set in 1, observe masters were flushed
nash:~/code/redis-operator$ k exec redis-cluster-leader-2 -- redis-cli cluster nodes
e71ab350c2aeda085ae4628137b10e3a26b221ca 10.244.0.53:6379@16379,redis-cluster-leader-1 master - 0 1728769019632 2 connected 5461-10922
8364e242894fb6357a0345e62f69f8384e9c99db 10.244.0.52:6379@16379,redis-cluster-leader-0 master - 0 1728769020600 1 connected 0-5460
d90a31a3ddd068bf559c450322ec33039fc9d461 10.244.0.54:6379@16379,redis-cluster-leader-2 myself,master - 0 1728769019000 3 connected 10923-16383
nash:~/code/redis-operator$ k exec redis-cluster-leader-2 -- redis-cli get k1

nash:~/code/redis-operator$

What did you expect to see?

When cluster is scaling up from zero nodes, I don't lose data

What did you see instead?

Data is wiped

Nashluffy commented 3 days ago

I think one possible solution is for the operator to optimistically repair any disconnected masters before failing over. It could do this via

  1. Discover disconnected masters according to leader 0
  2. For each disconnected master, if they contained a hostname, and the hostname matches the pods belonging to the operator-owned statefulset, resolve the hostname (unfortunately required as CLUSTER MEET does not perform DNS resolution)
  3. Issue CLUSTER MEET using resolved hostname and port

CLUSTER MEET seems smart enough to update an existing node, ie

# Immediately after bringing cluster up, cluster is down
nash:~/code/redis-operator$ k exec redis-cluster-leader-2 -- redis-cli cluster nodes
e71ab350c2aeda085ae4628137b10e3a26b221ca 10.244.0.53:6379@16379,redis-cluster-leader-1 master,fail? - 1728769456275 1728769455256 2 connected 5461-10922
d90a31a3ddd068bf559c450322ec33039fc9d461 10.244.0.58:6379@16379,redis-cluster-leader-2 myself,master - 0 1728769455257 3 connected 10923-16383
8364e242894fb6357a0345e62f69f8384e9c99db 10.244.0.52:6379@16379,redis-cluster-leader-0 master,fail? - 1728769457302 1728769455257 1 connected 0-5460

# Issue CLUSTER MEET for redis-cluster-leader-0 
nash:~/code/redis-operator$ k exec redis-cluster-leader-2 -- redis-cli cluster meet 10.244.0.57 6379
OK

# Issue CLUSTER MEET for redis-cluster-leader-1
nash:~/code/redis-operator$ k exec redis-cluster-leader-2 -- redis-cli cluster meet 10.244.0.56 6379
OK

# Observe all cluster nodes all connected
nash:~/code/redis-operator$ k exec redis-cluster-leader-2 -- redis-cli cluster nodes
e71ab350c2aeda085ae4628137b10e3a26b221ca 10.244.0.57:6379@16379,redis-cluster-leader-1 master - 0 1728769515866 2 connected 5461-10922
d90a31a3ddd068bf559c450322ec33039fc9d461 10.244.0.58:6379@16379,redis-cluster-leader-2 myself,master - 0 1728769515000 3 connected 10923-16383
8364e242894fb6357a0345e62f69f8384e9c99db 10.244.0.56:6379@16379,redis-cluster-leader-0 master - 0 1728769516871 1 connected 0-5460

# Observe cluster health is OK 
nash:~/code/redis-operator$ k exec redis-cluster-leader-2 -- redis-cli cluster info
cluster_state:ok
cluster_slots_assigned:16384