I am using your helm-chart to setup redis cluster. we noticed 2 issues when pod is restarted.
Scenario 1: when PV is dynamically created
when pod is restated it will be in pending state due to earlier created pod(redis-cluster-follower-0) PVC attached to dynamically created PV.
Solution : Manually edit pvc(redis-cluster-follower-0) finalizers section so new PV is created and attached to PVC and then pod will come up.
finalizers:
kubernetes.io/pvc-protection
Scenario 2: PV created before initiating helm-chart.
In this scenario when pod is restarted pod will come up successfully with out any issue but cluster is broken between the nodes and we see below error in the operator logs.
2021-11-03T19:51:57.827Z INFO controller_redis Pod Counted successfully {"Request.RedisManager.Namespace": "dft-qa", "Request.RedisManager.Name": "dft-qa-redis-cluster", "Count": 0, "Container Name": "dft-qa-redis-cluster-leader"}
2021-11-03T19:52:17.636Z ERROR controller_redis Could not execute command {"Request.RedisManager.Namespace": "dft-qa", "Request.RedisManager.Name": "dft-qa-redis-cluster", "Command": ["redis-cli", "--cluster", "add-node", "100.67.1.166:6379", "100.67.1.165:6379", "--cluster-slave"], "Output": "", "Error": "", "error": "Internal error occurred: error executing command in container: container is not created or running"}
Solution : Manually delete all files(dump.rdb, appendonly.aof, nodes.conf) under persistence volume and restart the pod.
Hi
I am using your helm-chart to setup redis cluster. we noticed 2 issues when pod is restarted.
Scenario 1: when PV is dynamically created
when pod is restated it will be in pending state due to earlier created pod(redis-cluster-follower-0) PVC attached to dynamically created PV. Solution : Manually edit pvc(redis-cluster-follower-0) finalizers section so new PV is created and attached to PVC and then pod will come up. finalizers:
kubernetes.io/pvc-protection
Scenario 2: PV created before initiating helm-chart.
In this scenario when pod is restarted pod will come up successfully with out any issue but cluster is broken between the nodes and we see below error in the operator logs.
2021-11-03T19:51:57.827Z INFO controller_redis Pod Counted successfully {"Request.RedisManager.Namespace": "dft-qa", "Request.RedisManager.Name": "dft-qa-redis-cluster", "Count": 0, "Container Name": "dft-qa-redis-cluster-leader"} 2021-11-03T19:52:17.636Z ERROR controller_redis Could not execute command {"Request.RedisManager.Namespace": "dft-qa", "Request.RedisManager.Name": "dft-qa-redis-cluster", "Command": ["redis-cli", "--cluster", "add-node", "100.67.1.166:6379", "100.67.1.165:6379", "--cluster-slave"], "Output": "", "Error": "", "error": "Internal error occurred: error executing command in container: container is not created or running"}
Solution : Manually delete all files(dump.rdb, appendonly.aof, nodes.conf) under persistence volume and restart the pod.
Do you have any solution for this?
Thanks