OT-CONTAINER-KIT / helm-charts

A repository which that will contain helm charts with best and security practices.
https://ot-container-kit.github.io/helm-charts
49 stars 84 forks source link

Pod restart failing when use redis-cluster #13

Closed batchuraghu closed 2 years ago

batchuraghu commented 3 years ago

Hi

I am using your helm-chart to setup redis cluster. we noticed 2 issues when pod is restarted.

Scenario 1: when PV is dynamically created

when pod is restated it will be in pending state due to earlier created pod(redis-cluster-follower-0) PVC attached to dynamically created PV. Solution : Manually edit pvc(redis-cluster-follower-0) finalizers section so new PV is created and attached to PVC and then pod will come up. finalizers:

2021-11-03T19:51:57.827Z INFO controller_redis Pod Counted successfully {"Request.RedisManager.Namespace": "dft-qa", "Request.RedisManager.Name": "dft-qa-redis-cluster", "Count": 0, "Container Name": "dft-qa-redis-cluster-leader"} 2021-11-03T19:52:17.636Z ERROR controller_redis Could not execute command {"Request.RedisManager.Namespace": "dft-qa", "Request.RedisManager.Name": "dft-qa-redis-cluster", "Command": ["redis-cli", "--cluster", "add-node", "100.67.1.166:6379", "100.67.1.165:6379", "--cluster-slave"], "Output": "", "Error": "", "error": "Internal error occurred: error executing command in container: container is not created or running"}

Solution : Manually delete all files(dump.rdb, appendonly.aof, nodes.conf) under persistence volume and restart the pod.

Do you have any solution for this?

Thanks

iamabhishek-dubey commented 2 years ago

Should be fixed now in v0.9.0