search

OT-CONTAINER-KIT / redis-operator

A golang based redis operator that will make/oversee Redis standalone/cluster/replication/sentinel mode setup on top of the Kubernetes.
https://ot-redis-operator.netlify.app/
Apache License 2.0
772 stars 214 forks source link

Wrong IPs assigned in cluster noes #640

Open davidroman0O opened 11 months ago

davidroman0O commented 11 months ago

Hello everyone, i'm testing the operator on my k8s cluster and i have a weird issue on the configuration of the leader pods which wrong IPs 🤷‍♂️

Is it the right behavior? did I missed something?

What version of redis operator are you using? latest one

{"level":"info","ts":1695241558.5557573,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1695241558.5561597,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1695241558.5565012,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":1695241558.5565639,"msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
I0920 20:25:58.556672       1 leaderelection.go:248] attempting to acquire leader lease development/6cab913b.redis.opstreelabs.in...
I0920 20:26:40.990874       1 leaderelection.go:258] successfully acquired lease development/6cab913b.redis.opstreelabs.in
{"level":"info","ts":1695241600.991832,"logger":"controller.redis","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis","source":"kind source: *v1beta1.Redis"}
{"level":"info","ts":1695241600.9919016,"logger":"controller.redis","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis"}
{"level":"info","ts":1695241600.9919605,"logger":"controller.redisreplication","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication","source":"kind source: *v1beta1.RedisReplication"}
{"level":"info","ts":1695241600.991977,"logger":"controller.rediscluster","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","source":"kind source: *v1beta1.RedisCluster"}
{"level":"info","ts":1695241600.9920225,"logger":"controller.rediscluster","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster"}
{"level":"info","ts":1695241600.992006,"logger":"controller.redisreplication","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication"}
{"level":"info","ts":1695241600.9921794,"logger":"controller.redissentinel","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel","source":"kind source: *v1beta1.RedisSentinel"}
{"level":"info","ts":1695241600.9922056,"logger":"controller.redissentinel","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel"}
{"level":"info","ts":1695241601.0925965,"logger":"controller.redissentinel","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisSentinel","worker count":1}
{"level":"info","ts":1695241601.0926392,"logger":"controller.redis","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis","worker count":1}
{"level":"info","ts":1695241601.0926392,"logger":"controller.redisreplication","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisReplication","worker count":1}
{"level":"info","ts":1695241601.093719,"logger":"controller.rediscluster","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","worker count":1}
{"level":"info","ts":1695241601.0939004,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"development","Request.Name":"redis-cluster"}
{"level":"info","ts":1695241601.1116838,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.75"}
{"level":"info","ts":1695241601.1216364,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":"7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241157042 1695241155129 2 connected 5461-10922\na85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.75:6379@16379 myself,master - 0 1695241155129 1 connected 0-5460\n452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241156034 1695241155129 3 connected 10923-16383\n"}
{"level":"info","ts":1695241601.1217797,"logger":"controller_redis","msg":"Number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"3","Type":"leader"}
{"level":"info","ts":1695241601.1301486,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241601.1492338,"logger":"controller_redis","msg":"Reconciliation Complete, no Changes required.","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241601.155956,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-headless"}
{"level":"info","ts":1695241601.1598084,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-headless"}
{"level":"info","ts":1695241601.1662288,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241601.1689153,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241601.1797252,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-additional"}
{"level":"info","ts":1695241601.1826177,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-additional"}
{"level":"info","ts":1695241601.189812,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"development","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241601.1898532,"logger":"controller_redis","msg":"Reconciliation Successful, no PodDisruptionBudget Found.","Request.PodDisruptionBudget.Namespace":"development","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241601.2307966,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241601.2453642,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241601.25394,"logger":"controller_redis","msg":"Reconciliation Complete, no Changes required.","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241601.2632942,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-follower-headless"}
{"level":"info","ts":1695241601.2656193,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-follower-headless"}
{"level":"info","ts":1695241601.289231,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241601.2918751,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241601.303579,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-follower-additional"}
{"level":"info","ts":1695241601.3066945,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-follower-additional"}
{"level":"info","ts":1695241601.314207,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"development","Request.PodDisruptionBudget.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241601.3142428,"logger":"controller_redis","msg":"Reconciliation Successful, no PodDisruptionBudget Found.","Request.PodDisruptionBudget.Namespace":"development","Request.PodDisruptionBudget.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241601.3196921,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241601.3197153,"logger":"controllers.RedisCluster","msg":"Creating redis cluster by executing cluster creation commands","Request.Namespace":"development","Request.Name":"redis-cluster","Leaders.Ready":"3","Followers.Ready":"3"}
{"level":"info","ts":1695241601.3292465,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.75"}
{"level":"info","ts":1695241601.3315086,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":"7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241157042 1695241155129 2 connected 5461-10922\na85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.75:6379@16379 myself,master - 0 1695241155129 1 connected 0-5460\n452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241156034 1695241155129 3 connected 10923-16383\n"}
{"level":"info","ts":1695241601.33162,"logger":"controller_redis","msg":"Total number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"3"}
{"level":"info","ts":1695241601.337728,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.75"}
{"level":"info","ts":1695241601.3482819,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":"7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241157042 1695241155129 2 connected 5461-10922\na85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.75:6379@16379 myself,master - 0 1695241155129 1 connected 0-5460\n452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241156034 1695241155129 3 connected 10923-16383\n"}
{"level":"info","ts":1695241601.348424,"logger":"controller_redis","msg":"Number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"3","Type":"leader"}
{"level":"info","ts":1695241601.34845,"logger":"controllers.RedisCluster","msg":"All leader are part of the cluster, adding follower/replicas","Request.Namespace":"development","Request.Name":"redis-cluster","Leaders.Count":3,"Instance.Size":3,"Follower.Replicas":3}
{"level":"info","ts":1695241601.3541691,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.75"}
{"level":"info","ts":1695241601.3576279,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":"7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241157042 1695241155129 2 connected 5461-10922\na85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.75:6379@16379 myself,master - 0 1695241155129 1 connected 0-5460\n452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241156034 1695241155129 3 connected 10923-16383\n"}
{"level":"info","ts":1695241601.3643105,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-follower-0","ip":"172.16.27.54"}
{"level":"info","ts":1695241601.3643398,"logger":"controller_redis","msg":"Checking if Node is in cluster","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Node":"172.16.27.54"}
{"level":"info","ts":1695241601.3643486,"logger":"controller_redis","msg":"Adding node to cluster.","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Node.IP":"172.16.27.54","Follower.Pod":{"PodName":"redis-cluster-follower-0","Namespace":"development"}}
{"level":"info","ts":1695241601.3711567,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-follower-0","ip":"172.16.27.54"}
{"level":"info","ts":1695241601.3778498,"logger":"controller_redis","msg":"Pod Counted successfully","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Count":0,"Container Name":"redis-cluster-leader"}
{"level":"error","ts":1695241712.3253217,"logger":"controller_redis","msg":"Could not execute command","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Command":["redis-cli","--cluster","add-node","redis-cluster-follower-0.redis-cluster-follower-headless.development.svc:6379","redis-cluster-leader-0.redis-cluster-leader-headless.development.svc:6379","--cluster-slave"],"Output":">>> Adding node redis-cluster-follower-0.redis-cluster-follower-headless.development.svc:6379 to cluster redis-cluster-leader-0.redis-cluster-leader-headless.development.svc:6379\n","Error":"","error":"command terminated with exit code 137","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.ExecuteRedisReplicationCommand\n\t/workspace/k8sutils/redis.go:187\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:169\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241712.33252,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-follower-1","ip":"172.16.9.203"}
{"level":"info","ts":1695241712.3325617,"logger":"controller_redis","msg":"Checking if Node is in cluster","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Node":"172.16.9.203"}
{"level":"info","ts":1695241712.3325744,"logger":"controller_redis","msg":"Adding node to cluster.","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Node.IP":"172.16.9.203","Follower.Pod":{"PodName":"redis-cluster-follower-1","Namespace":"development"}}
{"level":"info","ts":1695241712.3385382,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-follower-1","ip":"172.16.9.203"}
{"level":"info","ts":1695241712.3453166,"logger":"controller_redis","msg":"Pod Counted successfully","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Count":0,"Container Name":"redis-cluster-leader"}
{"level":"error","ts":1695241712.4482448,"logger":"controller_redis","msg":"Could not execute command","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Command":["redis-cli","--cluster","add-node","redis-cluster-follower-1.redis-cluster-follower-headless.development.svc:6379","redis-cluster-leader-1.redis-cluster-leader-headless.development.svc:6379","--cluster-slave"],"Output":"","Error":"","error":"","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.ExecuteRedisReplicationCommand\n\t/workspace/k8sutils/redis.go:187\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:169\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241712.457105,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-follower-2","ip":"172.16.29.156"}
{"level":"info","ts":1695241712.4571617,"logger":"controller_redis","msg":"Checking if Node is in cluster","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Node":"172.16.29.156"}
{"level":"info","ts":1695241712.457172,"logger":"controller_redis","msg":"Adding node to cluster.","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Node.IP":"172.16.29.156","Follower.Pod":{"PodName":"redis-cluster-follower-2","Namespace":"development"}}
{"level":"info","ts":1695241712.4633243,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-follower-2","ip":"172.16.29.156"}
{"level":"info","ts":1695241712.4709537,"logger":"controller_redis","msg":"Pod Counted successfully","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Count":0,"Container Name":"redis-cluster-leader"}
{"level":"error","ts":1695241712.546609,"logger":"controller_redis","msg":"Could not execute command","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Command":["redis-cli","--cluster","add-node","redis-cluster-follower-2.redis-cluster-follower-headless.development.svc:6379","redis-cluster-leader-2.redis-cluster-leader-headless.development.svc:6379","--cluster-slave"],"Output":"","Error":"","error":"unable to upgrade connection: container not found (\"redis-cluster-leader\")","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.ExecuteRedisReplicationCommand\n\t/workspace/k8sutils/redis.go:187\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:169\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241712.5540848,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.75"}
{"level":"error","ts":1695241712.5628033,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp 172.16.11.75:6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1695241712.5629091,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp 172.16.11.75:6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241712.562937,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":""}
{"level":"info","ts":1695241712.5629625,"logger":"controller_redis","msg":"Total number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"0"}
{"level":"info","ts":1695241712.5629687,"logger":"controllers.RedisCluster","msg":"Will reconcile redis cluster operator in again 10 seconds","Request.Namespace":"development","Request.Name":"redis-cluster"}
{"level":"info","ts":1695241722.5638053,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"development","Request.Name":"redis-cluster"}
{"level":"info","ts":1695241722.5709054,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":""}
{"level":"error","ts":1695241722.571113,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:74\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1695241722.5711718,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:74\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241722.5711868,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":""}
{"level":"info","ts":1695241722.5711985,"logger":"controller_redis","msg":"Number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"0","Type":"leader"}
{"level":"info","ts":1695241722.580384,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241722.588922,"logger":"controller_redis","msg":"Reconciliation Complete, no Changes required.","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241722.5943441,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-headless"}
{"level":"info","ts":1695241722.5958884,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-headless"}
{"level":"info","ts":1695241722.6006489,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241722.6029935,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241722.6083891,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-additional"}
{"level":"info","ts":1695241722.6115801,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-additional"}
{"level":"info","ts":1695241722.6159263,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"development","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241722.6159456,"logger":"controller_redis","msg":"Reconciliation Successful, no PodDisruptionBudget Found.","Request.PodDisruptionBudget.Namespace":"development","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241722.6218033,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241722.6288297,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241722.62886,"logger":"controllers.RedisCluster","msg":"Creating redis cluster by executing cluster creation commands","Request.Namespace":"development","Request.Name":"redis-cluster","Leaders.Ready":"0","Followers.Ready":"3"}
{"level":"info","ts":1695241722.6420698,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":""}
{"level":"error","ts":1695241722.6422477,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1695241722.6423001,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:151\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241722.6423151,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":""}
{"level":"info","ts":1695241722.6423259,"logger":"controller_redis","msg":"Total number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"0"}
{"level":"info","ts":1695241722.6484668,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":""}
{"level":"error","ts":1695241722.6486487,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:152\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1695241722.648702,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:152\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241722.6487248,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":""}
{"level":"info","ts":1695241722.6487381,"logger":"controller_redis","msg":"Number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"0","Type":"leader"}
{"level":"info","ts":1695241722.6487448,"logger":"controllers.RedisCluster","msg":"Not all leader are part of the cluster...","Request.Namespace":"development","Request.Name":"redis-cluster","Leaders.Count":0,"Instance.Size":3}
{"level":"info","ts":1695241722.6487722,"logger":"controller_redis","msg":"Redis Add Slots command for single node cluster is","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Command":["redis-cli","--cluster","create","redis-cluster-leader-0.redis-cluster-leader-headless.development.svc:6379","redis-cluster-leader-1.redis-cluster-leader-headless.development.svc:6379","redis-cluster-leader-2.redis-cluster-leader-headless.development.svc:6379","--cluster-yes"]}
{"level":"info","ts":1695241722.6487825,"logger":"controller_redis","msg":"Redis cluster creation command is","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Command":["redis-cli","--cluster","create","redis-cluster-leader-0.redis-cluster-leader-headless.development.svc:6379","redis-cluster-leader-1.redis-cluster-leader-headless.development.svc:6379","redis-cluster-leader-2.redis-cluster-leader-headless.development.svc:6379","--cluster-yes"]}
{"level":"info","ts":1695241722.6545098,"logger":"controller_redis","msg":"Pod Counted successfully","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Count":0,"Container Name":"redis-cluster-leader"}
{"level":"error","ts":1695241722.7060487,"logger":"controller_redis","msg":"Could not execute command","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Command":["redis-cli","--cluster","create","redis-cluster-leader-0.redis-cluster-leader-headless.development.svc:6379","redis-cluster-leader-1.redis-cluster-leader-headless.development.svc:6379","redis-cluster-leader-2.redis-cluster-leader-headless.development.svc:6379","--cluster-yes"],"Output":"","Error":"","error":"unable to upgrade connection: container not found (\"redis-cluster-leader\")","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.ExecuteRedisClusterCommand\n\t/workspace/k8sutils/redis.go:116\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:156\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241722.71611,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":""}
{"level":"error","ts":1695241722.7162986,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1695241722.7163546,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1695241722.7163818,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":""}
{"level":"info","ts":1695241722.7163973,"logger":"controller_redis","msg":"Total number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"0"}
{"level":"info","ts":1695241722.7164075,"logger":"controllers.RedisCluster","msg":"Will reconcile redis cluster operator in again 10 seconds","Request.Namespace":"development","Request.Name":"redis-cluster"}
{"level":"info","ts":1695241732.7166762,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"development","Request.Name":"redis-cluster"}
{"level":"info","ts":1695241732.723804,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.76"}
{"level":"info","ts":1695241732.725397,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":"7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241725516 1695241723597 2 connected 5461-10922\na85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.76:6379@16379 myself,master - 0 1695241723597 1 connected 0-5460\n452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241724513 1695241723597 3 connected 10923-16383\n"}
{"level":"info","ts":1695241732.7255385,"logger":"controller_redis","msg":"Number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"3","Type":"leader"}
{"level":"info","ts":1695241732.7318254,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241732.7397232,"logger":"controller_redis","msg":"Reconciliation Complete, no Changes required.","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241732.7453969,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-headless"}
{"level":"info","ts":1695241732.7474065,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-headless"}
{"level":"info","ts":1695241732.7526417,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241732.7543774,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241732.7590923,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-additional"}
{"level":"info","ts":1695241732.76067,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"development","Request.Service.Name":"redis-cluster-leader-additional"}
{"level":"info","ts":1695241732.765145,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"development","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241732.7651677,"logger":"controller_redis","msg":"Reconciliation Successful, no PodDisruptionBudget Found.","Request.PodDisruptionBudget.Namespace":"development","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241732.770068,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1695241732.7756398,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"development","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"info","ts":1695241732.7756634,"logger":"controllers.RedisCluster","msg":"Creating redis cluster by executing cluster creation commands","Request.Namespace":"development","Request.Name":"redis-cluster","Leaders.Ready":"1","Followers.Ready":"3"}
{"level":"info","ts":1695241732.7835164,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.76"}
{"level":"info","ts":1695241732.78798,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":"7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241725516 1695241723597 2 connected 5461-10922\na85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.76:6379@16379 myself,master - 0 1695241723597 1 connected 0-5460\n452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241724513 1695241723597 3 connected 10923-16383\n"}
{"level":"info","ts":1695241732.7880664,"logger":"controller_redis","msg":"Total number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"3"}
{"level":"info","ts":1695241732.7944686,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.76"}
{"level":"info","ts":1695241732.8005292,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":"7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241725516 1695241723597 2 connected 5461-10922\na85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.76:6379@16379 myself,master - 0 1695241723597 1 connected 0-5460\n452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241724513 1695241723597 3 connected 10923-16383\n"}
{"level":"info","ts":1695241732.800613,"logger":"controller_redis","msg":"Number of redis nodes are","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Nodes":"3","Type":"leader"}
{"level":"info","ts":1695241732.800621,"logger":"controllers.RedisCluster","msg":"All leader are part of the cluster, adding follower/replicas","Request.Namespace":"development","Request.Name":"redis-cluster","Leaders.Count":3,"Instance.Size":3,"Follower.Replicas":3}
{"level":"info","ts":1695241732.8281913,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-leader-0","ip":"172.16.11.76"}
{"level":"info","ts":1695241732.831645,"logger":"controller_redis","msg":"Redis cluster nodes are listed","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Output":"7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241725516 1695241723597 2 connected 5461-10922\na85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.76:6379@16379 myself,master - 0 1695241723597 1 connected 0-5460\n452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241724513 1695241723597 3 connected 10923-16383\n"}
{"level":"info","ts":1695241732.8456619,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-follower-0","ip":"172.16.27.54"}
{"level":"info","ts":1695241732.845696,"logger":"controller_redis","msg":"Checking if Node is in cluster","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Node":"172.16.27.54"}
{"level":"info","ts":1695241732.8457062,"logger":"controller_redis","msg":"Adding node to cluster.","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Node.IP":"172.16.27.54","Follower.Pod":{"PodName":"redis-cluster-follower-0","Namespace":"development"}}
{"level":"info","ts":1695241732.8907347,"logger":"controller_redis","msg":"Successfully got the ip for redis","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster-follower-0","ip":"172.16.27.54"}
{"level":"info","ts":1695241732.9145372,"logger":"controller_redis","msg":"Pod Counted successfully","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","Count":0,"Container Name":"redis-cluster-leader"}

redis-operator version: v0.15.0

Does this issue reproduce with the latest release? yes

What operating system and processor architecture are you using (kubectl version)? linux/amd64

What did you do? Just following the guide on the website, just change the namespace to development

What did you expect to see? When executing the command cluster nodes on my leader-0, i see there is only ONE correct IP

7586c640214536a4a0919fb8c2025aa0add96fef 172.16.9.201:6379@16379 master,fail? - 1695241725516 1695241723597 2 connected 5461-10922
a85a769278a3031e38ac3d9af713e07975eb263d 172.16.11.76:6379@16379 myself,master - 0 1695241723597 1 connected 0-5460
452cbbe79db6cbb60013d674a018f0f14c495941 172.16.10.62:6379@16379 master,fail? - 1695241724513 1695241723597 3 connected 10923-16383
master cluster cluster ips

What did you see instead? Shouldn't i see the IPs of the other leader pods?

davidroman0O commented 11 months ago

So I have a bash script for now to force them to forget their conf and reset the conf with the correct IPs If i find new behaviors or reasons about this issue, i will post here

shubham-cmyk commented 11 months ago

@davidroman0O

just change the namespace to development

I don't we could change namespace like that

isaru66 commented 11 months ago

@davidroman0O

I suspect that you use OpenShift cluster, or some Kubernete cluster that have network policies enabled.

{"level":"error","ts":1695241712.5628033,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp 172.16.11.75:6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}

This log indicate that redis-operator pod cannot connect to redis-cluster-leader-0 in namespace "development".

if you deploy in openshift, please create network policy inside namespace "development"

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-redis-port-from-redis-operator
spec:
  podSelector: {}
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: openshift-operators
      ports:
        - protocol: TCP
          port: 6379
  policyTypes:
  - Ingress

After applied network policy. Allow the operator to reconcile for about 5 minutes, then try to perform "cluster nodes" on redis-cluster-leader-0 again. you should see 6 pod of redis, 3 leader 3 follower.

isaru66 commented 11 months ago

@davidroman0O

Can you post rediscluster.yaml as well. In the comments, you said you expect more leader node ? What is the cluster size that you set ?

davidroman0O commented 11 months ago

I've deployed the manifest you gave me that namespace, I confirm it is deployed on the right namespace. Is it normal it doesn't need a pod selector?

The values are the default ones! I doubt you will see something new, I retested 5 mins ago, I copy/pasted the values from the website :)

For now, i just want to have the default behavior working

Every 10s, the operator is trying to reconcile the cluster. When I bash in the leader-0, and execute cluster nodes, I clearly see that the IPs the follower IPs are not the correct ones :/

Provider: RKE1 Kubernetes Version: v1.23.8

I search if there are any pod or service attached to those two Ips, and nothing exists in --all-namespaces with those Ips

isaru66 commented 11 months ago

Ok , so you use rancher. Then thing will be difference from OpenShift.

which namespace that you deploy redis-operator in ?

you need to allow network from redis-operator pod => redis-leader-0, redis-leader-1, redis-leader-2 pod. in my case [OpenShift] , the operator was deployed in openshift-operators namespace.

as per your question regarding network policy:

podSelector: {} # this mean, it will match all the pod

if the ingress: section match, the network policy will allow the traffic.

A quick check is that, you may try to deploy sleep pod (https://github.com/istio/istio/blob/master/samples/sleep/sleep.yaml) in the same namespace of your redis-operator pod. I stll guess that your redis-operator and your rediscluster resided in the difference namespace.

So please try to deploy sleep pod in 

# deploy sleep pod
kubectl apply -n [namespace of operator pod] -f sleep.yaml

# deploy exec to get inside sleep pod
kubectl -n [namespace of operator pod] exec -it sleep -- bash
# inside sleep pod
curl -v telnet://172.16.11.75:6379

I assume that 172.16.11.75 is the ip of redis-cluster-leader-0. base on the log file.

if the curl result show as 

Connected to 172.16.11.75 ... port 6379 - then there is no firewall or network policy blocking the traffic, then you should ignore my comment.

but if the curl result as 

Connection timed out ...

you need to find a way to allow traffic from redis-operator pod , to redis-leader-0 ( and other redis pod )