search

AmadeusITGroup / Redis-Operator

Redis Operator creates/configures/manages Redis clusters atop Kubernetes
MIT License
167 stars 62 forks source link

New cluster only creating a single pod #17

Closed calvinmorrow closed 6 years ago

calvinmorrow commented 6 years ago

I'm running into an error that is resulting in the redis cluster never fully forming.

A RedisCluster is created (via helm) in a namespace called "redis" with numberOfMaster: 3 and replicationFactor: 1. The operator starts working on the cluster and immediately spews some strange logs:

NumberOfMaster:0
MinReplicationFactor:0
MaxReplicationFactor:0
NodesPlacement:

NbPods:0
NbPodsReady:0
NbRedisRunning:0

Nodes (0): []
}}] error: Operation cannot be fulfilled on redisclusters.redisoperator.k8s.io "redis-cluster": the object has been modified; please apply your changes to the latest version and try again
I0705 23:40:23.226555       1 controller.go:168] Finished syncing RedisCluster "redis/redis-cluster" (61.243177ms
E0705 23:40:23.226620       1 controller.go:151] Error syncing rediscluster: Operation cannot be fulfilled on redisclusters.redisoperator.k8s.io "redis-cluster": the object has been modified; please apply your changes to the latest version and try again
I0705 23:40:23.226725       1 controller.go:165] sync() key:redis/redis-cluster
E0705 23:40:23.227655       1 controller.go:407] Unable to retrieve the associated Redis Node with the pod: rediscluster-redis-cluster-lhmf9, ip:, err:node not founded
I0705 23:40:23.227749       1 checks.go:107] compare status.NbPods: 0 - 1
I0705 23:40:23.258568       1 controller.go:168] Finished syncing RedisCluster "redis/redis-cluster" (31.79447ms
I0705 23:40:23.258676       1 controller.go:165] sync() key:redis/redis-cluster
E0705 23:40:23.259229       1 controller.go:407] Unable to retrieve the associated Redis Node with the pod: rediscluster-redis-cluster-lhmf9, ip:, err:node not founded
I0705 23:40:23.271476       1 controller.go:168] Finished syncing RedisCluster "redis/redis-cluster" (12.764577ms
I0705 23:40:23.271544       1 controller.go:165] sync() key:redis/redis-cluster

This repeats a few times until it settles into the following:

E0705 23:51:15.374957       1 controller.go:407] Unable to retrieve the associated Redis Node with the pod: rediscluster-redis-cluster-lhmf9, ip:10.128.4.117, err:node not founded
I0705 23:51:15.375198       1 controller.go:168] Finished syncing RedisCluster "redis/redis-cluster" (2.00105339s
I0705 23:51:43.374389       1 controller.go:165] sync() key:redis/redis-cluster
E0705 23:51:45.375369       1 controller.go:407] Unable to retrieve the associated Redis Node with the pod: rediscluster-redis-cluster-lhmf9, ip:10.128.4.117, err:node not founded
I0705 23:51:45.375714       1 controller.go:168] Finished syncing RedisCluster "redis/redis-cluster" (2.001262793s
I0705 23:52:13.374669       1 controller.go:165] sync() key:redis/redis-cluster
E0705 23:52:15.375377       1 controller.go:407] Unable to retrieve the associated Redis Node with the pod: rediscluster-redis-cluster-lhmf9, ip:10.128.4.117, err:node not founded
I0705 23:52:15.375644       1 controller.go:168] Finished syncing RedisCluster "redis/redis-cluster" (2.000922663s

A single pod is created named rediscluster-redis-cluster-lhmf9 which for most intents looks fine with mostly normal (as far as I can tell) logs, however further cluster nodes are never created.

calvinmorrow commented 6 years ago

Update: On a hunch I went back and checked to see if any NetworkPolicy objects were present (as it was a previously-existing namespace) and sure enough, a NetworkPolicy object existed that was over a month old. I deleted the object and the cluster formed successfully.