Open davidroman0O opened 11 months ago
So I have a bash script for now to force them to forget their conf and reset the conf with the correct IPs If i find new behaviors or reasons about this issue, i will post here
@davidroman0O
just change the namespace to development
I don't we could change namespace like that
@davidroman0O
I suspect that you use OpenShift cluster, or some Kubernete cluster that have network policies enabled.
{"level":"error","ts":1695241712.5628033,"logger":"controller_redis","msg":"Redis command failed with this error","Request.RedisManager.Namespace":"development","Request.RedisManager.Name":"redis-cluster","error":"dial tcp 172.16.11.75:6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CheckRedisNodeCount\n\t/workspace/k8sutils/redis.go:283\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisClusterReconciler).Reconcile\n\t/workspace/controllers/rediscluster_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
This log indicate that redis-operator pod cannot connect to redis-cluster-leader-0 in namespace "development".
if you deploy in openshift, please create network policy inside namespace "development"
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-redis-port-from-redis-operator
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-operators
ports:
- protocol: TCP
port: 6379
policyTypes:
- Ingress
After applied network policy. Allow the operator to reconcile for about 5 minutes, then try to perform "cluster nodes" on redis-cluster-leader-0 again. you should see 6 pod of redis, 3 leader 3 follower.
@davidroman0O
Can you post rediscluster.yaml as well. In the comments, you said you expect more leader node ? What is the cluster size that you set ?
I've deployed the manifest you gave me that namespace, I confirm it is deployed on the right namespace. Is it normal it doesn't need a pod selector?
The values are the default ones! I doubt you will see something new, I retested 5 mins ago, I copy/pasted the values from the website :)
For now, i just want to have the default behavior working
Every 10s, the operator is trying to reconcile the cluster. When I bash in the leader-0, and execute cluster nodes
, I clearly see that the IPs the follower IPs are not the correct ones :/
Provider: RKE1 Kubernetes Version: v1.23.8
I search if there are any pod or service attached to those two Ips, and nothing exists in --all-namespaces
with those Ips
Ok , so you use rancher. Then thing will be difference from OpenShift.
which namespace that you deploy redis-operator in ?
you need to allow network from redis-operator pod => redis-leader-0, redis-leader-1, redis-leader-2 pod. in my case [OpenShift] , the operator was deployed in openshift-operators namespace.
as per your question regarding network policy:
podSelector: {} # this mean, it will match all the pod
if the ingress: section match, the network policy will allow the traffic.
A quick check is that, you may try to deploy sleep pod (https://github.com/istio/istio/blob/master/samples/sleep/sleep.yaml) in the same namespace of your redis-operator pod. I stll guess that your redis-operator and your rediscluster resided in the difference namespace.
So please try to deploy sleep pod in
# deploy sleep pod
kubectl apply -n [namespace of operator pod] -f sleep.yaml
# deploy exec to get inside sleep pod
kubectl -n [namespace of operator pod] exec -it sleep -- bash
# inside sleep pod
curl -v telnet://172.16.11.75:6379
I assume that 172.16.11.75 is the ip of redis-cluster-leader-0. base on the log file.
if the curl result show as
Connected to 172.16.11.75 ... port 6379 - then there is no firewall or network policy blocking the traffic, then you should ignore my comment.
but if the curl result as
Connection timed out ...
you need to find a way to allow traffic from redis-operator pod , to redis-leader-0 ( and other redis pod )
Hello everyone, i'm testing the operator on my k8s cluster and i have a weird issue on the configuration of the leader pods which wrong IPs 🤷♂️
Is it the right behavior? did I missed something?
What version of redis operator are you using? latest one
redis-operator version: v0.15.0
Does this issue reproduce with the latest release? yes
What operating system and processor architecture are you using (
kubectl version
)? linux/amd64What did you do? Just following the guide on the website, just change the namespace to
development
What did you expect to see? When executing the command
cluster nodes
on my leader-0, i see there is only ONE correct IPWhat did you see instead? Shouldn't i see the IPs of the other leader pods?