search

sanderploegsma / redis-cluster

Redis Cluster setup running on Kubernetes
https://sanderp.nl/running-redis-cluster-on-kubernetes-e451bda76cad
MIT License
246 stars 132 forks source link

Waiting for the cluster to join takes forever.... #17

Closed khteh closed 4 years ago

khteh commented 5 years ago 
$ k logs -f redis-cluster-init-27fkk
+ yes yes
++ kubectl get pods -l app=redis-cluster -o 'jsonpath={range.items[*]}{.status.podIP}:6379 '
+ redis-cli --cluster create --cluster-replicas 1 10.1.1.97:6379 10.1.1.98:6379 10.1.1.99:6379 10.1.1.100:6379 10.1.1.101:6379 10.1.1.102:6379
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.1.1.101:6379 to 10.1.1.97:6379
Adding replica 10.1.1.102:6379 to 10.1.1.98:6379
Adding replica 10.1.1.100:6379 to 10.1.1.99:6379
M: 221c3e7f661f31dc842b07377250ac75fa996ab3 10.1.1.97:6379
   slots:[0-5460] (5461 slots) master
M: acc3fe09d0217e72be41e2fd935779295986c10e 10.1.1.98:6379
   slots:[5461-10922] (5462 slots) master
M: a3f890d05a2ccb8648c46d8cabfe0752272a74ff 10.1.1.99:6379
   slots:[10923-16383] (5461 slots) master
S: ef86021ee4fc53f5d643512604b41604f5065d1c 10.1.1.100:6379
   replicates a3f890d05a2ccb8648c46d8cabfe0752272a74ff
S: f7f8362b66a8ed1508c961bbd2a26044475f4877 10.1.1.101:6379
   replicates 221c3e7f661f31dc842b07377250ac75fa996ab3
S: d765c528867f297cda8e943bce5dae07954286a2 10.1.1.102:6379
   replicates acc3fe09d0217e72be41e2fd935779295986c10e
Can I set the above configuration? (type 'yes' to accept): >>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
......................................................................................................................................................................................................................................................................................................................................
khteh commented 5 years ago 
$ k logs -f redis-cluster-5
1:C 28 Mar 2019 06:12:32.597 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 28 Mar 2019 06:12:32.597 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 28 Mar 2019 06:12:32.597 # Configuration loaded
1:M 28 Mar 2019 06:12:32.597 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 28 Mar 2019 06:12:32.597 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 28 Mar 2019 06:12:32.597 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 28 Mar 2019 06:12:32.598 * No cluster configuration found, I'm d765c528867f297cda8e943bce5dae07954286a2
1:M 28 Mar 2019 06:12:32.602 * Running mode=cluster, port=6379.
1:M 28 Mar 2019 06:12:32.602 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 28 Mar 2019 06:12:32.602 # Server initialized
1:M 28 Mar 2019 06:12:32.602 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 28 Mar 2019 06:12:32.602 * Ready to accept connections
1:M 28 Mar 2019 06:13:17.229 # configEpoch set to 6 via CLUSTER SET-CONFIG-EPOCH
1:M 28 Mar 2019 06:13:22.627 # Cluster state changed: ok
Javid907 commented 4 years ago 
$ k logs -f redis-cluster-5
1:C 28 Mar 2019 06:12:32.597 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 28 Mar 2019 06:12:32.597 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 28 Mar 2019 06:12:32.597 # Configuration loaded
1:M 28 Mar 2019 06:12:32.597 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
1:M 28 Mar 2019 06:12:32.597 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
1:M 28 Mar 2019 06:12:32.597 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
1:M 28 Mar 2019 06:12:32.598 * No cluster configuration found, I'm d765c528867f297cda8e943bce5dae07954286a2
1:M 28 Mar 2019 06:12:32.602 * Running mode=cluster, port=6379.
1:M 28 Mar 2019 06:12:32.602 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 28 Mar 2019 06:12:32.602 # Server initialized
1:M 28 Mar 2019 06:12:32.602 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 28 Mar 2019 06:12:32.602 * Ready to accept connections
1:M 28 Mar 2019 06:13:17.229 # configEpoch set to 6 via CLUSTER SET-CONFIG-EPOCH
1:M 28 Mar 2019 06:13:22.627 # Cluster state changed: ok

Hello. Can you resolved this problem?

sandeep1987elex commented 4 years ago

Can I set the above configuration? (type 'yes' to accept): yes

Nodes configuration updated Assign a different config epoch to each node Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join ......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ───────────────────────────────────────────────────────────────────────────────

I am also facing the same issue, please help me.. working on POC. Thank you in advance.

khteh commented 4 years ago

Redis cluster formation is done through k8s initialization batch job after all the pods of redis nodes are running. https://github.com/khteh/kubernetes/blob/master/redis-cluster-initjob.yml