Need to expose Redis cluster to outside kubernetes cluster

[ravina-c]

Hi All

I want to expose the redis cluster outside K8s cluster using load balancer. In the process, I am not able to connect to all the redis nodes created when accessed from outside. It is connecting to one node at a time.

Below I have described how i tried to use Nodeport service : I manually set the following in a config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-external-config
data:
  redis-additional.conf: |
    cluster-announce-ip <node ip>
    cluster-announce-tls-port 6379
    cluster-announce-port 0
    cluster-announce-bus-port 6380

And the redis cluster yaml is :

  ---
apiVersion: redis.redis.opstreelabs.in/v1beta1
kind: RedisCluster
metadata:
  name: redis-cluster
spec:
  clusterSize: 3
  clusterVersion: v7
  securityContext:
    runAsUser: 1000
    fsGroup: 1000
  persistenceEnabled: true
  kubernetesConfig:
    image: quay.io/opstree/redis:v7.0.5
    imagePullPolicy: IfNotPresent
    service:
      serviceType: NodePort
  redisExporter:
    enabled: true
    image: quay.io/opstree/redis-exporter:v1.44.0
  redisLeader:
    redisConfig:
      additionalRedisConfig: redis-external-config
    readinessProbe:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 15
      successThreshold: 1
      timeoutSeconds: 5
    livenessProbe:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 15
      successThreshold: 1
      timeoutSeconds: 5
  redisFollower:
    redisConfig:
      additionalRedisConfig: redis-external-config
    readinessProbe:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 15
      successThreshold: 1
      timeoutSeconds: 5
    livenessProbe:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 15
      successThreshold: 1
      timeoutSeconds: 5
  storage:
    volumeClaimTemplate:
      spec:
        # storageClassName: standard
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

[drivebyer]

try this

1. build operator from the latest code
2. apply https://github.com/OT-CONTAINER-KIT/redis-operator/blob/master/tests/e2e-chainsaw/v1beta2/nodeport/redis-cluster/cluster.yaml

[ravina-c]

Hi @drivebyer , We tried as you said and now we have a node port for every Redis node. But when we tried to connect with redis from outside the cluster using redis-cli, we are not able to perform set and get operations, getting connection refused.

redis-cli -c -h <nodeip> -p 31164
<nodeip>:31164> set 1 1
-> Redirected to slot [9842] located at <nodeip>:32171
Could not connect to Redis at <nodeip>:32171: Connection refused
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected> get 1
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected>

Also, from within the cluster when I give keys * command, it retrieves the keys only from one node to which it is connected. Should we add something to get all the keys stored in Redis, or is the functionality itself is to retrieve only the keys from the node to which it is connected to?

[drivebyer]

We tried as you said and now we have a node port for every Redis node. But when we tried to connect with redis from outside the cluster using redis-cli, we are not able to perform set and get operations, getting connection refused.

redis-cli -c -h <nodeip> -p 31164
<nodeip>:31164> set 1 1
-> Redirected to slot [9842] located at <nodeip>:32171
Could not connect to Redis at <nodeip>:32171: Connection refused
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected> get 1
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected>

Try to connect :32171 out of cluster, see whether it connected or not

[ravina-c]

We tried as you said and now we have a node port for every Redis node. But when we tried to connect with redis from outside the cluster using redis-cli, we are not able to perform set and get operations, getting connection refused.

redis-cli -c -h <nodeip> -p 31164
<nodeip>:31164> set 1 1
-> Redirected to slot [9842] located at <nodeip>:32171
Could not connect to Redis at <nodeip>:32171: Connection refused
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected> get 1
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected>

Try to connect :32171 out of cluster, see whether it connected or not

For that port also, we get the same issue.

redis-cli -c -h nodeip -p 32171
nodeip:32171> set 1 1
OK
nodeip:32171> get 1
"1"
nodeip:32171> set 2 2
OK
nodeip:32171> get 2
"2"
nodeip:32171> set 3 3
-> Redirected to slot [1584] located at nodeip:31164
Could not connect to Redis at nodeip:31164: Connection refused
Could not connect to Redis at nodeip:31164: Connection refused
not connected>

[drivebyer]

We tried as you said and now we have a node port for every Redis node. But when we tried to connect with redis from outside the cluster using redis-cli, we are not able to perform set and get operations, getting connection refused.

redis-cli -c -h <nodeip> -p 31164
<nodeip>:31164> set 1 1
-> Redirected to slot [9842] located at <nodeip>:32171
Could not connect to Redis at <nodeip>:32171: Connection refused
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected> get 1
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected>

Try to connect :32171 out of cluster, see whether it connected or not

For that port also, we get the same issue.

redis-cli -c -h nodeip -p 32171
nodeip:32171> set 1 1
OK
nodeip:32171> get 1
"1"
nodeip:32171> set 2 2
OK
nodeip:32171> get 2
"2"
nodeip:32171> set 3 3
-> Redirected to slot [1584] located at nodeip:31164
Could not connect to Redis at nodeip:31164: Connection refused
Could not connect to Redis at nodeip:31164: Connection refused
not connected>

what's the resource yaml. let me reproduce.

[ravina-c]

We tried as you said and now we have a node port for every Redis node. But when we tried to connect with redis from outside the cluster using redis-cli, we are not able to perform set and get operations, getting connection refused.

redis-cli -c -h <nodeip> -p 31164
<nodeip>:31164> set 1 1
-> Redirected to slot [9842] located at <nodeip>:32171
Could not connect to Redis at <nodeip>:32171: Connection refused
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected> get 1
Could not connect to Redis at <nodeip>:32171: Connection refused
not connected>

Try to connect :32171 out of cluster, see whether it connected or not

For that port also, we get the same issue.

redis-cli -c -h nodeip -p 32171
nodeip:32171> set 1 1
OK
nodeip:32171> get 1
"1"
nodeip:32171> set 2 2
OK
nodeip:32171> get 2
"2"
nodeip:32171> set 3 3
-> Redirected to slot [1584] located at nodeip:31164
Could not connect to Redis at nodeip:31164: Connection refused
Could not connect to Redis at nodeip:31164: Connection refused
not connected>

what's the resource yaml. let me reproduce.

apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: RedisCluster
metadata:
  name: redis-cluster-v1beta2
spec:
  clusterSize: 3
  clusterVersion: v6
  persistenceEnabled: true
  podSecurityContext:
    runAsUser: 1000
    fsGroup: 1000
  kubernetesConfig:
    service:
      serviceType: NodePort
    image: quay.io/opstree/redis:v6.2.14
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 101m
        memory: 128Mi
      limits:
        cpu: 101m
        memory: 128Mi
  redisExporter:
    enabled: true
    image: quay.io/opstree/redis-exporter:v1.44.0
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
  storage:
    volumeClaimTemplate:
      spec:
        # storageClassName: standard
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
    nodeConfVolume: true
    nodeConfVolumeClaimTemplate:
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

[drivebyer]

@ravina-c I test it, it works for me:

Maybe you should check firewall or something.

[ravina-c]

@ravina-c I test it, it works for me:

Maybe you should check firewall or something.

Hi @drivebyer , Even I am able to connect from within the cluster.

bash-5.1$ cat /node-conf/nodes.conf f5d0379c9096437d6573e3ce84784f6df1c080d2 nodeip:31554@30948 slave 421b22a61839d15f5e32ef3af9e9f63882659a87 0 1705041527181 3 connected 421b22a61839d15f5e32ef3af9e9f63882659a87 nodeip:31056@30437 master - 0 1705041528000 3 connected 10923-16383 7d54bc66fe18bbeebcb4b40031d8599395d59da8 nodeip:31042@31581 slave 9d190d3c7516e0b7032eb36dbef0eaf89cd1b734 0 1705041529392 1 connected 45943c2647f6cfb788c5fb02b7004413a00b636f nodeip:31336@30572 master - 0 1705041529000 2 connected 5461-10922 f4d30785fcf1fe8cba77105f12c939132fc730f6 nodeip:30021@31786 slave 45943c2647f6cfb788c5fb02b7004413a00b636f 0 1705041528387 2 connected 9d190d3c7516e0b7032eb36dbef0eaf89cd1b734 nodeip:30672@31539 myself,master - 0 1705041527000 1 connected 0-5460 vars currentEpoch 3 lastVoteEpoch 0 bash-5.1$ redis-cli -c -h nodeip -p 31056 nodeip:31056> set a a OK nodeip:31056> set b b -> Redirected to slot [3300] located at nodeip:30672 OK nodeip:30672> get a -> Redirected to slot [15495] located at nodeip:31056 "a" nodeip:31056> get b -> Redirected to slot [3300] located at nodeip:30672 "b" nodeip:30672>

But, we want to connect from outside the cluster. For eg, we have two cluster - Cluster A and cluster B and we have redis deployed in cluster A and we are able to access redis cluster from cluster A as shown above. But when we are trying to access the redis from Cluster B via nodeport, we face the issue mentioned earlier .

redis-cli -c -h nodeip -p 32171
nodeip:32171> set 1 1
OK
nodeip:32171> get 1
"1"
nodeip:32171> set 2 2
OK
nodeip:32171> get 2
"2"
nodeip:32171> set 3 3
-> Redirected to slot [1584] located at nodeip:31164
Could not connect to Redis at nodeip:31164: Connection refused
Could not connect to Redis at nodeip:31164: Connection refused
not connected>

Get and set commands work fine until it is being stored in the same node. Getting connection refused, when it is getting redirected to another node or hash slot from outside the cluster (i.e., cluster B)

[drivebyer]

@ravina-c Try to test port in cluster A from cluster B, such as nc tool.