OT-CONTAINER-KIT / helm-charts

A repository which that will contain helm charts with best and security practices.
https://ot-container-kit.github.io/helm-charts
46 stars 79 forks source link

Redis Sentinel - Does this Chart Deploy a Database or Just Sentinel? #182

Open charles-horel opened 4 months ago

charles-horel commented 4 months ago

We are trying to deploy an instance of the redis-sentinel chart. We were initially under the impression that the chart would build sentinel pods which also contain/manage standalone redis nodes.

Helm values (deploying latest version of sentinel chart)

---
redisSentinel:
  name: "redis-dev"
  clusterSize: 3
  redisSecret:
    secretName: "redis-rw"

serviceMonitor:
  enabled: true

redisExporter:
  enabled: **true**

When the sentinel pods start up, we can see hints in the logs that a redis server is starting:

7:X 06 Mar 2024 16:09:57.173 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo                                                                                                                                                                                                                                                                 │
│ 7:X 06 Mar 2024 16:09:57.173 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=7, just started          

However we cannot use a sentinel client to connect to a working redis instance. We will get 'No master found for 'myMaster' trying to use a python redis.Sentinel object. If I connect to the sentinel CLI from inside of the pod, however, and tell it to give me the master, it tells me it's 0.0.0.0/6379:

bash-5.1$ redis-cli -h localhost -p 26379
localhost:26379> SENTINEL get-master-addr-by-name myMaster
1) "0.0.0.0"
2) "6379"
localhost:26379> 

But there are no services or ports exposed on the sentinel pods on port 6379. Inside of the sentinel container, nothing is bound to this port, and there is no process running other than sentinel:

bash-5.1$ ps -ef
PID   USER     TIME  COMMAND
    1 redis     0:00 {entrypoint-sent} /bin/bash /usr/bin/entrypoint-sentinel.sh
    7 redis     0:50 redis-sentinel *:26379 [sentinel]
28270 redis     0:00 bash
28393 redis     0:00 ps -ef

bash-5.1$ netstat -alnp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:26379           0.0.0.0:*               LISTEN      7/redis-sentinel *:
tcp        0      0 172.31.4.56:48020       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:50026       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:47360       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:46686       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:49392       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:50028       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:49394       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:48672       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:26379       172.31.5.4:43808        ESTABLISHED 7/redis-sentinel *:
tcp        0      0 172.31.4.56:26379       172.31.5.4:41474        ESTABLISHED 7/redis-sentinel *:
tcp        0      0 172.31.4.56:48018       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:48670       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:26379       172.31.5.4:37292        ESTABLISHED 7/redis-sentinel *:
tcp        0      0 172.31.4.56:46684       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 172.31.4.56:47358       172.31.4.56:26379       TIME_WAIT   -
tcp        0      0 :::26379                :::*                    LISTEN      7/redis-sentinel *:
tcp        0      0 :::9121                 :::*                    LISTEN      -

It almost looks like I need to deploy standalone redis instances that get leveraged by Sentinel. Can someone confirm to me whether or not this is the case? If it is, would I deploy 3x standalones or would I use a cluster of 3 masters?

alita1991 commented 2 months ago

Hi, I have the same question, from what I see, the Redis instance is not created, just sentinel. Hope some information will be provided.

alita1991 commented 2 months ago

@charles-horel I explored the repository issues/discussion threads and found out that it is required to deploy redis-replication as well.

replication + sentinel

ot-operators   redis-operator-67bcf649cb-5gwrb   1/1     Running   0          3d1h
ot-operators   redis-replication-0               1/1     Running   0          3m6s
ot-operators   redis-replication-1               1/1     Running   0          2m57s
ot-operators   redis-replication-2               1/1     Running   0          2m48s
ot-operators   redis-sentinel-sentinel-2         1/1     Running   0          87s
ot-operators   redis-sentinel-sentinel-1         1/1     Running   0          53s
ot-operators   redis-sentinel-sentinel-0         1/1     Running   0          18s

Logs from a sentinel pod

7:X 26 Apr 2024 12:52:23.057 * Running mode=sentinel, port=26379.
7:X 26 Apr 2024 12:52:23.062 * Sentinel new configuration saved on disk
7:X 26 Apr 2024 12:52:23.062 * Sentinel ID is 5992766d4aa736de7aea2b701c60faff194b28be
7:X 26 Apr 2024 12:52:23.062 # +monitor master myMaster 10.42.1.149 6379 quorum 2
7:X 26 Apr 2024 12:52:23.064 * +slave slave 10.42.2.34:6379 10.42.2.34 6379 @ myMaster 10.42.1.149 6379
7:X 26 Apr 2024 12:52:23.069 * Sentinel new configuration saved on disk
7:X 26 Apr 2024 12:52:23.069 * +slave slave 10.42.0.13:6379 10.42.0.13 6379 @ myMaster 10.42.1.149 6379
7:X 26 Apr 2024 12:52:23.073 * Sentinel new configuration saved on disk
7:X 26 Apr 2024 12:52:59.533 * +sentinel sentinel 0e90f8b2c2d4179b31dc36c4d401fb76463f4220 10.42.2.35 26379 @ myMaster 10.42.1.149 6379
7:X 26 Apr 2024 12:52:59.539 * Sentinel new configuration saved on disk
7:X 26 Apr 2024 12:53:34.457 * +sentinel sentinel ad0de1c35e240ec546511971cf91ae3b5666720a 10.42.1.150 26379 @ myMaster 10.42.1.149 6379
7:X 26 Apr 2024 12:53:34.463 * Sentinel new configuration saved on disk