Multiple masters with multiple sentinels

[cloudlms-co]

Hi there,

I have 9 physically separate hosts:

1. Sentinel1, Sentinel2 and Sentinel3
2. Master1, Master2 and Master3
3. Slave1, Slave2 and Slave3

Each of the sentinels is monitoring correctly each of the masters - so Sentinel1, monitors Master1, Master2 and Master3 - Sentinel2 and Sentinel3 do the same thing.

When I install the library, all works like a charm with the following config:

REDIS_SENTINEL_HOST="10.47.128.0, 10.43.0.0, 10.35.0.0" REDIS_PASSWORD=null REDIS_PORT=26379 REDIS_SENTINEL_SERVICE=masterone

Unfortunately that only works if the key-value pair is on the shard monitored by the "masterone" master->slave combo. If the key-value pair I'm setting/getting is located on let's say the "mastertwo" master->slave combo, I will get the dreaded "MOVED 164 xx.xx.xx.xx:6379" message.

Is there a way to also monitor multiple REDIS_SENTINEL_SERVICEs as is possible with multiple REDIS_SENTINEL_HOSTs ?

[cyrossignol]

Hi Michael,

Can you tell me a little more about your setup? It sounds like your application stores data in a Redis Cluster system. Normally, we don't need to run Sentinel servers alongside Redis Cluster because it supports replication, failover management, and redirection by itself. I wrote a bit more about it here.

We can tell the application to use Redis' native clustering by setting the 'cluster' option to 'redis' within the 'redis' configuration array in config/database.php:

...
'options' => [
    'cluster' => 'redis',
],
...

This feature is built-in to Laravel and does not require this package.

[cloudlms-co]

Hi Cy,

I was under the impression that redis-clustering is used to shard data across a number of master-slave instances and that sentinels - by monitoring each of the masters - would provide automatic failover so that if one of the masters becomes unresponsive, the sentinels agree that is the case via a quorum and than promote the failed master's slave to a master - all without the need for human intervention.

My setup is on Digital Ocean. I have 9 separate droplets: 3 sentinels, 3 masters and 3 slaves. Each of the sentinels is monitoring each of the masters. Each of the redis instances (whether sentinels, masters or slaves) runs within a docker container and they are all woven together through weave.works on the same network.

Am I missing something?

[cyrossignol]

Michael, the high-availability features of Sentinel and Cluster overlap--when we enable replication, Redis servers configured for Cluster mode will detect failures and elect a new master server for each replica group just like with Sentinel.

On the application side, the Redis client connects to the Cluster masters instead of Sentinels. We'd specify multiple masters in config/database.php for a high-availability configuration. If a connection fails, the client retries the command with a different master. The Predis and PhpRedis clients also discover master servers not included in the config file.

It sounds like you can turn off the three Sentinel droplets and reconfigure the Redis client for Redis Cluster. In this mode, the client will automatically handle slot mapping and the MOVED replies that redirect the command to the appropriate node.

Even though a Redis Cluster installation can also improve availability, there can be a non-trivial overhead for redirection and maintaining the distributed hash table. Sentinel exists for more simple Redis use cases that need availability but no extended storage capacity, so Redis provides both options. For example, we can spin-up one replicated Redis server each for cache, session, and queue, and it may be more flexible and easier to maintain with Sentinel than a Redis Cluster that stores all three. It works for some applications, but not others.

[cyrossignol]

Closing—Redis Cluster is outside the scope of this package. Please comment if you believe this is still an issue.