search

IBMStreams / streamsx.dps

The IBMStreams/streamsx.dps GitHub repository is home to the Streams toolkit named DPS (Distributed Process Store)
http://ibmstreams.github.io/streamsx.dps
Other
4 stars 9 forks source link

Does the DPS toolkit support Redis Sentinel? #33

Closed Alex-Cook4 closed 4 years ago

Alex-Cook4 commented 7 years ago

I know we have support for Redis cluster and there is support for running "multiple redis instances and having the DPS toolkit automatically treat the instances as shards," but I don't see any documentation on Redis Sentinel which is the "official" high availability solution for Redis. Would it work to just put all of the masters and slaves into the config, and if a failover happens we will automatically find the new master?

Alex-Cook4 commented 7 years ago

I ran a quick test on this, and it would seem that the answer is no. It makes sense, since if we provide multiple Redis instances, the dps toolkit tries to shard the keys across these instances.

I ran a master/slave test, which led to half of the writes failing (when the write was being directed to the slave). I'll leave this open just in case there might be something I'm not thinking of. Thanks!

dakshiagrawal commented 7 years ago

@Alex-Cook4 - based on your experiments, is it a good idea for DPS to support Sentinel or stick with multiple redis instances and sharding by DPS?

Alex-Cook4 commented 7 years ago

With the introduction of the Redis Cluster as the method for sharding, it seems like there may be less of a need for the DPS-driven sharding, but I don't think they have to be mutually exclusive. We could support Redis Sentinel by haveing a "redis-sentinel" option, the way we have a "redis-cluster" option today.

I imagine the behavior for redis-sentinel would be:

  1. Take a list of all of the redis sentinels.
  2. Identify the current master using the following command to the Sentinel: SENTINEL get-master-addr-by-name master-name (more info here).
  3. In the event of failover (which we detect via failed writes to current master), we identify the new master by the same command.

We would also want the ability to iterate through each of the sentinels in case they are part of the failover.

In my reading, Redis Sentinel seems to be the go-to for high availability, so it would make sense for us to prioritize supporting it. One area that I don't know, is if the long-term goal of the Redis Cluster, is to supplant Redis Sentinel as the high availability approach. That may require more research.