Automatically refresh connections to redis cluster on shard/replica addition/deletion

akshaypatidar1999 commented 2 years ago

Describe the feature

Refresh connections to redis cluster whenever there is a change in cluster

Current redis client implementation creates connections to redis cluster and uses them to send requests to redis cluster. However if redis cluster is changed (new shards/replicas are added/deleted) then it does not automatically refreshes the connection, due to which we see errors like MOVED 7652 10.0.4.210:6379 and we have to redeploy our service.

If we could somehow detect changes in the cluster and refresh connections then we can get rid of deploying whenver we scale/downscale redis cluster

One simple solution I can think of is to save the cluster information when connecting to cluster for the first time using CLUSTER INFO/CLUSTER SLOTS command. Then periodically run this command and if there is any change in the cluster we can refresh the connections.

Use cases

In today's world we upscale/downscale our data stores on the fly so it becomes cumbersome to redeploy micro-services after each scaling activity. This feature will remove the need to redeploy micro-services when scaling a redis cluster

Contribution

I am willing to contribute to this feature.

pmlopes commented 2 years ago

Hi @akshaypatidar1999 thanks for your issue. This is indeed a highly requested feature. We can break the task into 2 areas:

have the cluster discovery code run as a "singleton" per redis client instance
close the client should become an asynchronous call to ensure that any discovery task is close properly

The same, similar logic should also apply to sentinel mode.

This also means that the static factories createClient() : Redis should become asynchronous too, as we may need to perform discovery for cluster/sentinel.

So, we either deprecate the old blocking API or leave it with a warning on the javadoc that that mode will only work for single node or replication mode.

akshaypatidar1999 commented 9 months ago

Is there any plan to pick this feature? I am willing to contribute to this feature.

ivivanov-bg commented 5 months ago

Are there any plans for implementing this ? We are facing similar issue in quarkus when the bitnami redis running in sentinel mode switches the master node.

I assume the root cause is the same as described here

akshaypatidar1999 commented 5 months ago

@pmlopes @vietj I will happy to contribute to this feature. We have internally created a wrapper over vertx redis client in which we periodically fetch the cluster state using CLUSTER SLOTS and CLUSTER INFO command and store the state in memory. Whenever we detect any change in the state we create new connections to the redis cluster and close existing connections gracefully. Similar approach can be used here. What do you gus think?

ivivanov-bg commented 5 months ago

@akshaypatidar1999 If you have any ideas - I guess you can fork the repo and create a PR

vert-x3 / vertx-redis-client