[Feature request] Multi-master replication auto-configuration - REPLICAOF SLAVES

[ieslv]

Is your feature request related to a problem? Please describe. We use KeyDB in Multi-master configuration. As a configuration management system we use Amazon OpsWorks. When we add a new KeyDB node it is automatically configured to be replica of all nodes in the Layer and added to the load balancer.

Already working nodes:
Node A --> REPLICAOF Node B
Node B --> REPLICAOF Node A

Starting new node:
Node C --> REPLICAOF Node A, NodeB

Load balancer:
Node A
Node B
Node C

New replication settings:
Node A --> REPLICAOF Node B, Node C / restart
Node B --> REPLICAOF Node A, Node C / restart
Node C --> REPLICAOF Node A, Node B

Node C is a new, just started node. After some period of time, maybe just 30-60 seconds Node A and Node B will be automatically reconfigured to be replicated from the newly started node and KeyDB service will be restarted. As a new node can be added to the load balancer before if was added to the replication on already working Nodes A/B and we can have some inconsistency in the dataset, clients can receive the error about missing data if data will be added to the Node C and requested from Node A or Node B.

Describe the solution you'd like We can add a configuration key REPLICAOF SLAVES which should indicate KeyDB to automatically start replication from all its slaves.

Describe alternatives you've considered Right now we assume that we can have some inconsistency in the KeyDB data because of the lag in the configuration of the replication.

Additional context We also can consider to not serve request on KeyDB side if dataset is not yet replicated from the master nodes because we will receive an error. Such approach will help to not serve requests by e newly added node with zero keys and still not finished replication.

[yegors]

@ieslv Try the replica-serve-stale-data no config param.

# When a replica loses its connection with the master, or when the replication
# is still in progress, the replica can act in two different ways:
#
# 1) if replica-serve-stale-data is set to 'yes' (the default) the replica will
#    still reply to client requests, possibly with out of date data, or the
#    data set may just be empty if this is the first synchronization.
#
# 2) if replica-serve-stale-data is set to 'no' the replica will reply with
#    an error "SYNC with master in progress" to all the kind of commands
#    but to INFO, replicaOF, AUTH, PING, SHUTDOWN, REPLCONF, ROLE, CONFIG,
#    SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, PUNSUBSCRIBE, PUBLISH, PUBSUB,
#    COMMAND, POST, HOST: and LATENCY.
#

[ieslv]

@yegors, thank you for the information. With the replica-serve-stale-data no and in case of the Nginx as an LB it will pass connection to the replica but replica will return an error "SYNC with master in progress" and application probably should retry to connect to the cluster again. Nginx can pass connection to the next upstream only in case when it can't connect to the node(without any additional modules).

Maybe in case with HAProxy we can create some kind of check of the data on the replica.

[JohnSully]

@ieslv Yes its annoying a more advanced "down detection" is needed with HAProxy. We don't know in advance if a connection will be from a replica so we can't refuse connections outright.

[ieslv]

@JohnSully, the case with the "down detection" was an addition to the initial request. May we have an option like REPLICAOF SLAVES which will mean that the instance should be replicated from all its replicas?