How to configure a writable replica in a Redis Cluster

[srishtikeshari]

I have a Redis cluster setup. I have set the 'replica-read-only' to 'no' to make the replicas writable.

On performing any 'STORE' operation like 'ZRANGESTORE' or 'ZUNIONSTORE' on a replica node of the Cluster, I a getting MOVED exception. Is there any other configuration required to make a replica writable? I setup a cluster with 6 nodes(3 primary, each with 1 replica). Following is the nodes configuration -

I created a set on the nodes on a primary node(node on 8004 port) -

When I perform a 'ZUNIONSTORE' operation on its secondary node(i.e. node on 8001 port), i get MOVED exception -

Redis version - Redis server v=7.0.11 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=44169fc7b347b8ec

My use case for Writable replica - I am trying to perform secondary indexing in Redis as per https://redis.io/docs/manual/patterns/indexes/. For secondary indexing, I use commands like 'ZRANGESTORE', 'ZUNIONSTORE' and 'ZINTERSTORE', store the results in temporary keys. As there a multiple fields on which the data is indexed, there are multiple set commands('ZRANGESTORE', 'ZUNIONSTORE' and 'ZINTERSTORE') done in a transaction and I delete all the temporary keys used for storing intermediate results in the process in the same transaction. This works well in a single node Redis setup but now I want to use a Redis cluster instead of a single node. I want to use the replica nodes to perform all the set(indexing) operations and use the primary node just for writing data, and hence, require the write('STORE') commands in the replica node.

[hwware]

In redis.conf, there is one option named replica-read-only yes. If you disable it, replica should allow writing operation. But be care using it, Thanks

[srishtikeshari]

I have set the replica-read-only to 'no' Still facing the error. Following is the redis.conf I am using for all nodes-

port <port_number>
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
protected-mode no
dir <path>
replica-read-only no

Is there something I am missing?

[chijinxina]

@srishtikeshari You can try to use the -c parameter redis-cli -c -h 10.240.172.156 -p 8001

[srishtikeshari]

@srishtikeshari You can try to use the -c parameter redis-cli -c -h 10.240.172.156 -p 8001

On using -c, the commands get executed but on the master node.

[hpatro]

@srishtikeshari I looked at the code, writeable replica(s) seems to be not a possible option with cluster enabled setup. Currently, the server redirect(s) the client to the primary if it's a writeable command.

The config replica-read-only is only used for non cluster-enabled scenario.

@itamarhaber Could you explain how can it be configured in cluster-enabled scenario ? (https://github.com/redis/redis/discussions/9760)

[hpatro]

I think this feature would be beneficial for local caching/aggregation and also non breaking change while migrating to cluster-enabled setup. We've talked in the past for allowing additional database support in Cluster V2(https://github.com/redis/redis/issues/8948) for similar purpose but that won't work for replica for similar reasons (write commands on replica).

@madolson Any thoughts why this has not be builtin?

[madolson]

Any thoughts why this has not be builtin?

I assume this was more of an oversight than anything else. We haven't really wanted to support writable replicas, for various reason. If we wanted to add support for CME, I assume we would rather try to implement in a more sustainable long term way.

My suggestion would be that a replica could be able to create data that was either transient to a transaction (either multi or script) or live in some namespace that is not persisted across failovers. This separate namespace could be a separate database.

[hpatro]

Should we take a two stepped approach to allow having similar behavior to CMD.

Short term (possibly #12525):

• Piggyback on replica-read-only flag in CME to allow write commands on replica if the config is set to false.
• temporary keys created on replica gets overwritten if primary mutates the key.
• Cleanup ephemeral data created on replica if the slot migration happens from the primary to another node.
• Clients could use hashtags to benefit of colocality of actual data with temporary data generated on replica to coexist together. If they use arbitrary key they might redirected with existing redirection logic.

Long term approach like you mentioned we could think of having a separate namespace for local aggregation/caching. We have discussed this in the past #8948 , #5766.

[soloestoy]

I don't think we need support writable replica in cluster mode, it is not a good design, and we should avoid spreading it to other areas. Maintaining the current state is already sufficient and should be the bottom line.

[madolson]

I generally agree with soloestoy. To add some more detail though.

temporary keys created on replica gets overwritten if primary mutates the key.

This is complex, because when we receive the mutation, we don't know know the full key. The replication stream just includes the mutations.

Cleanup ephemeral data created on replica if the slot migration happens from the primary to another node.

I don't think we want any ephemeral date to exist on the replica outside of the context of a command. I think there is an argument to be made for allowing "temporary" data within a lua script or a multi.

[hpatro]

@srishtikeshari Could you please explain how do you use writeable replica feature?

[srishtikeshari]

@srishtikeshari Could you please explain how do you use writeable replica feature?

@hpatro I am not using it currently, I have a use-case for writable replica in Cluster as mentioned in the question and was checking if this is possible. As per https://github.com/redis/redis/discussions/9760, I was under the impression that writable replicas are possible in Redis cluster. But from the discussion here, it looks like it is not supported.

@itamarhaber From https://github.com/redis/redis/discussions/9760, it looks like writable replicas were supported in Cluster. Can you please comment if it can still be configured?