Snapchat / KeyDB

A Multithreaded Fork of Redis
https://keydb.dev
BSD 3-Clause "New" or "Revised" License
11.01k stars 564 forks source link

[HELP] active-replica READONLY You can't write against a read only replica #838

Open bilias opened 1 month ago

bilias commented 1 month ago

I'm new to keydb but long time redis/sentinel admin.

Can't make active replication to work in read/write mode. It only works in READONLY mode on both nodes. I always get:

(error) READONLY You can't write against a read only replica.

Connection between them is fine, and replication works (I see old data from each node on both of them).

I'm following the documentation config:

requirepass xxx
masterauth xxx
active-replica yes
# replicaof 10.3.8.22 6379
node1> replicaof 10.3.8.22 6379

node1-log:
3130:3140:S 27 May 2024 22:05:52.198 * MASTER <-> REPLICA sync: Finished with success

node2-log:
3008:3017:M 27 May 2024 22:05:52.194 * Synchronization with replica 10.3.8.21:6379 succeeded

node2> replicaof 10.3.8.21 6379

node2-log:
3008:3017:S 27 May 2024 22:06:36.301 * MASTER <-> REPLICA sync: Finished with success

node1-log:
3130:3140:S 27 May 2024 22:06:36.297 * Synchronization with replica 10.3.8.22:6379 succeeded

 > set foo test
(error) READONLY You can't write against a read only replica.

This happens on both nodes

node1> info replication
# Replication
role:active-replica
master_global_link_status:up
connected_masters:1
master_host:10.3.8.22
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_read_repl_offset:5035
slave_repl_offset:5035
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:1
slave0:ip=10.3.8.22,port=6379,state=online,offset=5742,lag=1
master_failover_state:no-failover
master_replid:1564bd9086965987a1b09ea2096d72fc1f008510
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:5742
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:5742

node2> info replication
# Replication
role:active-replica
master_global_link_status:up
connected_masters:1
master_host:10.3.8.21
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_read_repl_offset:6729
slave_repl_offset:6729
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:1
slave0:ip=10.3.8.21,port=6379,state=online,offset=6022,lag=1
master_failover_state:no-failover
master_replid:d18697b500d9cbda8b1d24ae588177128a07326d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:6022
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:6022

Both of them have: slave_read_only:1

> READWRITE
OK

but no change. Probably this is for cluster mode.

Even tried replica-read-only no didn't change anything.

Is this a bug or am I missing something? Documentation does not mention anything else on active replication.

Version is keydb-6.3.4-3.el9.x86_64 on Rocky 9. rpm comes from epel.

Thanks

bilias commented 1 month ago

ok, this is strange.

Problem is solved if I put my include /etc/keydb/local.conf at the end of keydb.conf and not at the top