dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
25.92k stars 950 forks source link

Replication stops when master becomes replica of another instance #2044

Open safa-topal opened 1 year ago

safa-topal commented 1 year ago

Describe the bug During a migration from Redis to Dragonfly via replication protocol (replicaof command), when replication is already working between Dragonfly master and replica, master successfully becomes the replica of another instance(other Redis service); however then Dragonfly master stops replicating to its already defined replica.

To Reproduce Steps to reproduce the behavior:

  1. Enable replication between two Dragonfly instances, one becomes master and the other becomes replica
  2. Set replicaof on the master with another Redis service (v 7.0.12) to initiate migration of data from the Redis instance to Dragonfly master instance
  3. Although replication between Redis service and Dragonfly master works as expected, Dragonfly master stops replicating to existing Dragonfly replica
  4. Issue replicaof no one on the Dragonfly master node
  5. Replication still does not happen between Dragonfly master and replica

Logs on Dragonfly master after running replicaof {redis_host} {redis_port} on the master Dragonfly instance to initiate replication with Redis(v7.0.12) service:

Oct 20 11:38:06 dragonfly-7d2f293-1 dragonfly[71]: I20231020 11:38:06.382553    73 server_family.cc:1632] Replicating ::1:6379
Oct 20 11:38:06 dragonfly-7d2f293-1 dragonfly[71]: I20231020 11:38:06.440667    73 protocol_client.cc:219] Resetting endpoint! ::1, 6379
Oct 20 11:38:11 dragonfly-7d2f293-1 dragonfly[71]: I20231020 11:38:11.064666    73 replica.cc:355] Starting full sync with Redis master
Oct 20 11:38:11 dragonfly-7d2f293-1 dragonfly[71]: W20231020 11:38:11.341627    73 rdb_load.cc:2220] Unrecognized RDB AUX field: 'aof-base'
Oct 20 11:38:11 dragonfly-7d2f293-1 dragonfly[71]: I20231020 11:38:11.428987    73 replica.cc:521] Transitioned into stable sync

Logs on Dragonfly master after running replicaof no one on the Dragonfly master to disconnect it from the Redis service:

Oct 20 11:54:42 dragonfly-7d2f293-1 dragonfly[71]: I20231020 11:54:42.923535    73 server_family.cc:1632] Replicating no:one
Oct 20 11:54:42 dragonfly-7d2f293-1 dragonfly[71]: E20231020 11:54:42.924376    73 protocol_client.cc:322] Socket error system:103
Oct 20 11:54:42 dragonfly-7d2f293-1 dragonfly[71]: W20231020 11:54:42.924441    73 replica.cc:229] Error stable sync with ::1:6379 system:103 Software caused connection abort

Expected behavior Replication should anyhow continue between Dragonfly master and replica even though Dragonfly master becomes the replica of another service.

Environment (please complete the following information):

Additional context Behavior works as expected for Redis 7.0.12

kostasrim commented 1 year ago

Hi @safa-topal,

we do not support replicating the replica. Is there a particular reason you want to do that ?

safa-topal commented 1 year ago

Hi @kostasrim, thanks for the response.

Yes, the reason is our platform supports migrating from a Redis service to another one without any disruption to the replication of the target service (on this case Dragonfly becomes the target service) and we were aiming to have the same support for Dragonfly. Otherwise workaround seems like reissuing replicaof {master_host} {master_port} on the Dragonfly replica to keep replication continuing after the initial migration finishes.

romange commented 1 year ago

I suggest to employ the workaround.