search

dragonflydb / dragonfly-operator

A Kubernetes operator to install and manage Dragonfly instances.
https://www.dragonflydb.io/docs/managing-dragonfly/operator/installation
Apache License 2.0
141 stars 33 forks source link

Better handling of master failover #238

Open laurivosandi opened 2 months ago

laurivosandi commented 2 months ago

It looks like most applications do have Redis reconnect support if Redis operation results in error such as ReplyError: READONLY You can't write against a read only replica. In fact the ioredis docs specifically mention this usecase.

This is related to https://github.com/dragonflydb/dragonfly-operator/issues/149#issuecomment-1921063199

To better support dumb clients I would advise implementing TCP disconnect when Dragonfly pod loses master role to ensure clients reconnect to the new Dragonfly master pod

Pothulapati commented 1 month ago

Hmm, This makes sense but

To better support dumb clients I would advise implementing TCP disconnect when Dragonfly pod loses master role to ensure clients reconnect to the new Dragonfly master pod

How can we do this? Considering we don't have a proxy to do different replies. It's upto the older pod to return READONLY (which they will once they are configured as a replica). wdyt? 🤔

laurivosandi commented 1 month ago

I think it would be fine to issue TCP connection reset for every open connection if pod loses write master status. You can make this behaviour optional via command line flag or something like that