Closed BHare1985 closed 3 months ago
After double checking the configurations I realized that server-1 was setup incorrectly with
replicaof 1.1.1.1 27081
replicaof 3.3.3.3 27081
I am not sure if this contributed the issue I've been seeing. I am surprised server-1 hasn't complained about the issue in the log file but server-2 is the one that had issues. I will continue to monitor and see if server-2 issues come back now that server-1 is fixed
Turns out this was caused by having redis-sentinel
still installed and running. I initially installed redis and sentinel before switching to keydb, must of forgot to uninstall sentinel.
Describe the bug
I have one server in particular (server 2 of 3) that keeps trying to connect to itself as a master and then gets "Reading from master: Connection timed out". This is a huge issue for me, as it makes keydb respond VERY slowly, like 1000-2000ms when this issue is happening. Once I restart keydb it would fix itself and work for another ~24 hours. Here is the setup, each server is on a different VPS
server-1 (1.1.1.1)
server-2 (2.2.2.2)
server-3 (3.3.3.3)
I noticed that after sometime, normally everyday server-2 started to connect to master 127.0.0.1, but only after it had
CONFIG REWRITE failed: Permission denied
. So I read that on Debian that you sometimes have to chown /etc/keydb with keydb:keydbOnce I did this, the issue didn't go away but instead of connecting to 127.0.0.1 it started to try to connect to 2.2.2.2 (which was the public IP of the server). I looked at the config and this is what server-2 config was now:
To reproduce
Seems to be intermittent and only happens to server-2
Expected behavior
The server only tries to replicate the servers listed as "replicaof" and not try to set the master as itself.
Additional information
Attached is a log file of what happened, with the IP addresses changed to fit the explanation log.txt