When redis-cluster-proxy (r-c-p for short) is connected to the cluster using authentication, it fails to AUTH to the cluster when the connections get recycled.
Instance one (easy to reproduce): r-c-p uses AUTH, clients do not. Connect r-c-p to a simple 3-instance redis cluster, all masters. When opening the initial connections to the cluster, r-c-p issues AUTH and everything works. Now restart ("service redis restart" or similar) any of the 3 redis masters. R-c-p will properly reconnect to the cluster, however it will not issue AUTH and all commands coming from r-c-ps clients will be unauthenticated and thus fail if using ACL.
Instance two (tedious to reproduce): r-c-p uses AUTH, clients do not. Connect r-c-p like before. Simply wait as connections will get eventually recycled. When they do, r-c-p will not issue AUTH on the new connections, thus again failing its clients.
A similar but far worse bug exists when clients use AUTH themselves. I will open a separate bug report for that as it is probably going to be much more difficult to fix.
Restarting r-c-p fixes the issue, but it is obvious that idea has no chance to survive in production.
On the network level, the problem is reproduced both when r-c-p receives a FIN packet from the redis server (and then they perform an orderly connection shutdown) and when it receives a RST packet.
Redis-cluster-proxy used in this case is built from git commit ac83840d11 on Ubuntu-20.04.
When redis-cluster-proxy (r-c-p for short) is connected to the cluster using authentication, it fails to AUTH to the cluster when the connections get recycled.
Instance one (easy to reproduce): r-c-p uses AUTH, clients do not. Connect r-c-p to a simple 3-instance redis cluster, all masters. When opening the initial connections to the cluster, r-c-p issues AUTH and everything works. Now restart ("service redis restart" or similar) any of the 3 redis masters. R-c-p will properly reconnect to the cluster, however it will not issue AUTH and all commands coming from r-c-ps clients will be unauthenticated and thus fail if using ACL.
Instance two (tedious to reproduce): r-c-p uses AUTH, clients do not. Connect r-c-p like before. Simply wait as connections will get eventually recycled. When they do, r-c-p will not issue AUTH on the new connections, thus again failing its clients.
A similar but far worse bug exists when clients use AUTH themselves. I will open a separate bug report for that as it is probably going to be much more difficult to fix.
Restarting r-c-p fixes the issue, but it is obvious that idea has no chance to survive in production.
On the network level, the problem is reproduced both when r-c-p receives a FIN packet from the redis server (and then they perform an orderly connection shutdown) and when it receives a RST packet.
Redis-cluster-proxy used in this case is built from git commit ac83840d11 on Ubuntu-20.04.