async-redis with falcon

[tleish]

We are using both a Rails and Falcon server. On the falcon server we connect to Redis using async-redis. Recently our hosting provider upgraded our redis cluster. After the upgrade the Rails server reconnected to Redis without issue, however the falcon server could not see new data written to Redis. It did it throw a redis connection error either. It's almost as if it kept hold of an other connection.

We setup our Redis client as a singleton to persist the connection, but it this a bad approach? Is there a better way to do this?

[ioquatix]

Async::Redis uses persistent connections.

So, it seems like your hypothesis is correct. If you terminate the old server, it will cause the persistent connections to close. Were both servers still running? Did you switch the hostname? Persistent connections would stick to the old server in that case.

[tleish]

Were both servers still running?

We do not know. We host our application on AWS and it's not clear to us when they upgrade if they keep the old one running. Our theory is the old one keeps running until all clients release their connection to the old one and connect to the new one, which could explain why we didn't see any errors logged.

Did you switch the hostname?

The hostname for the redis cluster did not change (which is what we point our applications to). I do not know if the hostname for the individual redis servers behind the cluster changed.

In falcon, we only establish the connection on the first request and persist it from then on. I assume Rails spawns a new connection with each new request, which is why it found the new connection, but have not yet confirmed. Not as efficient as only how we set it up in falcon, but reconnecting on redis upgrades (which happen multiple times a year). I'm exploring alternative solutions for falcon with Async::Redis to handle an upgrade occurrence like this.

[ioquatix]

Is there any way you can do a rolling restart of falcon when the redis upgrade happens? It seems like that would be a problem that would affect any system with persistent redis connections.

[tleish]

I can stop auto-updates of Redis on AWS, perform the upgrades manually and then restart falcon servers, but looking for a solution to avoid the manual process. It's not needed for the rails servers and I'd prefer the redis updates can occur without manual intervention.

[ioquatix]

My plan is to add a pluggable policy to connection pools.

You could add a fixed maximum life time or idle time.

But the problem would still stand.

The correct way should be for the instance of Redis that is going down, to kill all idle connections - or even ones in a transaction (it should be retried). I don't know if we can sufficiently solve this on the application side, unless you just want to make a new connection for every request and pay that latency overhead.

[tleish]

Thanks for the response. Any knowledge redis-rails gem to know if it resolved because it does persist the connection?