search

nekipelov / redisclient

Boost.asio based Redis-client library.
MIT License
321 stars 140 forks source link

Redis PUBSUB connection issue after idle period #52

Open gliatsos opened 6 years ago

gliatsos commented 6 years ago

I am using version 0.5.0 of the library and I have a pretty similar code as in the PUBSUB example. My application subscribes to a channel and receives messages. What I am facing is that every Monday, the application is not being able to receive messages from Redis.

Is there any timeout that I should handle in case the connection remains idle during the weekend? Shall I configure something extra in my application or in Redis to bypass this?

Thank you in advance, George

arrtchiu commented 5 years ago

I have the same issue here except using BLPOP with no timeout rather than publish/subscribe. If I stop pushing events to the queue for ~10 mins usually the most recent BLPOP doesn't fire my async callback.

Have not researched it yet, but at the moment I'm suspecting the underlying socket has timed out but the library hasn't been notified or fails to call the callback. It also does not call the callback installed by installErrorHandler.

Currently worked around it by setting a BLPOP timeout and re-sending the command each time. Kinda like an application-layer tcp-keepalive ;). You could add an extra "ping" message to your pub/sub but I can think of a lot of reasons why you probably wouldn't want to do that.

Will try get to the bottom of this... I don't like this kind of workaround!

In case it's useful to anyone else debugging this issue: I think I've only seen it occur on Linux. Mac doesn't seem to behave the same way.

mremski commented 5 years ago

@arrtchiu I've also seen that behavior on BLPOP using the sync client, our application uses very long lived connection in a few instances. I believe that the library defaults not sending tcp keepalives (you can set them), we tried setting a timeout on the BLPOP command itself and could still get "hung".
What seems to work for us: setCommandTimeout() on the client to 15 mins, BLPOP with timeout of 0. That definitely terminates the command at the 15 minute mark, and the error returned is End Of File (make sure you add the error parameter on the call to command). Unfortunately if you kill the connection to the server in the that timeout you get the same End Of File, so we treat timeout the same as socket close and call disconnect() on the client and go through a reconnect sequence.
We're running on Linux, both Fedora and Ubuntu, have seen the problem on both.

Oh, I just want to say Thanks to Alex for providing this.