search

debasishg / scala-redis-nb

Implementation of a non blocking Redis client in Scala using Akka IO
204 stars 38 forks source link

Reconnections fail from unknown termination #78

Closed fractaloop closed 10 years ago

fractaloop commented 10 years ago

So I've deployed the most recent 0.5 release with the reconnections, but the underlying issue of the client termination still plagues me. Worse, further calls to IO(Tcp) ! Connect(remote) fail and the JVM must be restarted for future connections.

This seems like it might be a bug in the IO layer, or perhaps the TcpPipeline, but the client dies so anonymously ("Child termination detected") it masks the underlying cause.

I am using a good deal of EVALSHA which might be causing pipeline problems, but the redis log doesn't have any errors. Do y'all have any idea what might be causing this?

debasishg commented 10 years ago

"Child termination detected" is supposed to print some more info. Does it say anything ? Is there any exception / stacktrace ?

fractaloop commented 10 years ago

I don't get much. What packages should I turn on debug logging for?

2014-04-29 17:08:18.077 ERROR com.redis.RedisConnection akka.tcp://Paparazzi@paparazzi0-int:2553/user/redis-client-8 - Child termination detected: Actor[akka://Paparazzi/user/redis-client-8/pipeline#-694129782]. Reconnecting in 100 ms...
debasishg commented 10 years ago

I was thinking of https://github.com/debasishg/scala-redis-nb/blob/master/src/main/scala/com/redis/RedisConnection.scala#L118 .. which has this x that I thought may give some information. But I was mistaken. Looks like we need to dig into the IO layer to see what's happening.

fractaloop commented 10 years ago

I did some deep debugging and its definitely a Terminated(connection) in TcpPipelineHandler. I'll keep looking.

fractaloop commented 10 years ago

This one is really strange. I added logging to TcpPipelineHandler, and it does in fact reconnect, and the client tells me it connected. After this, no further commands to Redis work. I haven't been able to reliably reproduce it.

fractaloop commented 10 years ago

I'm going to close this for now. I think this was an error in my own app and unrelated to reconnections. Sorry for the noise.

bearrito commented 10 years ago

Fractaloop, I may be running into the same issue as you described. Can you tell me what the issue and maybe fix was on your application side?

fractaloop commented 10 years ago

@bearrito I think it was because my Lua script had an error. :sweat: