Command still hasn't been written into connection! Increase nettyThreads and/or retryInterval settings

[r13ljj]

We are experiencing the error what seems like network related errors using the 3.13.2 version of redisson.

Caused by: org.redisson.client.RedisTimeoutException: Command still hasn't been written into connection! Increase nettyThreads and/or retryInterval settings. Payload size in bytes: 0. Node source: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=null], connection: RedisConnection@1897737917 [redisClient=[addr=redis://10.0.2.185:6379], channel=[id: 0xcfc764bc, L:/172.xx.xx.xx:36380 - R:/10.0.2.185:6379], currentCommand=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@7fdadeee(failure: java.util.concurrent.CancellationException)], command=(GET), params=[[115, 101, 114, 118, 105, 99, 101, 58, 109, 101, ...]], codec=org.redisson.client.codec.ByteArrayCodec]], command: (GET), params: [[115, 101, 114, 118, 105, 99, 101, 58, 109, 101, ...]] after 3 retry attempts
at org.redisson.command.RedisExecutor$2.run(RedisExecutor.java:200)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted

config:

nettyThreads: 32
retryInterval: 1.5s

We run code in the docker of rancher, then have been occurred on some pods. so confused am I.

[mrniko]

Did you try to set nettyThreads to 64? Do you use blocking calls in async callbacks of Redisson commands?

[r13ljj]

Did you try to set nettyThreads to 64? Do you use blocking calls in async callbacks of Redisson commands?

pod set up "cpu: 2", it is too many. it have blocking calls, also have async.

[mrniko]

you should avoid to use blocking calls in Redisson async callbacks.

[r13ljj]

you should avoid to use blocking calls in Redisson async callbacks.

i confirm that it is a bug, all error use the connection which is the same one. please fix it @mrniko .

[mrniko]

can you share your config?

[r13ljj]

can you share your config?

i saw history issues about this point, i'll try to set config 'pingConnectionInterval=30000'

[SplotyCode]

We have the same problem. This is does not seem to be related to netty threads at all. Nearly all threads are runnable and we still get this exception some time. If a client is effected it rejects nearly every command.

[mrniko]

@SplotyCode Can you make a memory dump at that moment?

[SplotyCode]

We made one but I am not allowed to open it myself. I was told that there is nothing unusually. If you tell me would you want to see j can ask an administrator

[mrniko]

@SplotyCode

how many connections affected?

[fumanix]

We have the same problem. How to solve？ solution ？

[ayu-webmaster]

We are also facing the same problem. Guys any solution on this ?

[mrniko]

@ayu-webmaster

Make sure that you don't use blocking calls in asyn/rx/reactive handlers.

[JanCizmar]

What do you mean by blocking commands? Can you show an example of wrong usage?

[mrniko]

@JanCizmar

Calls which blocks execution thread.