search

redisson / redisson

Redisson - Easy Redis Java client and Real-Time Data Platform. Valkey compatible. Sync/Async/RxJava/Reactive API. Over 50 Redis based Java objects and services: Set, Multimap, SortedSet, Map, List, Queue, Deque, Semaphore, Lock, AtomicLong, Map Reduce, Bloom filter, Spring Cache, Tomcat, Scheduler, JCache API, Hibernate, RPC, local cache ...
https://redisson.pro
Apache License 2.0
23.15k stars 5.33k forks source link

redission Unable to send command #5969

Open bitkbitk opened 2 months ago

bitkbitk commented 2 months ago

there are 2 hosts in our environment, after running several weeks, one host report the error below, the other host runs normal.

netty threads num has been set to 256. redis.stream.gateway.nettyThreads=256

error log as below:

org.redisson.client.RedisTimeoutException: Unable to send command! Try to increase 'nettyThreads' and/or connection pool size settings Node source: NodeSource [slot=null, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:49:queue:0, freeConnectionsAmount=201, freeConnectionsCounter=value:498:queue:0, freezed=false, freezeReason=null, nodeType=MASTER, firstFail=0]]], connection: RedisConnection@1834726787 [redisClient=[addr=com:6379], channel=[id: 0xb46628b8, ], currentCommand=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@69642cf0(failure: java.util.concurrent.CancellationException)], command=(HSCAN), params=[pctHeartBeatCache, 862, COUNT, 10], codec=org.redisson.codec.FstCodec]], command: (EVAL), params: [local v = redis.call('hget', KEYS[1], ARGV[1]); redis.call('hset', KEYS[1], ARGV[1], ARGV[2]); retur..., 1, liveHeartBeatCache, PooledUnsafeDirectByteBuf(ridx: 0, widx: 41, cap: 256), PooledUnsafeDirectByteBuf(ridx: 0, widx: 239, cap: 256)] after 3 retry attempts at org.redisson.command.CommandAsyncService$6.run(CommandAsyncService.java:714) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:682) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:757) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:485) at java.lang.Thread.run(Thread.java:748)

org.redisson.client.RedisTimeoutException: Unable to send command! Try to increase 'nettyThreads' and/or connection pool size settings Node source: NodeSource [slot=0, addr=null, redisClient=[addr=redis://com:6379], redirect=null, entry=null], connection: RedisConnection@1834726787 [redisClient=[addr=redis://com:6379], channel=[id: 0xb46628b8, ], currentCommand=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@69642cf0(failure: java.util.concurrent.CancellationException)], command=(HSCAN), params=[pctHeartBeatCache, 862, COUNT, 10], codec=org.redisson.codec.FstCodec]], command: (HSCAN), params: [liveHeartBeatCache, 2986, COUNT, 10] after 3 retry attempts at org.redisson.command.CommandAsyncService$6.run(CommandAsyncService.java:714) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:682) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:757) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:485) at java.lang.Thread.run(Thread.java:748)

mrniko commented 2 months ago

Can you share the full stacktrace?

mrniko commented 2 months ago

Which Redisson version?

bitkbitk commented 2 months ago

Which Redisson version?

3.11.1

bitkbitk commented 2 months ago

Which Redisson version?

we have restarted our service yesterday, after that, the error log is no longer reported

bitkbitk commented 2 months ago

Do you think this problem is related to bigkey? But I see that even if a certain key is read in full, the operation taken here is hscan, count=10, so can the bigkey cause be ruled out?

bitkbitk commented 2 months ago

Or should we continue to increase the number of nettythreads? What is the recommended increase?

mrniko commented 2 months ago

There are could be multiple causes: https://github.com/redisson/redisson/wiki/16.-FAQ#q-what-is-the-cause-of-redistimeoutexception

cooker commented 1 month ago

2024-07-16 22:00:44.706 [cms] ERROR [14f07ba06332d970] http-nio-19306-exec-51 com.x2era.xcloud.cms.exception.GlobalExceptionHandler-50 [服务异常:] org.springframework.dao.InvalidDataAccessApiUsageException: Unable to send command! Node source: NodeSource [slot=null, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=1, freeSubscribeConnectionsCounter=value:50:queue:0, freeConnectionsAmount=31, freeConnectionsCounter=value:63:queue:0, freezed=false, freezeReason=null, client=[addr=redis://10.226.139.7:6379], nodeType=MASTER, firstFail=0]]], connection: RedisConnection@387476549 [redisClient=[addr=redis://10.226.139.7:6379], channel=[id: 0x96f5727f, L:/10.226.137.15:47502 - R:10.226.139.7/10.226.139.7:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@19df90d6(failure: java.util.concurrent.CancellationException)], command=(GET), params=[[102, 109, 97, 112, 112, 58, 99, 109, 115, 58, ...]], codec=org.redisson.client.codec.ByteArrayCodec]], current command in queue: CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@19df90d6(failure: java.util.concurrent.CancellationException)], command=(GET),

mrniko commented 1 month ago

@cooker

Please refer to https://github.com/redisson/redisson/issues/5969#issuecomment-2187995978