redisson / redisson

Redisson - Easy Redis Java client and Real-Time Data Platform. Valkey compatible. Sync/Async/RxJava/Reactive API. Over 50 Redis or Valkey based Java objects and services: Set, Multimap, SortedSet, Map, List, Queue, Deque, Semaphore, Lock, AtomicLong, Map Reduce, Bloom filter, Spring, Tomcat, Scheduler, JCache API, Hibernate, RPC, local cache...
https://redisson.pro
Apache License 2.0
23.26k stars 5.34k forks source link

RedisTimeoutException: Command execution timeout for command #4938

Closed SheroZhong closed 1 year ago

SheroZhong commented 1 year ago

Hello, here are the errors reported by my program: My service is deployed in k8s, with 5 pods. After running for 1 day, 1 to 2 pods will have the following errors. Other pods are running normally, but over time, they will also have errors

2023-03-29 10:22:41.716 [ERROR] [pid] [] [] [] [redisson-timer-6-1] [o.r.c.handler.PingConnectionHandler 97] Unable to send PING command over channel: [id: 0xa34aa017, L:/10.90.140.103:53134 - R:10.1.128.56/10.1.128.56:7379]
org.redisson.client.RedisTimeoutException: Command execution timeout for command: (PING), params: [], Redis client: [addr=redis://10.1.128.56:7379]
    at org.redisson.client.RedisConnection.lambda$async$0(RedisConnection.java:244)
    at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:668)
    at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:743)
    at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:471)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

2023-03-29 10:44:15.353 [ERROR] [pid] [] [] [] [redisson-netty-4-37] [o.r.c.SentinelConnectionManager 372] Can't execute SENTINEL commands on 10.1.128.55/10.1.128.55:27379
java.util.concurrent.CompletionException: org.redisson.client.RedisTimeoutException: Command execution timeout for command: (SENTINEL GET-MASTER-ADDR-BY-NAME), params: [sentinel_cs], Redis client: [addr=redis://10.1.128.55:27379]
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
    at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:975)
    at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
    at org.redisson.misc.CompletableFutureWrapper.thenCompose(CompletableFutureWrapper.java:193)
    at org.redisson.connection.SentinelConnectionManager.checkMasterChange(SentinelConnectionManager.java:500)
    at org.redisson.connection.SentinelConnectionManager.updateState(SentinelConnectionManager.java:358)
    at org.redisson.connection.SentinelConnectionManager.lambda$checkState$4(SentinelConnectionManager.java:351)
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
    at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
    at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
    at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:110)
    at org.redisson.connection.SentinelConnectionManager.checkState(SentinelConnectionManager.java:343)
    at org.redisson.connection.SentinelConnectionManager.access$400(SentinelConnectionManager.java:48)
    at org.redisson.connection.SentinelConnectionManager$2.run(SentinelConnectionManager.java:321)
    at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:125)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

my pom.xml spring-boot-starter-parent 2.0.3.RELEASE redisson 3.17.6 spring-data-redis 2.0.8

public RedissonClient redissonSentinel() {
    Config config = new Config();
    String[] nodes = redissonSentinelConfig.getNodes().split(",");
    List<String> nodeList = new ArrayList(nodes.length);
    Arrays.stream(nodes).forEach(
            (index) -> nodeList.add(index.startsWith("redis://") ? index : "redis://" + index));
    SentinelServersConfig serverConfig = config.useSentinelServers()
            .addSentinelAddress(nodeList.toArray(new String[0]))
            .setMasterName(redissonSentinelConfig.getMaster())
            .setDatabase(redissonSentinelConfig.getDatabase())
            .setTimeout(6000)
            .setPingConnectionInterval(10000)
            .setSubscriptionsPerConnection(5000)
            .setSubscriptionConnectionPoolSize(1000);
    if (StringUtils.isNotBlank(redissonSentinelConfig.getPassword())) {
        serverConfig.setPassword(redissonSentinelConfig.getPassword());
    }
    config.setNettyThreads(64);
    return Redisson.create(config);
}
mrniko commented 1 year ago

Hello,

Please refer to this section https://github.com/redisson/redisson/wiki/16.-FAQ#q-what-is-the-cause-of-redistimeoutexception