Spring Boot Redis - always getting could not get resource from pool

[dreamstar-enterprises]

Describe the bug This is the bug I keep getting, when I load my Spring Application. I don't know why it is, but I've been stuck on it for 2 months.

2024-08-23T00:17:20.005+01:00  INFO 9758 --- [BFFApplication] [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 9090 (http)
2024-08-23T00:17:20.042+01:00  INFO 9758 --- [BFFApplication] [           main] com.example.bff.BFFApplicationKt         : Started BFFApplicationKt in 7.754 seconds (process running for 8.275)
2024-08-23T00:17:20.071+01:00  INFO 9758 --- [BFFApplication] [ioEventLoop-5-1] c.example.bff.auth.redis.SessionEvicter  : No sessions found to remove.
2024-08-23T00:17:20.071+01:00  INFO 9758 --- [BFFApplication] [ioEventLoop-5-1] c.example.bff.auth.redis.SessionEvicter  : Cleanup operation completed.
2024-08-23T00:17:25.823+01:00  WARN 9758 --- [BFFApplication] [oundedElastic-1] o.s.b.a.d.r.RedisReactiveHealthIndicator : Redis health check failed

org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1847) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1778) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1580) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.lambda$getConnection$0(LettuceConnectionFactory.java:1560) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.doInLock(LettuceConnectionFactory.java:1521) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1557) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedReactiveConnection(LettuceConnectionFactory.java:1268) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:1143) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:119) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:67) ~[reactor-core-3.6.8.jar:3.6.8]
    at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:228) ~[reactor-core-3.6.8.jar:3.6.8]
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.6.8.jar:3.6.8]
    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.6.8.jar:3.6.8]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:1623) ~[na:na]
Caused by: org.springframework.data.redis.connection.PoolException: Could not get a resource from the pool
    at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:104) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1776) ~[spring-data-redis-3.3.2.jar:3.3.2]
    ... 16 common frames omitted
Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to abc.redis.cache.windows.net/<unresolved>:6380
    at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
    at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
    at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:350) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
    at io.lettuce.core.RedisClient.connect(RedisClient.java:215) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
    at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.lambda$getConnection$1(StandaloneConnectionProvider.java:112) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at java.base/java.util.Optional.orElseGet(Optional.java:364) ~[na:na]
    at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.getConnection(StandaloneConnectionProvider.java:112) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.lambda$getConnection$0(LettucePoolingConnectionProvider.java:93) ~[spring-data-redis-3.3.2.jar:3.3.2]
    at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:211) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
    at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:201) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
    at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:70) ~[commons-pool2-2.11.1.jar:2.11.1]
    at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571) ~[commons-pool2-2.11.1.jar:2.11.1]
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:298) ~[commons-pool2-2.11.1.jar:2.11.1]
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223) ~[commons-pool2-2.11.1.jar:2.11.1]
    at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:122) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
    at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:117) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
    at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:99) ~[spring-data-redis-3.3.2.jar:3.3.2]
    ... 17 common frames omitted
Caused by: java.net.SocketException: Connection reset
    at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:401) ~[na:na]
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:434) ~[na:na]
    at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:255) ~[netty-buffer-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:356) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
    ... 1 common frames omitted

2024-08-23T00:19:20.054+01:00  INFO 9758 --- [BFFApplication] [ioEventLoop-5-1] c.example.bff.auth.redis.SessionEvicter  : No sessions found to remove.
2024-08-23T00:19:20.062+01:00  INFO 9758 --- [BFFApplication] [ioEventLoop-5-1] c.example.bff.auth.redis.SessionEvicter  : Cleanup operation completed.

Lettuce Connection Factory

I don't really understand what I'm doing wrong here:

@Configuration
internal class RedisConnectionFactoryConfig(
    private val springDataProperties: SpringDataProperties
) {

    // customize thread pool size
    private val clientResources = DefaultClientResources.builder()
        .ioThreadPoolSize(4)
        .computationThreadPoolSize(4)
        .build()

    @Bean
    @Primary
    fun reactiveRedisConnectionFactory(): ReactiveRedisConnectionFactory {

        // configure Redis standalone configuration
        val config = RedisStandaloneConfiguration()
        config.hostName = springDataProperties.redis.host
        config.port = springDataProperties.redis.port
        config.setPassword(RedisPassword.of(springDataProperties.redis.password))

        // create socket options
        val socketOptions = SocketOptions.builder()
            .keepAlive(SocketOptions.DEFAULT_SO_KEEPALIVE)
            .tcpNoDelay(SocketOptions.DEFAULT_SO_NO_DELAY)
            // time to wait for connection to be established, before considering it as a failed connection
            .connectTimeout(Duration.ofSeconds(60))
            .build()

        // create client options

        // Create SSL options if SSL is required
        val sslOptions = SslOptions.builder()
            .jdkSslProvider()  // Or use OpenSslProvider if you prefer
            .build()

        // Create timeout options
        val timeoutOptions = TimeoutOptions.builder()
            .fixedTimeout(Duration.ofSeconds(10))
            .timeoutCommands(true)
            .build()

        val clientOptions = ClientOptions.builder()
            .autoReconnect(true)
            .timeoutOptions(timeoutOptions)
            .sslOptions(sslOptions)
            .suspendReconnectOnProtocolFailure(true)
            .decodeBufferPolicy(DecodeBufferPolicies.ratio(0.5F))
            .requestQueueSize(1000)
            .disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
            .pingBeforeActivateConnection(true)
            .socketOptions(socketOptions)
            .build()

        // create Lettuce client configuration with authentication details
        val clientConfig = LettucePoolingClientConfiguration.builder()
            // maximum time allowed for a Redis command to execute before the operation is considered timed out.
            .commandTimeout(Duration.ofSeconds(60))
            .clientResources(clientResources)
            .clientOptions(clientOptions)
            .poolConfig(buildLettucePoolConfig())
            .useSsl()
            .build()

        // create Lettuce connection factory
        return LettuceConnectionFactory(config, clientConfig).apply {
            afterPropertiesSet()
            validateConnection = false
            setShareNativeConnection(true)
        }
    }

    // configure connection pool settings
    protected fun buildLettucePoolConfig(): GenericObjectPoolConfig<Any> {
        val poolConfig = GenericObjectPoolConfig<Any>()
        poolConfig.maxTotal = 100
        poolConfig.maxIdle = 50
        poolConfig.minIdle = 10
        poolConfig.setMaxWait(Duration.ofSeconds(120))
        poolConfig.timeBetweenEvictionRuns = Duration.ofSeconds(120)
        poolConfig.minEvictableIdleTime = Duration.ofMinutes(5)
        poolConfig.testOnBorrow = true
        poolConfig.testWhileIdle = true
        poolConfig.testOnReturn = true
        poolConfig.blockWhenExhausted = true
        poolConfig.lifo = true
        return poolConfig
    }

    @PreDestroy
    fun cleanup() {
        clientResources.shutdown()
    }

}

This always happens when I load Spring Boot. Can someone help?

To Reproduce

Please follow my github repo here. The specific class file that I think is causing the issue can be found here Steps to reproduce the behavior.

Expected behavior No error should be thrown, just like when I connect using the command line.

Sample

See above. The full bff repo folder can be found here: https://github.com/dreamstar-enterprises/docs/tree/master/Spring%20BFF/bff

[philwebb]

This doesn't look like a Spring Boot issue in itself. It looks like either an issue with your Redis server or the client. I'm afraid we lack the expertise to help you here.

[dreamstar-enterprises]

Hi Phil. I'll look into it. I'm using Azure Redis Cache, to it might be because of that. Is that what you meant by client too, or are Redis Server and Client two separate things?

[philwebb]

I mean the Lettuce library you are using to connect to the Redis server. Perhaps your timeouts are too tight. Just a guess, you'll probably be better off seeking out Redis experts for help.

[dreamstar-enterprises]

I ran this in Docker compose and do not get the same error: