Jedis dropping connections?

[dsyer]

I get a lot of these after we upgraded to Spring Data Gosling M1:

2015-06-02 16:01:00.803  WARN 10024 --- [ask-scheduler-9] o.s.b.a.m.export.AbstractMetricExporter  : Could not write to MetricWriter: class org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
...

They seem to affect some but not all writes to redis.

[dsyer]

More stack trace:

2015-06-02 16:01:21.810 ERROR 10069 --- [hannel-adapter1] .s.i.r.i.RedisQueueMessageDrivenEndpoint : Failed to execute listening task. Will attempt to resubmit in 5000 milliseconds.

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:162)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:251)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:58)
    at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:178)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153)
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86)
    at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:172)
    at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:92)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:188)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:54)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:323)
    at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
    at java.lang.Thread.run(Thread.java:744)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
    at redis.clients.util.Pool.getResource(Pool.java:50)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:82)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:12)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:155)
    ... 15 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused
    at redis.clients.jedis.Connection.connect(Connection.java:148)
    at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:79)
    at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1613)
    at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:85)
    at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
    at redis.clients.util.Pool.getResource(Pool.java:48)
    ... 18 common frames omitted
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at redis.clients.jedis.Connection.connect(Connection.java:142)
    ... 25 common frames omitted

[fawzyj]

This affects spring session too when used. Hope this will fix soon as i use spring session with redis.

[wilkinsona]

@fawzyj What versions are you using?

[dsyer]

I am still seeing this in PWS (but not with other redis instances). I wonder if it's a problem with Redislabs?

[zampettim]

I'm seeing these quite often in my dev environments or local, and sometimes in production with no apparent issue with the Redis server. Anyone have any ideas?

[snicoll]

@dsyer is that actually linked to Spring Boot at all?

[dsyer]

Probably not.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[user20161119]

got the same error：

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:162)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:251)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:58)
    at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:178)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153)
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86)
    at org.springframework.data.redis.core.DefaultValueOperations.set(DefaultValueOperations.java:169)
    at com.didispace.ApplicationTests.test(ApplicationTests.java:29)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
    at redis.clients.util.Pool.getResource(Pool.java:50)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:99)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:12)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:155)
    ... 39 more
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused: connect
    at redis.clients.jedis.Connection.connect(Connection.java:164)
    at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:82)
    at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1641)
    at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:85)
    at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
    at redis.clients.util.Pool.getResource(Pool.java:48)
    ... 42 more
Caused by: java.net.ConnectException: Connection refused: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at redis.clients.jedis.Connection.connect(Connection.java:158)
    ... 49 more

redis config is right,and redis server is connected in another no-spring boot project and my demo project is here:https://github.com/dyc87112/SpringBoot-Learning/tree/master/Chapter3-2-5

[dsyer]

In your case it looks like you maybe just didn't set the redis connection factory up properly (you create a bean but don't set the host, port etc.). This issue was closed under the assumption that the original problem is not to do with spring Boot. Your sample hasn't convinced me to change that conclusion yet.

[user20161119]

i set redis config property in application.properties,and those settings are correct,because in my another no spring boot project,i use the redis at local dev environment.

[dsyer]

You set those properties in application.properties but you never used them. If you have questions about usage please post on stack overflow.

[user20161119]

Please see my junit tests.those settings are used in the junit test.

发送自 Windows 10 版邮件应用

发件人: Dave Syer 发送时间: 2017年7月30日 21:48 收件人: spring-projects/spring-boot 抄送: user_github; Comment 主题: Re: [spring-projects/spring-boot] Jedis dropping connections? (#3091)

You set those properties in application.properties but you never used them. If you have questions about usage please post on stack overflow. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[odedia]

@dsyer, are you still seeing these errors? We have a spring boot application using 1.4.5 and spring cloud CAMDEN.SR6.

When redis db is restarting, client application cannot access db anymore and same error as above is noted.

Previously we used open source redis and this issue did not occur. Once we moved to redislabs enterprise we did it see this behavior.

Thanks for the help!

[odedia]

Sorry just to add - redis is only used with spring session redis in our application.

[dsyer]

Did you see my "Probably not" comment? Try asking Redislabs. I don't see it any more, but I don't use Redislabs as much as I did.