search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.05k stars 4.82k forks source link

Upgrading envoy causes JedisException: Could not get a resource from the pool #36695

Open Vasanthan-R opened 1 month ago

Vasanthan-R commented 1 month ago

Hi,

We have a envoy configured for a Redis cluster. The current version of it 1.18.3, and we have observed no issues. Recently, we upgraded it to 1.27.2, and we noticed errors in the clients trying to access the cache. The error is:

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Caused by: java.util.NoSuchElementException: Unable to validate object
 at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:502) ~[commons-pool2-2.2.jar:2.2]
 at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.2.jar:2.2]
 at redis.clients.util.Pool.getResource(Pool.java:48) ~[ContXJedis-2.8.0.jar:?]

However, I tried adding the below config on top of the existing envoy configuration and the issue got sorted out:

upstream_connection_options:
        tcp_keepalive:
          keepalive_interval: 5
          keepalive_probes: 1
          keepalive_time: 30

Can anyone explain the reason for this behaviour?

tyxia commented 1 month ago

@Vasanthan-R from your description , it seems to point to regression. Would you either (1) do some regression check to see the difference in Envoy between 1.18.3 and 1.27.2. with the clue you found (2) Update to latest version to see if this still occurs.

I can later add the code/domain owner if you don't have luck with either of them above.

Vasanthan-R commented 2 weeks ago

Hey @tyxia, it looks like even after adding the tcp_keepalive configuration, I am still getting into the error. Can you add the code/domain owner?