Closed freshgeek closed 3 years ago
redis 4.0.8
加入超时时间后全部异常,这种现象好像再重启后一段时间内不会出现,但是过几个小时就会出现
java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:448)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
at redis.clients.util.Pool.getResource(Pool.java:49)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at org.crazycake.shiro.RedisManager.getJedis(RedisManager.java:38)
at org.crazycake.shiro.common.WorkAloneRedisManager.get(WorkAloneRedisManager.java:51)
at org.crazycake.shiro.RedisSessionDAO.doReadSession(RedisSessionDAO.java:202)
at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:168)
at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236)
at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222)
at org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118)
at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148)
at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getSession(AbstractNativeSessionManager.java:140)
at org.apache.shiro.mgt.SessionsSecurityManager.getSession(SessionsSecurityManager.java:156)
at org.apache.shiro.mgt.DefaultSecurityManager.resolveContextSession(DefaultSecurityManager.java:460)
at org.apache.shiro.mgt.DefaultSecurityManager.resolveSession(DefaultSecurityManager.java:446)
at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:342)
at org.apache.shiro.subject.Subject$Builder.buildSubject(Subject.java:845)
at org.apache.shiro.web.subject.WebSubject$Builder.buildWebSubject(WebSubject.java:148)
at org.apache.shiro.web.servlet.AbstractShiroFilter.createSubject(AbstractShiroFilter.java:292)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:359)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.firestone.config.Filter.CorsFilter.doFilter(CorsFilter.java:40)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:151)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
更新了最新版本也会出现这个问题
看了源码后是不是在使用的时候没有将 纳入Try 块中
如 org.crazycake.shiro.common.WorkAloneRedisManager#set
public byte[] set(byte[] key, byte[] value, int expireTime) {
if (key == null) {
return null;
} else {
Jedis jedis = this.getJedis();
try {
jedis.set(key, value);
if (expireTime > 0) {
jedis.expire(key, expireTime);
}
} finally {
jedis.close();
}
return value;
}
}
是否应该将 Jedis jedis = this.getJedis();
这一行纳入try 块
是否可能出现 return internalPool.borrowObject(); 拿到了jedis 在后续代码中出现异常没有回收导致的连接泄露
出现java.util.NoSuchElementException: Timeout waiting for idle object有多个原因: https://stackoverflow.com/questions/34155413/need-explanation-about-java-util-nosuchelementexception-timeout-waiting-for-id https://knowledge.informatica.com/s/article/525925 https://github.com/redis/jedis/issues/1874
将Jedis jedis = this.getJedis();纳入try是一个好主意,我会加入下一个版本。但是它并不能解决你的问题,因为你的问题在于为什么jedis连接会用完。是否考虑检查连接迟的最大连接数上限?
出现java.util.NoSuchElementException: Timeout waiting for idle object有多个原因: https://stackoverflow.com/questions/34155413/need-explanation-about-java-util-nosuchelementexception-timeout-waiting-for-id https://knowledge.informatica.com/s/article/525925 redis/jedis#1874
将Jedis jedis = this.getJedis();纳入try是一个好主意,我会加入下一个版本。但是它并不能解决你的问题,因为你的问题在于为什么jedis连接会用完。是否考虑检查连接迟的最大连接数上限?
是的 这个问题很奇怪,基本无法复现, 我也考虑了是否连接上限问题 , 同时使用jemeter 多个线程同时请求登录接口也无法复现,一定是使用一段时间后才会触发
我使用的是默认配置连接数idle 8 max 8
后面加大连接数 idle 10 max 50
目前没有这个问题出现
逆向测试 把配置数改为idle 1 max 1
同时使用jmeter 却无法复现这个问题
@Bean
public RedisManager redisManager() {
RedisManager redisManager = new RedisManager();
String host = redisProperties.getHost();
int port = redisProperties.getPort();
redisManager.setHost(String.format("%s:%s", host, port));
redisManager.setTimeout((int) redisProperties.getTimeout().getSeconds());
//连接redis超时
if (StrUtil.isNotEmpty(redisProperties.getPassword())) {
redisManager.setPassword(redisProperties.getPassword());
}
redisManager.setDatabase(redisProperties.getDatabase());
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxWaitMillis(redisProperties.getTimeout().toMillis());
redisManager.setJedisPoolConfig(config);
return redisManager;
}
在反向测试时idle 1 max 1
使用jmeter 测试正常 , 但是实际使用发生问题时间果然缩短了 ,同时日志打印如下
2020-10-28 15:26:42.855 --- [io-9011-exec-72] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Filtered request failed.] with root cause
java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:448)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
at redis.clients.util.Pool.getResource(Pool.java:49)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at org.crazycake.shiro.RedisManager.getJedis(RedisManager.java:38)
at org.crazycake.shiro.common.WorkAloneRedisManager.get(WorkAloneRedisManager.java:51)
at org.crazycake.shiro.RedisSessionDAO.doReadSession(RedisSessionDAO.java:202)
at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:168)
at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236)
at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222)
at org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118)
at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148)
at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getSession(AbstractNativeSessionManager.java:140)
at org.apache.shiro.mgt.SessionsSecurityManager.getSession(SessionsSecurityManager.java:156)
at org.apache.shiro.mgt.DefaultSecurityManager.resolveContextSession(DefaultSecurityManager.java:460)
at org.apache.shiro.mgt.DefaultSecurityManager.resolveSession(DefaultSecurityManager.java:446)
at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:342)
at org.apache.shiro.subject.Subject$Builder.buildSubject(Subject.java:845)
at org.apache.shiro.web.subject.WebSubject$Builder.buildWebSubject(WebSubject.java:148)
at org.apache.shiro.web.servlet.AbstractShiroFilter.createSubject(AbstractShiroFilter.java:292)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:359)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.firestone.config.Filter.CorsFilter.doFilter(CorsFilter.java:40)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:151)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
2020-10-28 15:26:50.000 [pool-4-thread-1] INFO com.xxxxx FireApplication.java : 48] - getNumWaiters : 0,getNumIdle: 0,getNumActive:1,getMeanBorrowWaitTimeMillis :0,getMaxBorrowWaitTimeMillis:35
可以从监控日志看到当前是有一个 getNumActive 未释放的为 1 , jemeter 多线程测试中getNumActive是0
, 同时从后续的日志中可以观察到此时并没有请求过来,可以确定是有泄露的问题
而加大连接数 只不过是把这个问题出现的时间往后推迟了
很多人也出现过jedis 连接池泄露问题 , 到这里也大概概率是jedis 组件的问题
这确实是个严重的问题
我也遇到同样的问题。
Jedis版本:
`
我也遇到同样的问题。 Jedis版本:
<java.version>1.8</java.version> <jedis.version>2.9.0</jedis.version>
有如下报错:redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool at redis.clients.util.Pool.getResource(Pool.java:51) at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) at com.navi.rtm.rule.AbstractRule.subMainProc(AbstractRule.java:77) at com.navi.rtm.rule.RuleCheckContext.call(RuleCheckContext.java:142) at com.navi.rtm.rule.RuleCheckContext.call(RuleCheckContext.java:16) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.NoSuchElementException: Timeout waiting for idle object at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) at redis.clients.util.Pool.getResource(Pool.java:49) ... 8 more
升级jedis 可以解决
非常感谢反馈
发自我的iPhone
------------------ 原始邮件 ------------------ 发件人: chen.chao @.> 发送时间: 2022年8月8日 22:43 收件人: alexxiyang/shiro-redis @.> 抄送: alex @.>, Comment @.> 主题: Re: [alexxiyang/shiro-redis] [bug] 在3.2.3 中出现死锁,造成应用假死 (#137)
我也遇到同样的问题。 Jedis版本: <java.version>1.8</java.version> <jedis.version>2.9.0</jedis.version> 有如下报错: redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool at redis.clients.util.Pool.getResource(Pool.java:51) at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) at com.navi.rtm.rule.AbstractRule.subMainProc(AbstractRule.java:77) at com.navi.rtm.rule.RuleCheckContext.call(RuleCheckContext.java:142) at com.navi.rtm.rule.RuleCheckContext.call(RuleCheckContext.java:16) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.NoSuchElementException: Timeout waiting for idle object at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) at redis.clients.util.Pool.getResource(Pool.java:49) ... 8 more
升级jedis 可以解决
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
使用
出现应用假死现象所以前端请求都是pending , 内存cpu 表象正常 ,
在使用jstack pid 之后发现大量