search

dubboclub / dubbokeeper

dubbo服务管理以及监控系统
Apache License 2.0
1.28k stars 688 forks source link

Thread pool is EXHAUSTED #110

Open kitikousei opened 7 years ago

kitikousei commented 7 years ago

dubbo-framework/DUBBO.log 出现如下日志：

2017-05-07 00:37:44.458 [NettyServerWorker-thread-9] WARN c.a.d.c.t.s.AbortPolicyWithReport - [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-192.168.17.13:20980, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 894668 (completed: 894468), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://192.168.17.13:20980!, dubbo version: 2.5.3, current host: 127.0.0.1

dubbo-mysql.properties配置文件里，我已经把线程数增加到400了。register信息里面看，线程数也生效了。但是系统运行一段时间之后，线程又用完了。 dubbo.protocol.threads=400

Monitor界面点开也显示：后端系统出现异常，请稍后再试！

ps：重启dubbokeeper，Monitor页面又能打开。用一段时间，线程就消耗完了。Monitor又打不开了。

qct commented 7 years ago

请问你服务调用量有多大？ 如果调用量很大， 把 ThreadPool 改成cached， 缓存线程池，空闲一分钟自动删除，需要时重建。

<dubbo:protocol name="dubbo" dispatcher="all" threadpool="cached" threads="100" />
kitikousei commented 7 years ago

按照你的提示，已经把线程池改为cached，并且配置的300个。但是生产实际观察日志发现 2017-07-03 05:29:38.171 [DubboServerHandler-192.168.17.13:20980-thread-293] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:29:40.126 [DubboServerHandler-192.168.17.13:20980-thread-294] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:29:41.942 [DubboServerHandler-192.168.17.13:20980-thread-295] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:29:43.752 [DubboServerHandler-192.168.17.13:20980-thread-296] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:29:45.620 [DubboServerHandler-192.168.17.13:20980-thread-297] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:29:47.587 [DubboServerHandler-192.168.17.13:20980-thread-298] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:29:49.918 [DubboServerHandler-192.168.17.13:20980-thread-299] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:29:55.084 [DubboServerHandler-192.168.17.13:20980-thread-300] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:29:58.250 [DubboServerHandler-192.168.17.13:20980-thread-300] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:30:02.255 [DubboServerHandler-192.168.17.13:20980-thread-300] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:30:06.198 [DubboServerHandler-192.168.17.13:20980-thread-300] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1 2017-07-03 05:30:11.468 [DubboServerHandler-192.168.17.13:20980-thread-300] DEBUG c.a.d.r.transport.DecodeHandler - [DUBBO] Decode decodeable message com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation, dubbo version: 2.5.3, current host: 127.0.0.1

线程名DubboServerHandler-192.168.17.13:20980-thread-XXX这边XXX从启动的0，到配置的300，很快就用光。最后就只剩下DubboServerHandler-192.168.17.13:20980-thread-300这个线程在工作。 最终会报 [NettyServerWorker-thread-14] WARN c.a.d.c.t.s.AbortPolicyWithReport - [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-192.168.17.13:20980, Pool Size: 300 (active: 300, core: 300, max: 300, largest: 300), Task: 981475 (completed: 981175), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://192.168.17.13:20980!, dubbo version: 2.5.3, current host: 127.0.0.1 如果不重启dubbokeeper，安装dubbokeeper的服务器cpu使用率一直都是100%

njkfei commented 7 years ago

我也发生了这个问题。 st: 200), Task: 2426828 (completed: 2426628), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1 2017-08-30 11:34:57.602 [NettyServerWorker-thread-22] WARN c.a.d.c.t.s.AbortPolicyWithReport - [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 2426828 (completed: 2426628), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1 2017-08-30 11:34:57.603 [NettyServerWorker-thread-22] WARN c.a.d.c.t.s.AbortPolicyWithReport - [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 2426828 (completed: 2426628), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1 2017-08-30 11:34:57.822 [NettyServerWorker-thread-9] WARN c.a.d.c.t.s.AbortPolicyWithReport - [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 2426828 (completed: 2426628), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1 2017-08-30 11:34:57.822 [NettyServerWorker-thread-9] WARN c.a.d.c.t.s.AbortPolicyWithReport - [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 2426828 (completed: 2426628), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0

hesy007 commented 6 years ago

线程池中的线程耗光了，要查下原因：