search

dubboclub / dubbokeeper

dubbo服务管理以及监控系统
Apache License 2.0
1.29k stars 688 forks source link

[DUBBO] Thread pool is EXHAUSTED! Thread Name: #128

Open njkfei opened 7 years ago

njkfei commented 7 years ago

cpu load 非常 大

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                        
27150 staging   20   0 20.6g 223m  11m S 1643  0.7   7766:25 /usr/lib/jvm/java-7-sun/bin/java -Dmonitor.log.home=/data/dubbo/dubbokeeper/bin/../logs -Ddubbo.properties.file=dubbo-mysql.properties

上下文切换异常

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  3   0  96   0   0   0|5126B  137k|   0     0 |   0     2B|3365    15k
 17  62  21   0   0   0|   0     0 |  80k  685k|   0     0 | 476k 3272k
 18  61  21   0   0   0|   0   112k|  80k  661k|   0     0 | 479k 3294k
 18  61  20   1   0   0|   0   636k|  85k  673k|   0     0 | 484k 3301k
 19  61  20   0   0   0|   0   168k|6200k 1587k|   0     0 | 479k 3262k
 19  60  20   0   0   0|   0   344k| 217k 5695k|   0     0 | 488k 3307k
 19  60  21   0   0   0|   0   268k|  92k  675k|   0     0 | 489k 3307k
 17  61  21   0   0   0|   0   392k|  60k  662k|   0     0 | 476k 3276k

线程资源没有了

st: 200), Task: 8898183 (completed: 8897983), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1
2017-08-31 09:40:13.978 [NettyServerWorker-thread-18] WARN  c.a.d.c.t.s.AbortPolicyWithReport -  [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 8898183 (completed: 8897983), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1
2017-08-31 09:40:13.979 [NettyServerWorker-thread-18] WARN  c.a.d.c.t.s.AbortPolicyWithReport -  [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 8898183 (completed: 8897983), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1
2017-08-31 09:40:14.015 [NettyServerWorker-thread-21] WARN  c.a.d.c.t.s.AbortPolicyWithReport -  [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 8898183 (completed: 8897983), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1
2017-08-31 09:40:14.015 [NettyServerWorker-thread-21] WARN  c.a.d.c.t.s.AbortPolicyWithReport -  [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 8898183 (completed: 8897983), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1
2017-08-31 09:40:14.103 [NettyServerWorker-thread-5] WARN  c.a.d.c.t.s.AbortPolicyWithReport -  [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 8898183 (completed: 8897983), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1
2017-08-31 09:40:14.104 [NettyServerWorker-thread-5] WARN  c.a.d.c.t.s.AbortPolicyWithReport -  [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.0.0.114:20884, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 8898183 (completed: 8897983), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.0.0.114:20884!, dubbo version: 2.8.4, current host: 127.0.0.1
xingtaoshi commented 6 years ago

我也遇到相同的问题cpu 过高一会儿就爆仓了

njkfei commented 6 years ago

@xingtaoshi 写mysql速度跟不大，导致线程资源不足，解决办法，降低写的速度，不重要的性能日志不入库即可。