线上问题记录 - Githubissues

线程池打满

活动业务量级升高，导致A->B后，B的游服务的load过高，A大量重试，而超时时间过长，导致线程池打满。所以调用下游，一方面要考虑业务对下游bean的限流，一方面，不能设置太长的超时时间，以免把线程池打满。可监控上，也要将线上服务器的业务线程池做相关的监控。

核心线程80，最大线程400
(队列，当前线程数，空闲线程数，最大线程数)
2019-05-20 14:56:48,558 - (0,0,80,80,)
2019-05-20 14:57:18,558 - (0,4,76,80,)
2019-05-20 14:57:48,558 - (0,1,79,80,)
2019-05-20 14:58:18,558 - (0,2,78,80,)
2019-05-20 14:58:48,559 - (0,2,78,80,)
2019-05-20 14:59:18,559 - (0,4,76,80,)
2019-05-20 14:59:48,559 - (0,2,78,80,)
2019-05-20 15:00:18,559 - (0,13,67,80,)
2019-05-20 15:00:48,559 - (671,80,0,80,)
2019-05-20 15:01:18,559 - (1000,400,0,400,)
2019-05-20 15:01:48,560 - (1000,400,0,400,)
2019-05-20 15:02:18,560 - (768,400,0,400,)
2019-05-20 15:02:48,560 - (690,400,0,400,)
2019-05-20 15:03:18,560 - (793,400,0,400,)
2019-05-20 15:03:48,561 - (813,400,0,400,)
2019-05-20 15:04:18,666 - (723,332,68,400,)
2019-05-20 15:04:48,666 - (826,400,0,400,)
2019-05-20 15:05:18,666 - (854,400,0,400,)
2019-05-20 15:05:48,667 - (808,400,0,400,)
2019-05-20 15:06:18,667 - (779,400,0,400,)
2019-05-20 15:06:48,667 - (752,400,0,400,)
2019-05-20 15:07:18,667 - (808,400,0,400,)
2019-05-20 15:07:48,667 - (772,400,0,400,)
2019-05-20 15:08:18,667 - (708,400,0,400,)
2019-05-20 15:08:48,668 - (0,5,395,400,)
2019-05-20 15:09:18,668 - (0,2,398,400,)

Time           ---cpu-- ---mem-- ---tcp-- -----traffic---- --sda--- nvme0n1- nvme0n1p  ---load- 
Time             util     util   retran    bytin  bytout     util     util     util     load1   
20/05/19-14:55   5.10    42.68     0.25   507.3K  541.5K     0.41     3.11     3.12      0.73   
20/05/19-14:56   5.31    42.69     0.23   529.8K  547.5K     0.39     3.26     3.26      0.78   
20/05/19-14:57   5.52    42.58     0.23   581.5K  592.5K     0.44     3.27     3.27      1.09   
20/05/19-14:58   6.10    42.61     0.23   677.5K  672.5K     0.37     3.11     3.12      0.75   
20/05/19-14:59   6.81    42.65     0.22   813.6K  765.5K     0.34     3.54     3.54      0.83   
20/05/19-15:00   8.93    42.68     0.19     1.2M    1.0M     0.34     3.55     3.56      0.68   
20/05/19-15:01  13.87    43.33     0.22     2.3M    1.6M     0.34     3.90     3.90      0.73   
20/05/19-15:02  17.76    42.95     0.30     4.5M    1.9M     0.39     4.20     4.20      2.04   
20/05/19-15:03  16.78    43.06     0.32     2.8M    1.9M     0.38     3.95     3.95      2.35   
20/05/19-15:04  17.17    43.26     0.32     2.9M    1.9M     0.38     4.45     4.45      1.82   
20/05/19-15:05  16.66    43.12     0.31     3.0M    1.9M     0.36     4.10     4.11      1.41   
20/05/19-15:06  17.63    43.17     0.30     3.1M    2.1M     0.33     4.21     4.21      1.58   
20/05/19-15:07  17.22    43.07     0.32     2.9M    2.1M     0.33     3.87     3.86      1.56   
20/05/19-15:08  18.64    44.03     0.28     2.9M    2.1M     0.35     4.87     4.88      1.37   
20/05/19-15:09  12.89    43.31     0.24     1.8M    1.7M     0.35     3.58     3.59      1.09   
20/05/19-15:10   6.33    43.34     0.22   704.5K  689.9K     0.34     3.03     3.04      1.00   
20/05/19-15:11   5.85    43.44     0.22   578.1K  647.6K     0.34     3.63     3.63      0.58

wittyResry / myIssue

线上问题记录 #102

线程池打满