XiaoMi / minos

Minos is beyond a hadoop deployment system.
Apache License 2.0
522 stars 200 forks source link

ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 364 #32

Open xiaoguanyu opened 10 years ago

xiaoguanyu commented 10 years ago

两个节点部署了supervisord,其中59正常运行日志打印信息也正常,8节点supervisor显得是状态总是fatal, 如下是通过10.38.11.8:9001访问显示的supervisor status信息: State Description Name fatal Exited too quickly (process log may have details) crashmailbatch-monitor fatal Exited too quickly (process log may have details) processexit-monitor fatal Exited too quickly (process log may have details) zookeeper--dptst--zookeeper 部分日志信息如下: 2014-05-16 13:17:07,290 INFO spawned: 'zookeeper--dptst--zookeeper' with pid 22291 2014-05-16 13:17:07,290 INFO supervisord wrote sub-process pidfile 2014-05-16 13:17:07,299 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 364 2014-05-16 13:17:07,299 INFO exited: zookeeper--dptst--zookeeper (exit status 126; not expected) 2014-05-16 13:17:07,299 INFO supervisord wrote sub-process pidfile 2014-05-16 13:17:08,300 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 365 2014-05-16 13:17:08,304 INFO spawned: 'zookeeper--dptst--zookeeper' with pid 22295 2014-05-16 13:17:08,305 INFO supervisord wrote sub-process pidfile 2014-05-16 13:17:08,314 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 366 2014-05-16 13:17:08,314 INFO exited: zookeeper--dptst--zookeeper (exit status 126; not expected) 2014-05-16 13:17:08,315 INFO supervisord wrote sub-process pidfile 2014-05-16 13:17:10,315 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 367 2014-05-16 13:17:10,319 INFO spawned: 'zookeeper--dptst--zookeeper' with pid 22299 2014-05-16 13:17:10,319 INFO supervisord wrote sub-process pidfile 2014-05-16 13:17:10,319 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 368 2014-05-16 13:17:10,329 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 369 2014-05-16 13:17:10,329 INFO exited: zookeeper--dptst--zookeeper (exit status 126; not expected) 2014-05-16 13:17:10,329 INFO supervisord wrote sub-process pidfile 2014-05-16 13:17:13,330 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 370 2014-05-16 13:17:13,332 INFO spawned: 'zookeeper--dptst--zookeeper' with pid 22303 2014-05-16 13:17:13,332 INFO supervisord wrote sub-process pidfile 2014-05-16 13:17:13,343 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 371 2014-05-16 13:17:13,343 INFO exited: zookeeper--dptst--zookeeper (exit status 126; not expected) 2014-05-16 13:17:13,344 INFO supervisord wrote sub-process pidfile 2014-05-16 13:17:14,344 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 372 2014-05-16 13:17:14,344 INFO gave up: zookeeper--dptst--zookeeper entered FATAL state, too many start retries too quickly 2014-05-16 13:17:15,344 ERRO pool crashmailbatch-monitor event buffer overflowed, discarding event 373

wuzesheng commented 10 years ago

感觉是supervisord布得有问题,连系统自带的crashmailbatch-monitor和processexit-monitor都没起来,你发一下这两个进程的log,在supervisord目录下

eddy85br commented 5 years ago

Lots of: 2019-01-31 15:34:22,636 ERRO pool crashmailbatch event buffer overflowed, discarding event 1158 every time a process is killed by SIGTERM and also every time a process starts smoothly.