search

Unitech / pm2

Node.js Production Process Manager with a built-in Load Balancer.
https://pm2.keymetrics.io/docs/usage/quick-start/
Other
41.56k stars 2.63k forks source link

pm2 crashes with too many unstable restarts on scripts.sh #4545

Open rsuchit opened 4 years ago

rsuchit commented 4 years ago

i am writing my scripts in python and run this script using pm2 like we run normally the node.js scripts but some after the error display like this.

I do:

pm2 start script.sh

The app doesn't start and I get this repeated in the logs:

root@ubuntu-s-photomaths:/var/www/html/sym_math# pm2 logs

[TAILING] Tailing last 15 lines for [all] processes (change the value with --lines option) /root/.pm2/pm2.log last 15 lines: PM2 | 2019-12-11T04:29:23: PM2 log: App [script:0] online PM2 | 2019-12-11T04:29:23: PM2 log: App [script:0] exited with code [0] via signal [SIGINT] PM2 | 2019-12-11T04:29:23: PM2 log: App [script:0] starting in -fork mode- PM2 | 2019-12-11T04:29:23: PM2 log: App [script:0] online PM2 | 2019-12-11T04:29:24: PM2 log: App [script:0] exited with code [0] via signal [SIGINT] PM2 | 2019-12-11T04:29:24: PM2 log: App [script:0] starting in -fork mode- PM2 | 2019-12-11T04:29:24: PM2 log: App [script:0] online PM2 | 2019-12-11T04:29:25: PM2 log: App [script:0] exited with code [0] via signal [SIGINT] PM2 | 2019-12-11T04:29:25: PM2 log: App [script:0] starting in -fork mode- PM2 | 2019-12-11T04:29:25: PM2 log: App [script:0] online PM2 | 2019-12-11T04:29:26: PM2 log: App [script:0] exited with code [0] via signal [SIGINT] PM2 | 2019-12-11T04:29:26: PM2 log: App [script:0] starting in -fork mode- PM2 | 2019-12-11T04:29:26: PM2 log: App [script:0] online PM2 | 2019-12-11T04:29:26: PM2 log: App [script:0] exited with code [0] via signal [SIGINT] PM2 | 2019-12-11T04:29:26: PM2 log: Script /var/www/html/sym_math/script.sh had too many unstable restarts (16). Stopped. "errored" ****]

my error logs can be display like this

/root/.pm2/logs/script-error.log last 15 lines: 0|script | 0|script | Usage: 0|script | kill [options] [...] 0|script | 0|script | Options: 0|script | [...] send signal to every listed 0|script | -, -s, --signal 0|script | specify the to be sent 0|script | -l, --list=[] list all signal names, or convert one to a name 0|script | -L, --table list all signal names in a nice table 0|script | 0|script | -h, --help display this help and exit 0|script | -V, --version output version information and exit 0|script | 0|script | For more details see kill(1).

can anyone please help me about this

rsuchit commented 4 years ago

@Unitech @jshkurti @vmarchaud sir, any idea about this error?

rsuchit commented 4 years ago

@chriswiggins @58bits sir do you have any idea about this?

mifenfeifen commented 4 years ago

@rsuchit I had met the same question, same with python script ( haven't tested on other script like js ). I can use pm2 update to stop the mentioned restart loop. But every time I reboot the system, the loop will begin.

log here:

===============================================================================
2019-12-19T22:25:51: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2019-12-19T22:25:51: PM2 log: Time                 : Thu Dec 19 2019 22:25:51 GMT+0800 (GMT+08:00)
2019-12-19T22:25:51: PM2 log: PM2 version          : 4.2.1
2019-12-19T22:25:51: PM2 log: Node.js version      : 10.16.3
2019-12-19T22:25:51: PM2 log: Current arch         : x64
2019-12-19T22:25:51: PM2 log: PM2 home             : /root/.pm2
2019-12-19T22:25:51: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2019-12-19T22:25:51: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2019-12-19T22:25:51: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2019-12-19T22:25:51: PM2 log: Application log path : /root/.pm2/logs
2019-12-19T22:25:51: PM2 log: Worker Interval      : 30000
2019-12-19T22:25:51: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2019-12-19T22:25:51: PM2 log: Concurrent actions   : 2
2019-12-19T22:25:51: PM2 log: SIGTERM timeout      : 1600
2019-12-19T22:25:51: PM2 log: ===============================================================================
2019-12-19T22:25:51: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2019-12-19T22:25:51: PM2 log: App [pm2-logrotate:0] online
2019-12-19T22:25:51: PM2 log: App [stream_10001:1] starting in -fork mode-
2019-12-19T22:25:51: PM2 log: App [detect_10010:2] starting in -fork mode-
2019-12-19T22:25:51: PM2 log: App [stream_10001:1] online
2019-12-19T22:25:51: PM2 log: App [detect_10010:2] online
2019-12-19T22:25:51: PM2 log: App [reg_10020:3] starting in -fork mode-
2019-12-19T22:25:51: PM2 log: App [reg_10020:3] online
2019-12-19T22:25:51: PM2 log: App [notify_10030:4] starting in -fork mode-
2019-12-19T22:25:51: PM2 log: App [notify_10030:4] online
2019-12-19T22:27:21: PM2 log: pm2 has been killed by signal, dumping process list before exit...
2019-12-19T22:27:21: PM2 log: Deleting process 0
2019-12-19T22:27:21: PM2 log: Stopping app:pm2-logrotate id:0
2019-12-19T22:27:21: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGTERM]
2019-12-19T22:27:21: PM2 log: App [notify_10030:4] exited with code [0] via signal [SIGTERM]
2019-12-19T22:27:21: PM2 log: App [stream_10001:1] exited with code [0] via signal [SIGTERM]
2019-12-19T22:27:21: PM2 log: App [notify_10030:4] starting in -fork mode-
2019-12-19T22:27:21: PM2 log: App [notify_10030:4] online
2019-12-19T22:27:21: PM2 log: App [stream_10001:1] starting in -fork mode-
2019-12-19T22:27:21: PM2 log: App [stream_10001:1] online
2019-12-19T22:27:21: PM2 log: pid=22712 msg=process killed
2019-12-19T22:27:21: PM2 log: Deleting process 1
2019-12-19T22:27:21: PM2 log: Stopping app:stream_10001 id:1
2019-12-19T22:27:21: PM2 log: App [stream_10001:1] exited with code [1] via signal [SIGINT]
2019-12-19T22:27:21: PM2 log: App [detect_10010:2] exited with code [0] via signal [SIGTERM]
2019-12-19T22:27:21: PM2 log: App [detect_10010:2] starting in -fork mode-
2019-12-19T22:27:21: PM2 log: App [detect_10010:2] online
2019-12-19T22:27:21: PM2 log: pid=22998 msg=process killed
2019-12-19T22:27:21: PM2 log: Deleting process 2
2019-12-19T22:27:21: PM2 log: Stopping app:detect_10010 id:2
2019-12-19T22:27:21: PM2 log: App [detect_10010:2] exited with code [1] via signal [SIGINT]
2019-12-19T22:27:21: PM2 log: pid=23015 msg=process killed
2019-12-19T22:27:21: PM2 log: Deleting process 3
2019-12-19T22:27:21: PM2 log: Stopping app:reg_10020 id:3
2019-12-19T22:27:21: PM2 log: App [reg_10020:3] exited with code [0] via signal [SIGTERM]
2019-12-19T22:27:21: PM2 log: pid=22723 msg=process killed
2019-12-19T22:27:21: PM2 log: Deleting process 4
2019-12-19T22:27:21: PM2 log: Stopping app:notify_10030 id:4
2019-12-19T22:27:21: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:22: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:23: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:23: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:23: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:23: PM2 log: pid=22997 msg=failed to kill - retrying in 100ms
2019-12-19T22:27:23: PM2 log: Process with pid 22997 still alive after 1600ms, sending it SIGKILL now...
2019-12-19T22:27:23: PM2 log: App [notify_10030:4] exited with code [0] via signal [SIGKILL]
2019-12-19T22:27:23: PM2 log: pid=22997 msg=process killed
2019-12-19T22:27:23: PM2 log: Exited peacefully
2019-12-19T22:27:24: PM2 log: ===============================================================================
2019-12-19T22:27:24: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2019-12-19T22:27:24: PM2 log: Time                 : Thu Dec 19 2019 22:27:24 GMT+0800 (GMT+08:00)
2019-12-19T22:27:24: PM2 log: PM2 version          : 4.2.1
2019-12-19T22:27:24: PM2 log: Node.js version      : 10.16.3
2019-12-19T22:27:24: PM2 log: Current arch         : x64
2019-12-19T22:27:24: PM2 log: PM2 home             : /root/.pm2
2019-12-19T22:27:24: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2019-12-19T22:27:24: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2019-12-19T22:27:24: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2019-12-19T22:27:24: PM2 log: Application log path : /root/.pm2/logs
2019-12-19T22:27:24: PM2 log: Worker Interval      : 30000
2019-12-19T22:27:24: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2019-12-19T22:27:24: PM2 log: Concurrent actions   : 2
2019-12-19T22:27:24: PM2 log: SIGTERM timeout      : 1600
2019-12-19T22:27:24: PM2 log: ===============================================================================
2019-12-19T22:27:24: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2019-12-19T22:27:24: PM2 log: App [pm2-logrotate:0] online
2019-12-19T22:27:24: PM2 log: App [stream_10001:1] starting in -fork mode-
2019-12-19T22:27:24: PM2 log: App [detect_10010:2] starting in -fork mode-
2019-12-19T22:27:24: PM2 log: App [stream_10001:1] online
2019-12-19T22:27:24: PM2 log: App [detect_10010:2] online
2019-12-19T22:27:24: PM2 log: App [reg_10020:3] starting in -fork mode-
2019-12-19T22:27:24: PM2 log: App [reg_10020:3] online
2019-12-19T22:27:24: PM2 log: App [notify_10030:4] starting in -fork mode-
2019-12-19T22:27:24: PM2 log: App [notify_10030:4] online
2019-12-19T22:28:27: PM2 log: pm2 has been killed by signal, dumping process list before exit...
2019-12-19T22:28:27: PM2 log: Deleting process 0
2019-12-19T22:28:27: PM2 log: Stopping app:pm2-logrotate id:0
2019-12-19T22:28:27: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGTERM]
2019-12-19T22:28:27: PM2 log: App [stream_10001:1] exited with code [0] via signal [SIGTERM]
2019-12-19T22:28:27: PM2 log: App [notify_10030:4] exited with code [0] via signal [SIGTERM]
2019-12-19T22:28:27: PM2 log: App [stream_10001:1] starting in -fork mode-
2019-12-19T22:28:27: PM2 log: App [notify_10030:4] starting in -fork mode-
2019-12-19T22:28:27: PM2 log: App [stream_10001:1] online
2019-12-19T22:28:28: PM2 log: App [notify_10030:4] online
2019-12-19T22:28:28: PM2 log: pid=23074 msg=process killed
2019-12-19T22:28:28: PM2 log: Deleting process 1
2019-12-19T22:28:28: PM2 log: Stopping app:stream_10001 id:1
2019-12-19T22:28:28: PM2 log: App [detect_10010:2] exited with code [0] via signal [SIGTERM]
2019-12-19T22:28:28: PM2 log: App [detect_10010:2] starting in -fork mode-
2019-12-19T22:28:28: PM2 log: App [detect_10010:2] online
2019-12-19T22:28:28: PM2 log: App [stream_10001:1] exited with code [1] via signal [SIGINT]
2019-12-19T22:28:28: PM2 log: pid=23393 msg=process killed
2019-12-19T22:28:28: PM2 log: Deleting process 2
2019-12-19T22:28:28: PM2 log: Stopping app:detect_10010 id:2
2019-12-19T22:28:28: PM2 log: App [detect_10010:2] exited with code [1] via signal [SIGINT]
2019-12-19T22:28:28: PM2 log: pid=23454 msg=process killed
2019-12-19T22:28:28: PM2 log: Deleting process 3
2019-12-19T22:28:28: PM2 log: Stopping app:reg_10020 id:3
2019-12-19T22:28:28: PM2 log: App [reg_10020:3] exited with code [0] via signal [SIGTERM]
2019-12-19T22:28:28: PM2 log: pid=23085 msg=process killed
2019-12-19T22:28:28: PM2 log: Deleting process 4
2019-12-19T22:28:28: PM2 log: Stopping app:notify_10030 id:4
2019-12-19T22:28:28: PM2 log: App [notify_10030:4] exited with code [1] via signal [SIGINT]
2019-12-19T22:28:28: PM2 log: pid=23396 msg=process killed
Unitech commented 4 years ago

Could you please send me sample script.sh with python app so I can try to reproduce this issue?

mifenfeifen commented 4 years ago

Could you please send me sample script.sh with python app so I can try to reproduce this issue?

@Unitech I can't reboot the problem machine now, so I try to reproduce the issue on my own machine. The python codes have too much dependecies and I try to prune them, but I failed. I can't find the exactly python code to reproduce the issue, so I can't provide the python script.

But I had tried to stop the apps in pm2 and reboot the problem machine this morning, and it start the loop too.

problem machine info: Linux localhost.localdomain 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

restart loop log:

2019-12-20T11:09:55: PM2 log: ===============================================================================
2019-12-20T11:09:55: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2019-12-20T11:09:55: PM2 log: Time                 : Fri Dec 20 2019 11:09:55 GMT+0800 (GMT+08:00)
2019-12-20T11:09:55: PM2 log: PM2 version          : 4.2.1
2019-12-20T11:09:55: PM2 log: Node.js version      : 10.16.3
2019-12-20T11:09:55: PM2 log: Current arch         : x64
2019-12-20T11:09:55: PM2 log: PM2 home             : /root/.pm2
2019-12-20T11:09:55: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2019-12-20T11:09:55: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2019-12-20T11:09:55: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2019-12-20T11:09:55: PM2 log: Application log path : /root/.pm2/logs
2019-12-20T11:09:55: PM2 log: Worker Interval      : 30000
2019-12-20T11:09:55: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2019-12-20T11:09:55: PM2 log: Concurrent actions   : 2
2019-12-20T11:09:55: PM2 log: SIGTERM timeout      : 1600
2019-12-20T11:09:55: PM2 log: ===============================================================================
2019-12-20T11:09:55: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2019-12-20T11:09:55: PM2 log: App [pm2-logrotate:0] online
2019-12-20T11:11:25: PM2 log: pm2 has been killed by signal, dumping process list before exit...
2019-12-20T11:11:25: PM2 log: Deleting process 0
2019-12-20T11:11:25: PM2 log: Stopping app:pm2-logrotate id:0
2019-12-20T11:11:25: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGTERM]
2019-12-20T11:11:25: PM2 log: pid=1438 msg=process killed
2019-12-20T11:11:25: PM2 log: Deleting process 1
2019-12-20T11:11:25: PM2 log: Deleting process 2
2019-12-20T11:11:25: PM2 log: Deleting process 3
2019-12-20T11:11:25: PM2 log: Deleting process 4
2019-12-20T11:11:25: PM2 log: Exited peacefully
2019-12-20T11:11:26: PM2 log: ===============================================================================
2019-12-20T11:11:26: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2019-12-20T11:11:26: PM2 log: Time                 : Fri Dec 20 2019 11:11:26 GMT+0800 (GMT+08:00)
2019-12-20T11:11:26: PM2 log: PM2 version          : 4.2.1
2019-12-20T11:11:26: PM2 log: Node.js version      : 10.16.3
2019-12-20T11:11:26: PM2 log: Current arch         : x64
2019-12-20T11:11:26: PM2 log: PM2 home             : /root/.pm2
2019-12-20T11:11:26: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2019-12-20T11:11:26: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2019-12-20T11:11:26: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2019-12-20T11:11:26: PM2 log: Application log path : /root/.pm2/logs
2019-12-20T11:11:26: PM2 log: Worker Interval      : 30000
2019-12-20T11:11:26: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2019-12-20T11:11:26: PM2 log: Concurrent actions   : 2
2019-12-20T11:11:26: PM2 log: SIGTERM timeout      : 1600
2019-12-20T11:11:26: PM2 log: ===============================================================================
2019-12-20T11:11:26: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2019-12-20T11:11:26: PM2 log: App [pm2-logrotate:0] online
2019-12-20T11:12:55: PM2 log: pm2 has been killed by signal, dumping process list before exit...
2019-12-20T11:12:55: PM2 log: Deleting process 0
2019-12-20T11:12:55: PM2 log: Stopping app:pm2-logrotate id:0
2019-12-20T11:12:55: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGTERM]
2019-12-20T11:12:56: PM2 log: pid=9252 msg=process killed
2019-12-20T11:12:56: PM2 log: Deleting process 1
2019-12-20T11:12:56: PM2 log: Deleting process 2
2019-12-20T11:12:56: PM2 log: Deleting process 3
2019-12-20T11:12:56: PM2 log: Deleting process 4
2019-12-20T11:12:56: PM2 log: Exited peacefully
2019-12-20T11:12:56: PM2 log: ===============================================================================
rsuchit commented 4 years ago

@Unitech

Script.sh This will be just start the google app engine.

sudo kill `sudo lsof -t -i:8080`
sudo pkill java
export SDK_LOCATION="$(pwd)/google_appengine"
cd sympy_gamma
python deploy.py --generate-only --generate-test 1000
#sudo kill `sudo lsof -t -i:8080`
python $SDK_LOCATION/dev_appserver.py --host=165.22.190.231 --port=8080 --skip_$
#python $SDK_LOCATION/dev_appserver.py --host=165.22.190.231 --port=8080 --admi$

can you please help me about that?

mifenfeifen commented 4 years ago

@Unitech well，I have started the same code on my own machine, but I got confused that I can't reproduce the issue. SO I will tell the difference between the problem machine and my own mahcine to help your consideration. In addition, I will post some of my assumption.

……

I had wrote a lot in this comment. Before I submit it, I think of the situation of nginx: my own compiling edition in root folder can't startup with the machine because of SELinux. So I disabled selinux and reboot the machine, and pm2 works well.

SO, in my case, it's an SELinux issue.

mifenfeifen commented 4 years ago

root@ubuntu-s-photomaths:/var/www/html/sym_math# pm2 logs

@rsuchit I find you login as root , so do I. Maybe you can check my comment above and try to disable selinux and reboot the system again to find whether it's an selinux issue.