search

Unitech / pm2

Node.js Production Process Manager with a built-in Load Balancer.
https://pm2.keymetrics.io/docs/usage/quick-start/
Other
41.61k stars 2.62k forks source link

PM2 save doesn't seem to be working #1035

Closed memphisraynz closed 7 years ago

memphisraynz commented 9 years ago

I am running centos7 and have just installed PM2. Added a service "pm2 start app.js --name lets-chat" which starts the program and its working fine. But after running pm2 startup centos and the pm2 save, upon reboot there is nothing list when I run pm2 list.

I'm not sure I am doing something wrong or if there is a bug.

Thanks

Unitech commented 9 years ago

Can you try to do pm2 resurrect to see if the process are spawned back? Once the server is restarted, can you see the process PM2 in memory (via top/htop)?

memphisraynz commented 9 years ago

While pm2 is working and I run pm2 resurrect it spawns another process After reboot pm2 is in top, right at the bottom ( 2061 root 20 0 661384 21796 5160 S 0.0 2.2 0:00.14 PM2 v0.12.6: Go) Running pm2 list and pm2 resurrect after reboot brings up an emtpy table.

Thank you for your help

soyuka commented 9 years ago

Try this:

pm2 kill #start on a fresh basis
pm2 start myapp.js 
pm2 save #dump
pm2 kill #kill again to test
pm2 resurrect 
pm2 list #is the process myapp in the list?

If it's not in the list we got an issue. Rebooting seems not to be the issue here.

memphisraynz commented 9 years ago

Doing the above steps results in having the app.js handled by pm2 website is running no problems

On Wed Feb 25 2015 at 19:01:55 Antoine Bluchet notifications@github.com wrote:

Try this:

pm2 kill #start on a fresh basis pm2 start myapp.js pm2 save #dump pm2 kill #kill again to test pm2 resurrect pm2 list #is the process myapp in the list?

If it's not in the list we got an issue. Rebooting seems not to be the issue here.

— Reply to this email directly or view it on GitHub https://github.com/Unitech/PM2/issues/1035#issuecomment-75926601.

memphisraynz commented 9 years ago

Any other ideas on why PM2 is having troubles resurrecting the processes after reboot?

Thanks

webuniverseio commented 9 years ago

Same issue on debian 7. When I manually use save and resurrect it is working just fine. When I'm rebooting (obviously I executed pm2 startup before that) I'm getting empty list.

frankized commented 9 years ago

same issue on beaglebone black debian. As a temporary workaround, I set the dump file as immutable after "pm2 save": $ chattr +i /path/to/my/dump.pm2

EdHubbell commented 9 years ago

This happens to me, but only when the server runs out of memory. I get an out of memory error in my pm2 logs like so (which I know is my fault, not pm2's):

PM2: 2015-04-20 21:04:48: ERROR CAUGHT BY DOMAIN:
PM2: Error: spawn ENOMEM
PM2:     at exports._errnoException (util.js:746:11)
PM2:     at ChildProcess.spawn (child_process.js:1162:11)
PM2:     at exports.spawn (child_process.js:995:9)
PM2:     at Object.exports.execFile (child_process.js:689:15)
PM2:     at exports.exec (child_process.js:649:18)
PM2:     at getUrl (/usr/local/lib/node_modules/pm2/node_modules/vizion/lib/git.js:17:5)
PM2:     at fn (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
PM2:     at Immediate._onImmediate (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
PM2:     at processImmediate [as _immediateCallback] (timers.js:358:17)
PM2: child_process.js:1162
PM2:     throw errnoException(err, 'spawn');
PM2:           ^
PM2: Error: spawn ENOMEM
PM2:     at exports._errnoException (util.js:746:11)
PM2:     at ChildProcess.spawn (child_process.js:1162:11)
PM2:     at exports.spawn (child_process.js:995:9)
PM2:     at Object.exports.execFile (child_process.js:689:15)
PM2:     at exports.exec (child_process.js:649:18)
PM2:     at getUrl (/usr/local/lib/node_modules/pm2/node_modules/vizion/lib/git.js:17:5)
PM2:     at fn (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
PM2:     at Immediate._onImmediate (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
PM2:     at processImmediate [as _immediateCallback] (timers.js:358:17)

Then on restart, pm2 list show no processes and pm2 restart does nothing. This happens even though from what I can tell, the dump.pm2 file in ~/.pm2 doesn't get changed in any way. I saved a copy of a file that was fine and a file that was no good (after crash), and they were binary same.

Need to dig into this some more - I can't seem to change my dump.pm2 file to be read only. Probably because I'm a debian idiot. As it stands, if I write some bad node.js code, rebooting the server won't restart what had been running. This is keeping me from being able to migrate servers - Any tips would be helpful. I've been running pm2 for over a year on another server just fine. Maybe wasn't wheezy tho..

EdHubbell commented 9 years ago

I changed my dump.pm2 file to be immutable (essentially read only even for root) as suggested by @frankized, and that seems to have helped. Actually got a restart after an out of memory error. 

PM2: 2015-04-20 22:22:09: ERROR CAUGHT BY DOMAIN:
PM2: Error: spawn ENOMEM
PM2:     at exports._errnoException (util.js:746:11)
PM2:     at ChildProcess.spawn (child_process.js:1162:11)
PM2:     at exports.spawn (child_process.js:995:9)
PM2:     at Object.exports.execFile (child_process.js:689:15)
PM2:     at exports.exec (child_process.js:649:18)
PM2:     at getUrl (/usr/local/lib/node_modules/pm2/node_modules/vizion/lib/git.js:17:5)
PM2:     at fn (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
PM2:     at Immediate._onImmediate (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
PM2:     at processImmediate [as _immediateCallback] (timers.js:358:17)
PM2: Be sure to have the latest version by doing `npm install pm2@latest -g` before doing this procedure.
PM2: Error: EACCES, permission denied '/root/.pm2/dump.pm2'
PM2:     at Error (native)
PM2:     at Object.fs.openSync (fs.js:500:18)
PM2:     at Object.fs.writeFileSync (fs.js:1099:15)
PM2:     at fin (/usr/local/lib/node_modules/pm2/lib/CLI.js:646:12)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:655:28)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:661:14)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:661:14)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:661:14)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:661:14)
PM2:     at /usr/local/lib/node_modules/pm2/lib/CLI.js:662:7
PM2: [PM2] Stopping PM2...
PM2: 2015-04-20 22:22:16: Stopping app:mainOauth id:0
PM2: 2015-04-20 22:22:16: App name:mainOauth id:0 exited with code SIGTERM
PM2: 2015-04-20 22:22:16: [HandleExit] PM2 is being killed, stopping restart procedure...
PM2: 2015-04-20 22:22:16: Process with pid 9708 killed
PM2: [PM2] deleteProcessId process id 0
PM2: 2015-04-20 22:22:16: Stopping app:geotweet id:1
PM2: 2015-04-20 22:22:16: App name:geotweet id:1 exited with code SIGTERM
PM2: 2015-04-20 22:22:16: [HandleExit] PM2 is being killed, stopping restart procedure...
PM2: 2015-04-20 22:22:16: Process with pid 9711 killed
PM2: [PM2] deleteProcessId process id 1
PM2: 2015-04-20 22:22:16: Stopping app:crontruxie id:2
PM2: 2015-04-20 22:22:16: App name:crontruxie id:2 exited with code SIGTERM
PM2: 2015-04-20 22:22:16: [HandleExit] PM2 is being killed, stopping restart procedure...
PM2: 2015-04-20 22:22:16: Process with pid 9715 killed
PM2: [PM2] deleteProcessId process id 2
PM2: 2015-04-20 22:22:16: Stopping app:cronCalendar id:3
PM2: 2015-04-20 22:22:16: App name:cronCalendar id:3 exited with code SIGTERM
PM2: 2015-04-20 22:22:16: [HandleExit] PM2 is being killed, stopping restart procedure...
PM2: 2015-04-20 22:22:16: Process with pid 9722 killed
PM2: [PM2] deleteProcessId process id 3
PM2: [PM2] All processes have been stopped and deleted
PM2: 2015-04-20 22:22:16: PM2 is being killed via kill method
PM2: 2015-04-20 22:22:16: RPC socket closed
PM2: 2015-04-20 22:22:16: PUB socket closed
PM2: [PM2] PM2 stopped
PM2: 2015-04-20 22:22:17: [PM2][WORKER] Started with refreshing interval: 30000
PM2: 2015-04-20 22:22:17: [[[[ PM2/God daemon launched ]]]]
PM2: 2015-04-20 22:22:17: BUS system [READY] on port /root/.pm2/pub.sock
PM2: 2015-04-20 22:22:17: RPC interface [READY] on port /root/.pm2/rpc.sock
PM2: 2015-04-20 22:22:17: Starting execution sequence in -fork mode- for app name:mainOauth id:0
PM2: 2015-04-20 22:22:17: App name:mainOauth id:0 online
PM2: Process /srv/truxie/mainOauth.coffee launched
PM2: 2015-04-20 22:22:17: Starting execution sequence in -fork mode- for app name:geotweet id:1
PM2: 2015-04-20 22:22:18: App name:geotweet id:1 online
PM2: Process /srv/geotweet/geotweet.coffee launched
PM2: 2015-04-20 22:22:18: Starting execution sequence in -fork mode- for app name:crontruxie id:2
PM2: 2015-04-20 22:22:18: App name:crontruxie id:2 online
PM2: Process /srv/crontruxie/crontruxie.coffee launched
PM2: 2015-04-20 22:22:18: Starting execution sequence in -fork mode- for app name:cronCalendar id:3
PM2: 2015-04-20 22:22:18: App name:cronCalendar id:3 online
PM2: Process /srv/crontruxie/cronCalendar.coffee launched
PM2: >>>>>>>>>> PM2 updated
PM2: ┌──────────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────────────┬──────────┐
PM2: │ App name     │ id │ mode │ pid   │ status │ restart │ uptime │ memory      │ watching │
PM2: ├──────────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────────────┼──────────┤
PM2: │ mainOauth    │ 0  │ fork │ 10043 │ online │ 0       │ 1s     │ 16.914 MB   │ disabled │
PM2: │ geotweet     │ 1  │ fork │ 10046 │ online │ 0       │ 1s     │ 13.301 MB   │ disabled │
PM2: │ crontruxie   │ 2  │ fork │ 10051 │ online │ 0       │ 1s     │ 12.383 MB   │ disabled │
PM2: │ cronCalendar │ 3  │ fork │ 10058 │ online │ 0       │ 1s     │ 8.000 MB    │ disabled │
PM2: └──────────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────────────┴──────────┘
PM2:  Use `pm2 show <id|name>` to get more details about an app

I need a reliable way to create out of memory issues so I can test and make sure this is going to settle things.

jshkurti commented 9 years ago

It should work better with PM2 v0.12.13

buunguyen commented 9 years ago

@jshkurti I still have this issue running the latest version of PM2 (0.12.15) on Ubuntu 15.

After starting a process and running pm2 dump, the dump file is created with contents as expected. Testing with pm2 resurrect also works. However, after reboot, the process is not resurrected because the dump file is empty (cat ~/.pm2/dump.pm2 prints []).

Please reopen this issue and have a look if you could. Thank you.

PM2 0.12.15 Node 0.12.3 Ubuntu 15.04

jshkurti commented 9 years ago

@buunguyen Are you using Keymetrics ? Does ps aux | grep Keymetrics show something about 'Keymetrics.io Agent' ?

buunguyen commented 9 years ago

No, I haven't installed Keymetrics on this particular host yet but I intend to. If I install the agent now, do you think it would fix the problem?

crashbell commented 9 years ago

@buunguyen I got the same issue and updating to 0.12.16 seems working now.

buunguyen commented 9 years ago

@crashbell thanks. Currently I'm working around by making dump file immutable. A bit annoying but not too painful. Will try the new version later.

ravishivt commented 9 years ago

I'm using v0.14.5 and still have the issue. On every reboot, the .pm2/dump.pm2 gets cleared.

arisAlexis commented 9 years ago

doesn't work with vanilla instructions on ubuntu 14.04 either (npm install pm2) and startup steps and save.

sfakir commented 9 years ago

Same problem on RedHat 7.0 :( the dump file is empty after reboot

tangjie0710 commented 9 years ago

I also got the same problem on ubuntu 14.04 and PM2 0.14.7 I found a solution in https://github.com/Unitech/PM2/issues/1055. rbudiharso gave the solution. You have to add PM2_HOME to the start up script. I hope this will give help.

crashbell commented 9 years ago

My server restarted last night and then all processes were started automatically. The dump file wasn't empty and Keymetrics was installed. I don't remember if I did any configuration. I'll check around and let you know soon.

Still using 0.12.16

mojojoseph commented 9 years ago

Likewise, I have the same issue - a reboot wipes out ~/.pm2/dump.pm2. I am on a BeagleBone Black running Ubuntu with pm2 0.14.7.

@tangjie0710 My start-up script does have PM2_HOME set: export PATH=/usr/bin:$PATH export PM2_HOME="/root/.pm2"

glskiba commented 9 years ago

I've has similar problems, and checked other online threads. try making sure all files in ~/user/.pm2 are owned by the user, not root.
This worked for me: sudo chown ~/.pm2/*

jshkurti commented 9 years ago

Thanks @glskiba !

EdHubbell commented 9 years ago

I love it when I google a problem I'm having and I get back to a GitHub issue thread that I don't even remember commenting on.

Lost a process after a reboot earlier this week. Couldn't write to the dump.pm2 file because I had set it to immutable 6 months ago. I don't remember doing that specifically, but there it is in my comment.

For the record, updating PM2 to the latest version seems to have solved my reboot problem from 6 months ago - Now all my processes restart. Debian server working just fine. Apologies for using this thread as my dev log. But maybe this helps someone else.

artursudnik commented 9 years ago

I did some narrowing of the same problem that I have now.

Ubuntu 15.04 node v0.12.7 pm2 0.14.7

configured pm2 with:

sudo su -c "env PATH=$PATH:/usr/bin pm2 startup ubuntu -u artur"

edited pm2-init.sh file:

NAME=pm2
PM2=/usr/lib/node_modules/pm2/bin/pm2
USER=artur

export PATH=/usr/bin:$PATH
export PM2_HOME="/home/artur/.pm2"

after reboot

dump file is being emptied

pm2.log file contains then (this is only what has been written to the log after executing reboot):

2015-09-10 17:14:10: [PM2][WORKER] Started with refreshing interval: 30000
2015-09-10 17:14:10: [[[[ PM2/God daemon launched ]]]]
2015-09-10 17:14:10: BUS system [READY] on port /home/artur/.pm2/pub.sock
2015-09-10 17:14:10: RPC interface [READY] on port /home/artur/.pm2/rpc.sock
2015-09-10 17:14:11: PM2 is being killed via kill method
2015-09-10 17:14:11: RPC socket closed
2015-09-10 17:14:11: PUB socket closed
2015-09-10 17:14:41: [PM2][WORKER] Started with refreshing interval: 30000
2015-09-10 17:14:41: [[[[ PM2/God daemon launched ]]]]
2015-09-10 17:14:41: BUS system [READY] on port /home/artur/.pm2/pub.sock
2015-09-10 17:14:41: RPC interface [READY] on port /home/artur/.pm2/rpc.sock
akmoulai commented 9 years ago

Debian GNU/Linux 8 (jessie) / systemd / io.js 2.5.0 / pm2 0.14.7

Same issue: after a reboot, the dump.pm2 contains an empty array (i.e. []) instead of what it contained after pm2 save. pm2.log contains about the same as @artursudnik 's one (no issue reported).

I had to resort to @frankized 's solution (thank you!) which is an easy workaround for my small project.

artursudnik commented 9 years ago

I just made a test on Ubuntu 14.04.3 LTS and there pm2 seems to work correctly.

I guess problem is somehow related to shutdown sequence od operating system. Maybe it is killed then starts and pm2 dump is being made before applications manage to resurrect?

OriginalEXE commented 9 years ago

I am also experiencing this issue.

Ubuntu 14.04.3 LTS, dump.pm2 being emptied reboot (leaves only [] in the file).

sydcanem commented 9 years ago

Had the same issue in Centos7 with pm2 v0.14.7 running as normal user.

After reboot dump.pm2 is empty.

qraynaud commented 9 years ago

Same problem here. I have made my file immutable in the meantime but it is not a good solution for the long run.

lklepner commented 9 years ago

I had a similar problem on my end running on Ubuntu 14.04.

When I ran - sudo su -c "env PATH=$PATH:/usr/bin pm2 startup ubuntu -u ds"

The init.d script has - export PM2_HOME="/root/.pm2"

Whereas it should be - export PM2_HOME="/home/ds/.pm2"

ashleydavis commented 9 years ago

It seems I can't have my pm2 processes survive restart. This used to work. I still use the same deployment script. But 'pm2 save' doesn't persist across VM restart. Using Ubuntu 14.04.

iongion commented 9 years ago

@lklepner same issue - same wrong $PM2_HOME, this ticket should be re-opened

artursudnik commented 9 years ago

wrong $PM2_HOME value is not the only reason of not restarting processes

iongion commented 9 years ago

When installing with an user: iongion: pm2 startup ubuntu

[PM2] You have to run this command as root. Execute the following command:
sudo su -c "env PATH=$PATH:/usr/local/bin pm2 startup ubuntu -u iongion"

I do that and the pm2-init.sh header becomes:

NAME=pm2
PM2=/usr/local/lib/node_modules/pm2/bin/pm2
USER=root
export PATH=/usr/local/bin:$PATH
export PM2_HOME="/home/iongion/.pm2"

When installing from a user with sudo: sudo pm2 install ubuntu

NAME=pm2
PM2=/usr/local/lib/node_modules/pm2/bin/pm2
USER=iongion
export PATH=/usr/local/bin:$PATH
export PM2_HOME="/root/.pm2"

Aren't they both wrong ? Shouldn't the config be:

NAME=pm2
PM2=/usr/local/lib/node_modules/pm2/bin/pm2
USER=iongion
export PATH=/usr/local/bin:$PATH
export PM2_HOME="/home/iongion/.pm2"

In all cases, after each restart the dump file is empty, it gets populated by save but becomes empty after each restart

tomlankhorst commented 9 years ago

I do experience the same problem. My situation:

Using an init.d script generated using the su env ... command as proposed by pm2 startup. However, because the dedicated user is not in the sudoers I executed this command as another user and changed the init.d script to the right user and home folder. Setting the immutable bit solves this (temporarily) for me too.

choxnox commented 9 years ago

I have the same issue on the following:

What also worked for me is setting the immutable attribute which is an ugly hack because I'll most probably forget I did it when I want to modify dump.pm2 in the future (adding/removing applications in PM2).

Holger-Will commented 9 years ago

Same issue here

dump.pm2 is empty after reboot. I'm also using the immutable hack now...

ccravens commented 9 years ago

PING Any help with this?? My dump.pm2 also clears after reboot and none of my processes start, which kind of defeats the whole purpose of using PM2 as a process manager in the first place! I think marking the dump file as immutable on the filesystem is a terrible workaround... Why doesn't PM2 just do what it claims to do instead?

PM2 v0.15.10 CentOS 7 node v4.2.1

tomlankhorst commented 9 years ago

Please reopen the issue.

ccravens commented 9 years ago

+1 for re-opening this issue

lklepner commented 9 years ago

+1 for reopen

dimd13 commented 9 years ago

Debian 8.2 /home/user/.pm2/dump.pm2 is empty after reboot

mlota commented 8 years ago

Same problem for me I'm using pm2 v0.15.10 on Raspbian Jessie. The $PM2_HOME looks correct but it loses the data after every reboot

bhaveshgohel commented 8 years ago

@mlota : Same here. :( Node.js : v4.2.3 Pm2 version : v0.15.10 Hardware : Raspberry Pi 1 B+ OS : DietPi Jessie

Workaround if anyone needed: Make your dump.pm2 file IMMUTABLE (Unchangeable).

Run below commands to make dump.pm2 IMMUTABLE: 

sudo pm2 save
sudo chattr +i  /your/path/to/dump.pm2

The problem is that for any change you have to, 

sudo chattr -i  /your/path/to/dump.pm2
sudo pm2 save
sudo chattr +i  /your/path/to/dump.pm2

Hoping this issue in pm2 solves soon. :)

artursudnik commented 8 years ago

Why it takes so long to address this very serious issue?

webuniverseio commented 8 years ago

@artursudnik are you offering your help with these words?

artursudnik commented 8 years ago

No, I am just curious about a reason. Is this matter of design? If I only were familiar with the source code I could try to debug it. What I has done is performing some tests on number of linux versions but cannot find any correlations. Seems to be random for me.

mlota commented 8 years ago

I've also tried all the workarounds posted but in the end just had to ditch PM2 and use some startup scripts instead. It's a real shame as I have it working on another system flawlessly which has an identical setup. Like @artursudnik, I find the issue is totally random

vamship commented 8 years ago

I'm having the same issue on a minimal jessie on the raspberry pi.

It appears that PM2 is dumping all running processes at shutdown, but this is somehow taking place after all the PM2 processes have been killed, which means that the dump file now has an empty array, and nothing starts on bootup.

My workaround was to change the /etc/init.d/pm2-init.sh, and comment out the pm2 dump statement under the stop() function. I can then add scripts to startup by manually calling pm2 dump after launching the processes.

It would be cool if the pm2 startup script took an option that would automatically suppress auto save of processes at shutdown, instead of me having to go in there and tweak the shell script.