PM2 save doesn't seem to be working

[memphisraynz]

I am running centos7 and have just installed PM2. Added a service "pm2 start app.js --name lets-chat" which starts the program and its working fine. But after running pm2 startup centos and the pm2 save, upon reboot there is nothing list when I run pm2 list.

I'm not sure I am doing something wrong or if there is a bug.

Thanks

[artursudnik]

But it is really strange that it does not happen always and apparently there is no correlation to OS version.

[Holger-Will]

In my production machines which run ubuntu 14.04 LTS it all runs fine. On the devel boxes which are on ubuntu 15.04 or 15.10 however i can See this bug... has to be connected OS Version, as i update pm2 on all machines regularly.

[OriginalEXE]

has to be connected OS Version

I doubt it is since many of us has problems on different OS versions. I for example have this issue on 14.04

[yoava]

Same bug for me on raspbian (jessie/8). Worked fine 6 months ago (don't know which version). Node v4.2.1

[Holger-Will]

this might be connected to upstart, or ubuntus transition to systemd. I'm currently running a fresh install of arch with systemd, and pm2 works as expected. As soon as i have access to an ubuntu which shows the problem, i can check.

[Holger-Will]

oh, for everyone experiencing this error, please try

pm2 startup systemd

[rishabhmhjn]

@Holger-Will I tried the systemd and it also didn't work!

2 issues with systemd approach:

• /usr/bin/env: node if we use the auto generated startup pm2.service file
• When fixed that by adding source /path/to/nvm.sh, it still didn't work

PM2 v0.15.10
CentOS 7
node v5.3.0

PS: going with the immutable option for now as it seems to be solving the issue temporarily

[Holger-Will]

well ok :-( thanks for testing anyways...

[wacky6]

Same problem on Raspberry Pi 2, Debian 7, node v4.2.2, PM2 0.14.7

@vamship's solution seems better to me. Maybe PM2 should stop dumping process list in pm2-init.sh stop, and require user to manually commit(save) process list(similar to update-rc.d command)?

I am also curious about how PM2 dumps process list, if the process is killed but not restarted by PM2 during system shutdown.

[rudders]

Whilst I appreciate that keymetrics is an excellent commercialisation strategy for and open source project like PM2 the fact that this bug continues to not be resolved makes a mockery of their claims for production worthy use of this tool. A starting price of $30/month puts it at business level pricing but this bug flys in the face of anyone wanting to consider this a worthy platform for production use. #IMHO

[artursudnik]

+10

[ronaldb]

Commenting out the "super $PM2 dump" in /etc/init.d/pm2-init.sh seems to work. You just have to remember to do a "pm2 dump" manually after making changes.

A crude work around, but easier to do than making pm2.dump read-only and re-enabling writes when you make a change.

[artursudnik]

Ronalddb, seems it is most convenient workaround. Thanks! I've just used it.

[kenhowardpdx]

@ronaldb thanks for your workaround!

[h4t0n]

@ronaldb It works for me. Thanks.

pm2 v1.0.0 node v4.2.4 Centos 7.2

[benforshey]

@vamship Your workaround helped—thanks!

pm2 v1.0.0 node v5.4.1 Ubuntu 15.10

[Braynid]

@ronaldb - that works, thanks!

pm2 1.0.0 node v5.0.0 CentOS 7.2.1511

[pwang2]

Take me a while to get this workaround. is it an issue under working?

[artursudnik]

This issue is real mystery for me.

[OriginalEXE]

What I'm surprised about the most is the lack of interest from the authors on this issue. It's obviously widespread and a very important part of the software doesn't seem to be working, so why they still have not solved this (or at least let us know it can't be solved), I have no idea. I mean, I am grateful for their contribution and all, but this should be taken more seriously IMO.

[Unitech]

The issue about node paths / user permissions have been fixed since the PM2 0.15.x.

As I understand by reading this thread is that the automatic process dump (pm2 dump on stop) is the main issue.

I will disable this behavior and now users who wants to keep their current processes will have to expressly type pm2 save

[artursudnik]

Great decision!

[OriginalEXE]

Thank you @Unitech !

[Unitech]

Alright just tried on different machines, it works like a charm!

I've updated the documentation if anyone would have some time to verify that it's grammatically correct

[Unitech]

I wonder if this issue is linked with #1321

[Unitech]

1891

[Tomino2112]

I am not sure why, but for me pm2 is totally ignoring user configs.

what I need to do is : sudo -u www-data pm2 start ecosystem.json

But I get error: EACCES: permission denied, open '/home/tomas/.pm2/pm2.log'

How is it that even if I run command as a different user, it still goes into the original user's home?

I tried the startup script changing user to www-data but after restart no apps are running.

Also when I run pm2 start ecosystem -u www-data pm2 starts the app but still as a user tomas

[roccomuso]

Same error for me in Raspbian on Raspberry pi 2.

Tried to pm2 save and restart, but the processes are gone.

This is a crucial feature.

[kylecordes]

I am also surprised to see the trouble with this, so far in to the life of a generally very slick and polished tools - it seems to be right at the heart of pm2's purpose.

[artursudnik]

The workaround has been already implemented: https://github.com/Unitech/pm2/commit/f2714c7385681c7592cf4c5629a6a75efc7db4d9

[willkramer]

@kylecordes I agree and I've given up on the pm2 save functionality. The workarounds don't seem to work for me reliably on the Amazon Linux AMIs.

My solution was to create a bash script that runs the "pm2 start server.js" command and then create another bash script as a service in /etc/init.d/myserver. I can then start, stop, and restart pm2 with the command "service myserver start"

My pm2 start/stop/restart scripts look something like (this is my avoneu_start.sh):

#!/bin/bash
cd /avoneu/api
pm2 start server.js

I then create a file in /etc/init.d/ called avoneu. My service bash script looks like:

#!/bin/bash
# chkconfig: - 85 15
# description: The Avoneu Node Server
HOME_FOLDER="/home/ec2-user"

start(){
        echo "Starting node server"

        eval "runuser - ec2-user -c $HOME_FOLDER/avoneu_start.sh"
}
stop(){
        echo "Stopping node server"

        eval "runuser - ec2-user -c $HOME_FOLDER/avoneu_stop.sh"
}
restart(){
    echo "Restarting node server"

        eval "runuser - ec2-user -c $HOME_FOLDER/avoneu_restart.sh"
}
case "$1" in
  start)
        start
    ;;
  stop)
        stop
    ;;
  restart)
    restart
        ;;
  *)
        echo "Usage: avoneu {start|stop|restart}"
        exit 1
esac

exit 0

Also, I can easily ensure the the service auto starts on reboots by running the command

chkconfig avoneu on

This solution has been 100% reliable for the last 8 months and I was able to make it part of my server formation templates. I don't fault pm2 for having auto start issues; auto start on each Linux OS can be very different and it is hard to account for every minor difference.

[kylecordes]

I am using Amazon Linux. I found that it worked for me if I did the chattr thing to forcibly prevent the file from being damaged. The version I'm using already had the "save" command in the script commented out, that was not enough to prevent the breakage. So apparently it is not just that one command causing the breakage, there is something beyond that going on.

[zettam]

So what is the final state with this? Do we have to manually "save" the state now?

If this is the case, there is still a bug, which is: when you don't save anything and just power off the machine (no reboot or shutdown command, direct power off) and restart the final state is there (even though pm2 save command was not run before!)

So some things apparently get saved. I just do not understand why it just fails to work on machine reboots.

[minwe]

Not working +1：

Ubuntu 14.04.3 LTS
node -v v4.2.1
pm2 -v 1.1.3

I'm using Microsoft Azure, install pm2 with azureuser.

UPDATE：

I think it caused by fish shell. It works well after changing default shell to bash.

So, just check your default shell, does it work with export order? Fish shell doesn't.

[mjkoster]

I had to make the file immutable also, on Ubuntu 15.10. The dump file is being cleared when I do "pm2 kill". Apparently there is some problem...

[cmundi]

Problem still exists. Behavior suggests a race condition. Debian 8.5 pm2 version 1.1.3

Sometimes apps are restarted after reboot, sometimes not. When the dump file is wiped out, at least there is a reason for the failure to restart apps. But it also sometimes happens that the dump file is still intact after reboot, but apps are not restarted. Smells like race condition(s) to me, but I have no proof. These are multiple, successive reboots. There's no other change in system configuration between reboots. This is disappointing, but fortunately there are lots of other ways to run jobs at startup on a linux box if pm2 ends up not being reliable on my boxes.

Making the dump file immutable may solve the problem for some systems. But immutability does not solve the problem for my minimal Debian 8.5 boxes. I made the dump file immutable, so at least it "can't" be rewritten. (And I have cached a copy and verify that it is not changing between reboots.) Then I do a series of reboots and checked to see if my app restarted:

reboot 1 ... app running reboot 2 ... app NOT running reboot 3 ... app running reboot 4 ... app running reboot 5 ... app running reboot 6 ... app NOT running reboot 7 ...app running

So even after the file is made immutable, the behavior is inconsistent. I think it's fair to call that a surprising result. I suspect a race condition. As others have reported, the logs reflect the problem but offer no clues to the cause.

Not that it should matter but this experiment was performed on a VM. All pm2 commands run as root.

[boatrite]

Just to throw the solution I found into this massive issue... I was following the instructions at https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps and ran sudo env PATH=$PATH:/opt/node-v4.4.6-linux-x64/bin/ pm2 startup ubuntu -u deploy. This is as the ubuntu user of an Amazon AWS instance, Ubuntu 14.04.

However, this is what was generated at /etc/init.d/pm2-init.sh (relevant bit only):

export PM2_HOME="/home/ubuntu/.pm2"

So the problem is that pm2 startup didn't set PM2_HOME correct, even with specifying the user. After changing PM2_HOME to be correct, everything worked perfectly.

[cmundi]

@boatrite Thanks for that link and example. That addresses one of the several behaviors which make RCA challenging for this constellation of related behaviors:

1. User not being set correctly, as you describe.
2. The dump file being overwritten, sometimes.
3. My report that even when user is correct (including root) and dump file is made immutable, sometimes pm2 starts on boot and sometimes it does not. It seems random.

I like the goals of pm2 and wish I had time to dive into the code. Right now I need to go with the reliability and transparency of init/systemd. Even though our clusters are small and well instrumented, we'll keep an eye on pm2.

Thanks!

[qraynaud]

@cmundi : your problem seems to be a completely separate bug then.

Maybe you should file it separately then? My rationale is that since the issue described here is properly fixed by the changes suggested by @boatrite, we need 2 patches : one or this issue and one for yours.

[github-jeff]

Having read this commentary. I have tried everything which was plausible. The dump file was not getting overwritten (wiped out) on reboot, but the dump file structure was changed from human readable to one single line. By making it read only this did save the human readable structure. The pm2-init.sh file already had the super dump line already commented out. On reboot pm2 does not run as a service, but it does run demonized when I try pm2 list. In any case the app does not survive reboot on ubuntu 14.04.

Any other tips and tricks I should try?

[Unitech]

New startup system that will surely fix a bunch of problems mentioned on this issue:

https://github.com/Unitech/pm2/issues/2559

[dikkedimi]

I'm currently having this issue on a raspberry pi 3, with raspbian (latest build) and pm2 v2.4.2

[vmarchaud]

@dikkedimi Check this issue : https://github.com/Unitech/pm2/issues/2747

[Ext3R]

"pm2 resurrect" works for me after rebooting - but not automatically. -> added the code: "@reboot pm2 resurrect" to "crontab -e" -> now everything works just fine. Don´t know what fails here. It saves the logs perfectly but cant start the apps automatically.

[lukepighetti]

Is there any way to save and resurrect the running state of each?

Lets say:

fooApp  online
barApp  stopped
bazApp  online

Can I set it up so that fooApp/bazApp are online, and barApp is stopped on reboot?

Currently they start up all online.

[wallet77]

Hi @lukepighetti

Currently is not possible to save and resurrect the exact state. PM2 as a process manager always tries to restart your apps.

But this is part of a reflection about pm2 save/resurrect, see this issue : https://github.com/Unitech/pm2/issues/3759

[RicardoZap]

Una disculpa por comentar en español en un foro en Inglés pero encontré la solución al problema. Ami me funcionó ejecutando el siguiente método:

1.- pm2 start "ejemplo.js" 2.- pm2 save 3.- pm2 startup 4.- sudo env PATH=$PATH:/usr/bin /usr/local/lib/node_modules/pm2/bin/pm2 startup systemd -u pi --hp /home/pi

Después de ejecutar lo anterior hacer un "sudo reboot" y con eso el problema queda resuelto. Saludos.

[kadalcj]

Hi,

I currently having this issue on RHEL 8, I am a bit confused because previously I was testing it on CentOS and it was perfectly fine. On CentOS, saved process was running after reboot. But on Red Hat, it was not and it saying something like "Not Sync".

[roychowdhuryrohit-dev]

@Ext3R Is this issue fixed yet? According to the official docs, they make it sound like adding the startup script and pm2 save are enough to start all saved process on reboot. But I still need to pm2 resurrect every single time after it reboots.

@wallet77 How can I auto resurrect all saved processes without running any cronjob for pm2 resurrect on reboot?

[mxavelli]

Una disculpa por comentar en español en un foro en Inglés pero encontré la solución al problema. Ami me funcionó ejecutando el siguiente método:

1.- pm2 start "ejemplo.js" 2.- pm2 save 3.- pm2 startup 4.- sudo env PATH=$PATH:/usr/bin /usr/local/lib/node_modules/pm2/bin/pm2 startup systemd -u pi --hp /home/pi

Después de ejecutar lo anterior hacer un "sudo reboot" y con eso el problema queda resuelto. Saludos.

• Tendrás alguna equivalencia para Windows?

• Do you have any similar process for Windows?