search

Unitech / pm2

Node.js Production Process Manager with a built-in Load Balancer.
https://pm2.keymetrics.io/docs/usage/quick-start/
Other
41.44k stars 2.62k forks source link

[QUESTION] pm2 startup not starting up Ubuntu 16 reboot #3148

Closed austintoddj closed 2 years ago

austintoddj commented 7 years ago

Your issue may already be reported! Please search on the issue tracker before creating one.

What's going wrong?

On server reboot, pm2 is not restarting the two applications that I've got. I run my process as such:

pm2 start npm --name "project" -- start
pm2 startup
// Run the command that's given
pm2 save

I don't know if it has something to do with what user permissions have to be regarding pm2, but every time I reboot the server, the applications aren't running anymore and I need to run pm2 resurrect.

How could we reproduce this issue?

Run the steps given above on a Ubuntu 16 server.

Supporting information

None

Please run the following command (available on PM2 >= 2.6)

===============================================================================
--- PM2 REPORT (Mon Sep 11 2017 19:59:10 GMT-0400 (EDT)) ----------------------
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 2.6.1
node version         : 8.4.0
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/usr/local/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : austintoddj
uid                  : 1000
gid                  : 1000
uptime               : 19min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 2.6.1
node version         : 8.4.0
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/usr/local/bin/pm2,report
argv0                : node
user                 : austintoddj
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
cpus nb              : 1
freemem              : 158756864
totalmem             : 1040515072
home                 : /home/austintoddj
===============================================================================
--- PM2 list -----------------------------------------------
┌───────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬─────────────┬──────────┐
│ App name      │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ user        │ watching │
├───────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼─────────────┼──────────┤
│ docs.cnvs.io  │ 1  │ fork │ 2559 │ online │ 0       │ 16m    │ 0%  │ 38.1 MB   │ austintoddj │ disabled │
│ toddaustin.io │ 0  │ fork │ 2517 │ online │ 0       │ 16m    │ 0%  │ 39.4 MB   │ austintoddj │ disabled │
└───────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴─────────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
┌───────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬─────────────┬──────────┐
│ App name      │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ user        │ watching │
├───────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼─────────────┼──────────┤
│ docs.cnvs.io  │ 1  │ fork │ 2559 │ online │ 0       │ 16m    │ 0%  │ 38.1 MB   │ austintoddj │ disabled │
│ toddaustin.io │ 0  │ fork │ 2517 │ online │ 0       │ 16m    │ 0%  │ 39.4 MB   │ austintoddj │ disabled │
└───────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴─────────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app
vmarchaud commented 7 years ago

Could you run sudo service --status-all and give me the output ? Also could you check that the file ~/.pm2/dump.pm2 is not empty (should contains all processes metadatas) ?

austintoddj commented 7 years ago

@vmarchaud Here is the output of sudo service --status-all:

[ + ]  acpid
 [ + ]  apache-htcacheclean
 [ + ]  apache2
 [ + ]  apparmor
 [ + ]  apport
 [ + ]  atd
 [ - ]  bootmisc.sh
 [ - ]  checkfs.sh
 [ - ]  checkroot-bootclean.sh
 [ - ]  checkroot.sh
 [ + ]  console-setup
 [ + ]  cron
 [ - ]  cryptdisks
 [ - ]  cryptdisks-early
 [ + ]  dbus
 [ + ]  denyhosts
 [ + ]  fail2ban
 [ + ]  grub-common
 [ - ]  hostname.sh
 [ - ]  hwclock.sh
 [ + ]  irqbalance
 [ + ]  iscsid
 [ - ]  keyboard-setup.dpkg-bak
 [ - ]  killprocs
 [ + ]  kmod
 [ - ]  lvm2
 [ + ]  lvm2-lvmetad
 [ + ]  lvm2-lvmpolld
 [ + ]  lxcfs
 [ - ]  lxd
 [ + ]  mdadm
 [ - ]  mdadm-waitidle
 [ - ]  mountall-bootclean.sh
 [ - ]  mountall.sh
 [ - ]  mountdevsubfs.sh
 [ - ]  mountkernfs.sh
 [ - ]  mountnfs-bootclean.sh
 [ - ]  mountnfs.sh
 [ + ]  mysql
 [ + ]  networking
 [ + ]  ondemand
 [ + ]  open-iscsi
 [ - ]  plymouth
 [ - ]  plymouth-log
 [ + ]  postfix
 [ + ]  procps
 [ + ]  rc.local
 [ + ]  resolvconf
 [ - ]  rsync
 [ + ]  rsyslog
 [ - ]  screen-cleanup
 [ - ]  sendsigs
 [ + ]  ssh
 [ + ]  udev
 [ + ]  ufw
 [ - ]  umountfs
 [ - ]  umountnfs.sh
 [ - ]  umountroot
 [ + ]  unattended-upgrades
 [ + ]  urandom
 [ - ]  uuidd

And I did verify that the file at ~/.pm2/dump.pm2 is NOT empty.

vmarchaud commented 7 years ago

looks like you didn't run pm2 startup because you don't have pm2 service listed here

austintoddj commented 7 years ago

@vmarchaud Ran both pm2 startup and sudo pm2 startup and neither made pm2 show up in that list. Any idea why?

vmarchaud commented 7 years ago

Because pm2 startup just output you the command you need to run to register itself

austintoddj commented 7 years ago

@vmarchaud Ok, ran the startup command AND then this: sudo env PATH=$PATH:/usr/local/bin /usr/local/lib/node_modules/pm2/bin/pm2 startup systemd -u austintoddj --hp /home/austintoddj and still nothing shows up in the services list. Ideas?

austintoddj commented 7 years ago

@vmarchaud Here are the contents of ~/.pm2/dump.pm2. Does this give any insight?

...
vmarchaud commented 7 years ago

Nope, can you try to ls /etc/systemd/system to see if the service is written to the disk ?

austintoddj commented 7 years ago

@vmarchaud Here's the output of that:

cloud-init.target.wants  getty.target.wants      multi-user.target.wants      pm2-austintoddj.service  sshd.service          timers.target.wants
default.target.wants     graphical.target.wants  network-online.target.wants  pm2-root.service         sysinit.target.wants  udev-finish.service
final.target.wants       iscsi.service           paths.target.wants           sockets.target.wants     syslog.service
nivoset commented 7 years ago

having the same problem.

pm2 -v 2.6.1

node -v v6.11.0

npm -v 3.10.10

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial

service --status-all [ + ] pm2-dd

/etc/systemd/system/pm2-dd.service exists

I can start up all the processes by doing a pm2 resurrect every boot no problem

This was updated from an earlier ubuntu 14. don't think it matters, but at this point ill add any data.

vmarchaud commented 7 years ago

@nivoset You should have a daemon running on boot but under user dd

@austintoddj Checkout that service are running with sudo service pm2-root status and sudo service pm2-austintoddj status, you have two daemon registered, maybe that you aren't connected to the good one.

austintoddj commented 7 years ago

@vmarchaud So here's the output when I run the commands you gave me:

sudo service pm2-root status

...outputs:

● pm2-root.service - PM2 process manager
   Loaded: loaded (/etc/systemd/system/pm2-root.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Tue 2017-09-19 08:21:33 EDT; 2 days ago
     Docs: https://pm2.keymetrics.io/

Sep 19 08:20:03 whispering-ocean systemd[1]: Starting PM2 process manager...
Sep 19 08:20:08 whispering-ocean pm2[1261]: [PM2] Spawning PM2 daemon with pm2_home=/home/austintoddj/.pm2
Sep 19 08:21:33 whispering-ocean systemd[1]: pm2-root.service: Start operation timed out. Terminating.
Sep 19 08:21:33 whispering-ocean systemd[1]: Failed to start PM2 process manager.
Sep 19 08:21:33 whispering-ocean systemd[1]: pm2-root.service: Unit entered failed state.
Sep 19 08:21:33 whispering-ocean systemd[1]: pm2-root.service: Failed with result 'timeout'.
sudo service pm2-austintoddj status

...outputs:

● pm2-austintoddj.service - PM2 process manager
   Loaded: loaded (/etc/systemd/system/pm2-austintoddj.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-09-19 08:32:03 EDT; 2 days ago
     Docs: https://pm2.keymetrics.io/
 Main PID: 2487 (PM2 v2.7.0: God)
    Tasks: 0
   Memory: 1.5M
      CPU: 495ms
   CGroup: /system.slice/pm2-austintoddj.service
           ‣ 2487 PM2 v2.7.0: God Daemon (/home/austintoddj/.pm2)   

Sep 19 08:32:03 whispering-ocean pm2[2645]: [PM2][ERROR] Failed to read dump file in /home/austintoddj/.pm2/dump.pm2.bak
Sep 19 08:32:03 whispering-ocean pm2[2645]: [PM2][ERROR] No processes saved; DUMP file doesn't exist
Sep 19 08:32:03 whispering-ocean pm2[2645]: ┌───────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬─────────────┬──────────┐
Sep 19 08:32:03 whispering-ocean pm2[2645]: │ App name      │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ user        │ watching │
Sep 19 08:32:03 whispering-ocean pm2[2645]: ├───────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼─────────────┼──────────┤
Sep 19 08:32:03 whispering-ocean pm2[2645]: │ docs.cnvs.io  │ 1  │ fork │ 2546 │ online │ 0       │ 43s    │ 0%  │ 41.4 MB   │ austintoddj │ disabled │
Sep 19 08:32:03 whispering-ocean pm2[2645]: │ toddaustin.io │ 0  │ fork │ 2506 │ online │ 0       │ 55s    │ 0%  │ 41.4 MB   │ austintoddj │ disabled │
Sep 19 08:32:03 whispering-ocean pm2[2645]: └───────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴─────────────┴──────────┘
Sep 19 08:32:03 whispering-ocean pm2[2645]:  Use `pm2 show <id|name>` to get more details about an app
Sep 19 08:32:03 whispering-ocean systemd[1]: Started PM2 process manager.
nivoset commented 7 years ago

i see it, but it isn't reading the dump file

Sep 21 12:17:44 dest-dispatch pm2[1100]: [PM2] Restoring processes located in /home/dd/.pm2/dump.pm2 Sep 21 12:17:44 dest-dispatch pm2[1100]: [PM2][ERROR] Failed to read dump file in /home/dd/.pm2/dump.pm2 Sep 21 12:17:44 dest-dispatch pm2[1100]: [PM2] Restoring processes located in /home/dd/.pm2/dump.pm2 Sep 21 12:17:44 dest-dispatch pm2[1100]: [PM2][ERROR] Failed to read dump file in /home/dd/.pm2/dump.pm2.bak Sep 21 12:17:44 dest-dispatch pm2[1100]: [PM2][ERROR] No processes saved; DUMP file doesn't exist Sep 21 12:17:44 dest-dispatch pm2[1100]: ┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐ Sep 21 12:17:44 dest-dispatch pm2[1100]: │ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │ Sep 21 12:17:44 dest-dispatch pm2[1100]: └──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘ Sep 21 12:17:44 dest-dispatch pm2[1100]: Use pm2 show <id|name> to get more details about an app

nivoset commented 7 years ago

the file exsists, if i do pm2 resurrect it starts it up

vmarchaud commented 7 years ago

Will checkout if there is anything wrong with the code tomorrow

austintoddj commented 7 years ago

Any further clarification on this @vmarchaud?

vanessasoutoc commented 6 years ago

Me too have this is problem

vlntb commented 6 years ago

I'm having exactly the same error as described, with same log output [PM2][ERROR] Failed to read dump file ....

ghost commented 6 years ago

Me too have some problem!

ghost commented 6 years ago

os Ubuntu 16.04 After system startup pm2-web.service is active and I if I type pm2 list I see my app is working. But app not respond!!! If I type "$ sudo systemctl reload pm2-web" - all works!

alexwendte commented 6 years ago

For anyone still having this problem, I had a very similar problem, and #3175 seemed to fix it for me. I had two different instances of pm2 messing each other up I believe.

darkn3rd commented 6 years ago

This is broken for me, as per instructions. Surprised it has been a year, and this is not fixed. I only have one pm2 created.

/usr/lib/node_modules/pm2/bin/pm2 -v
3.0.0
npm -v
5.6.0
node -v
v10.1.0

The file generated is this:

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=webapp
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/opt/webapp/.pm2
PIDFile=/opt/webapp/.pm2/pm2.pid

ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

I noticed that the file is copied there directly, while everything else symlinks it:

# ls -la /etc/systemd/system/
total 56
drwxr-xr-x 13 root root 4096 Jul  5 21:40 .
drwxr-xr-x  5 root root 4096 Mar 25 05:26 ..
lrwxrwxrwx  1 root root    9 Mar 25 05:24 apt-daily.service -> /dev/null
lrwxrwxrwx  1 root root    9 Mar 25 05:24 apt-daily-upgrade.service -> /dev/null
drwxr-xr-x  2 root root 4096 Mar 25 05:18 default.target.wants
drwxr-xr-x  2 root root 4096 Mar 25 05:21 final.target.wants
drwxr-xr-x  2 root root 4096 Mar 25 05:18 getty.target.wants
drwxr-xr-x  2 root root 4096 Mar 25 05:20 graphical.target.wants
lrwxrwxrwx  1 root root   38 Mar 25 05:21 iscsi.service -> /lib/systemd/system/open-iscsi.service
drwxr-xr-x  2 root root 4096 Jul  5 21:40 multi-user.target.wants
drwxr-xr-x  2 root root 4096 Mar 25 05:18 network-online.target.wants
drwxr-xr-x  2 root root 4096 Mar 25 05:21 paths.target.wants
-rw-r--r--  1 root root  546 Jul  5 21:40 pm2-webapp.service
drwxr-xr-x  2 root root 4096 Mar 25 05:22 remote-fs.target.wants
drwxr-xr-x  2 root root 4096 Mar 25 05:22 sockets.target.wants
lrwxrwxrwx  1 root root   31 Mar 25 05:22 sshd.service -> /lib/systemd/system/ssh.service
drwxr-xr-x  2 root root 4096 Mar 25 05:31 sysinit.target.wants
lrwxrwxrwx  1 root root   35 Mar 25 05:18 syslog.service -> /lib/systemd/system/rsyslog.service
drwxr-xr-x  2 root root 4096 Mar 25 05:29 timers.target.wants
minotaurrr commented 6 years ago

same issue here, still no fix?

davidfaj commented 5 years ago

This solved the problem for me: https://stackoverflow.com/questions/34821063/pm2-startup-not-starting-up-on-ubuntu#54193188 (I think it could be more clear in the pm2 startup instructions that pm2 start <app.js> should be started through systemctl)

The solution for me was restarting pm2 with systemctl: systemctl reload-or-restart pm2-root

When I setup my Ubuntu 18.04 server on the first time, I ran pm2 start app.js to start my app. Then when I tried to run pm2 startup + pm2 save to restart the app on boot, this seemed to be not working, as by running systemctl list-units, pm2 didn’t show up in the services list. Even though the app was running (pm2 list confirmed that). So I ran systemctl list-units -all, and pm2 showed as “inactive” and “dead”.

So I did:

systemctl status pm2-root (just to confirm that it was "inactive"/"dead")

systemctl reload-or-restart pm2-root (to restart pm2 through systemctl)

systemctl status pm2-root (to confirm that pm2 now was “active”/“running”)

Then I also ran systemctl enable pm2-root, to enable PM2 to run on startup (not sure if necessary)

And pm2 startup + pm2 save again (for the start on boot)

OBS.: I used pm2-root in my commands, as I was running pm2 with root user, but you should replace for your user (pm2-<USER>), if needed.

danilo-jlle commented 5 years ago

Same problem here solved by commenting out "Type=forking" on service unit.

geochanto commented 5 years ago

In my case this happened after I shut down and restarted my ubuntu server. Starting/stopping/restarting pm2 was not working - I noticed that pm2 was running the script from the wrong exec cwd. You can see this if you run pm2 show <processId>. It should be in the same directory as your package.json containing your start script.

I ran pm2 kill to kill off the process, then cd into the directory with your package.json and run your start command (in my case pm2 start npm -- run start).

mauriciogior commented 4 years ago

On my system it was returning status=205/LIMITS, meaning that the process was exceeding the limits of my system. I had to remove the following lines to make it work:

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity