search

Unitech / pm2

Node.js Production Process Manager with a built-in Load Balancer.
https://pm2.keymetrics.io/docs/usage/quick-start/
Other
41.34k stars 2.61k forks source link

Apps do not restart on Amazon Linux after pm2 startup command #2574

Closed tademianhess closed 7 years ago

tademianhess commented 7 years ago

If reporting a bug, please use the following template.

Expected behaviour

After running 1) pm2 startup and specifying "amazon" as the platform, 2) starting nodejs apps,and 3) running pm2 save, all apps should restart after server reboot. ...

Actual behaviour

When running sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup amazon -u ec2-user --hp /home/ec2-user get error /usr/lib/node_modules/pm2/lib/API/Startup.js:214 throw new Error('Unknown platform / init system name'); When run pm2 startup, system suggests specifying platform as 'rcd'. However, after running with that platform, running pm2 save, and rebooting, no apps restart. ...

Steps to reproduce

Spin up new Amazon Linux server and install pm2 (v 2.2.1). Start nodejs apps. SSH to server as ec2-user

sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup amazon -u ec2-user --hp /home/ec2-user => Results in error.

Rerun with suggested platform of rcd: sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup rcd -u ec2-user --hp /home/ec2-user`

Start all nodejs applications Run pm2 save Exit from shell Reboot server No apps restart ... ...

Software versions used

OS         : Amazon Linux
node.js    : 6.9.1
PM2        : 2.2.1

PM2 Log output

Use the command: tail --lines 50 ~/.pm2/pm2.log 2016-12-14 22:30:43: =============================================================================== 2016-12-14 22:30:43: --- New PM2 Daemon started ---------------------------------------------------- 2016-12-14 22:30:43: Time : Wed Dec 14 2016 22:30:43 GMT+0000 (UTC) 2016-12-14 22:30:43: PM2 version : 2.2.1 2016-12-14 22:30:43: Node.js version : 6.9.1 2016-12-14 22:30:43: Current arch : x64 2016-12-14 22:30:43: PM2 home : /home/ec2-user/.pm2 2016-12-14 22:30:43: PM2 PID file : /home/ec2-user/.pm2/pm2.pid 2016-12-14 22:30:43: RPC socket file : /home/ec2-user/.pm2/rpc.sock 2016-12-14 22:30:43: BUS socket file : /home/ec2-user/.pm2/pub.sock 2016-12-14 22:30:43: Application log path : /home/ec2-user/.pm2/logs 2016-12-14 22:30:43: Process dump file : /home/ec2-user/.pm2/dump.pm2 2016-12-14 22:30:43: Concurrent actions : 2 2016-12-14 22:30:43: SIGTERM timeout : 1600 2016-12-14 22:30:43: =============================================================================== 2016-12-14 22:30:50: [Watch] Start watching 0 2016-12-14 22:30:50: Starting execution sequence in -fork mode- for app name:oqc id:0 2016-12-14 22:30:50: App name:oqc id:0 online 2016-12-14 22:41:08: pm2 has been killed by signal, dumping process list before exit... 2016-12-14 22:41:08: Deleting process 0 2016-12-14 22:41:08: Stopping app:oqc id:0 2016-12-14 22:41:08: App [oqc] with id [0] and pid [2785], exited with code [0] via signal [SIGTERM] 2016-12-14 22:41:08: pid=2785 msg=process killed 2016-12-14 22:41:08: [PM2] Exited peacefully

...

Dump.pm2 looks like this:

[ec2-user@ip-10-0-0-239 ~]$ cat /home/ec2-user/.pm2/dump.pm2  | more
[
  {
    "env_integration": {
      "NODE_ENV": "etl"
    },
    "env_production": {
      "NODE_ENV": "production"
    },
    "exec_mode": "fork_mode",
    "watch": true,
    "treekill": true,
    "autorestart": true,
    "automation": true,
    "pmx": true,
    "vizion": true,
    "cwd": "/var/opt/oqc/current/webapp",
    "name": "oqc",
    "node_args": [],
    "pm_exec_path": "/var/opt/oqc/current/webapp/bin/www",
    "env": {
      "PM2_JSON_PROCESSING": "true",
      "PM2_INTERACTOR_PROCESSING": "true",
      "PM2_USAGE": "CLI",
      "_": "/usr/bin/pm2",
      "LESS_TERMCAP_se": "\u001b[0m",
      "LESSOPEN": "||/usr/bin/lesspipe.sh %s",
      "SSH_CONNECTION": "199.72.235.66 54073 10.0.0.239 22",
      "AWS_ELB_HOME": "/opt/aws/apitools/elb",
      "LOGNAME": "ec2-user",
      "AWS_AUTO_SCALING_HOME": "/opt/aws/apitools/as",
      "AWS_PATH": "/opt/aws",
      "HOME": "/home/ec2-user",
      "SHLVL": "1",
      "AWS_CLOUDWATCH_HOME": "/opt/aws/apitools/mon",
      "LANG": "en_US.UTF-8",
      "JAVA_HOME": "/usr/lib/jvm/jre",
      "PWD": "/var/opt/oqc/current/webapp",
      "PATH": "/usr/local/bin:/bin:/usr/bin:/opt/aws/bin",
      "MAIL": "/var/mail/ec2-user",
      "LESS_TERMCAP_us": "\u001b[04;38;5;111m",
      "EC2_HOME": "/opt/aws/apitools/ec2",
      "USER": "ec2-user",
      "LESS_TERMCAP_ue": "\u001b[0m",
      "SSH_CLIENT": "199.72.235.66 54073 22",
      "EC2_AMITOOL_HOME": "/opt/aws/amitools/ec2",
      "SHELL": "/bin/bash",
      "LESS_TERMCAP_me": "\u001b[0m",
      "LESS_TERMCAP_md": "\u001b[01;38;5;208m",
      "LESS_TERMCAP_mb": "\u001b[01;31m",
      "PM2_HOME": "/home/ec2-user/.pm2",
      "oqc": "{}",
      "NODE_ENV": "production"
    },
    "pm_cwd": "/var/opt/oqc/current/webapp",
    "exec_interpreter": "node",
    "pm_out_log_path": "/home/ec2-user/.pm2/logs/oqc-out-0.log",
    "pm_err_log_path": "/home/ec2-user/.pm2/logs/oqc-error-0.log",
    "pm_pid_path": "/home/ec2-user/.pm2/pids/oqc-0.pid",
    "NODE_APP_INSTANCE": 0,
    "vizion_running": false,
    "PM2_JSON_PROCESSING": "true",
    "PM2_INTERACTOR_PROCESSING": "true",
    "PM2_USAGE": "CLI",
    "_": "/usr/bin/pm2",
    "LESS_TERMCAP_se": "\u001b[0m",
    "LESSOPEN": "||/usr/bin/lesspipe.sh %s",
    "SSH_CONNECTION": "199.72.235.66 54073 10.0.0.239 22",
    "AWS_ELB_HOME": "/opt/aws/apitools/elb",
    "LOGNAME": "ec2-user",
    "AWS_AUTO_SCALING_HOME": "/opt/aws/apitools/as",
    "AWS_PATH": "/opt/aws",
    "HOME": "/home/ec2-user",
    "SHLVL": "1",
    "AWS_CLOUDWATCH_HOME": "/opt/aws/apitools/mon",
    "LANG": "en_US.UTF-8",
    "JAVA_HOME": "/usr/lib/jvm/jre",
    "PWD": "/var/opt/oqc/current/webapp",
    "PATH": "/usr/local/bin:/bin:/usr/bin:/opt/aws/bin",
    "MAIL": "/var/mail/ec2-user",
    "LESS_TERMCAP_us": "\u001b[04;38;5;111m",
    "EC2_HOME": "/opt/aws/apitools/ec2",
    "USER": "ec2-user",
    "LESS_TERMCAP_ue": "\u001b[0m",
    "SSH_CLIENT": "199.72.235.66 54073 22",
    "EC2_AMITOOL_HOME": "/opt/aws/amitools/ec2",
    "SHELL": "/bin/bash",
    "LESS_TERMCAP_me": "\u001b[0m",
    "LESS_TERMCAP_md": "\u001b[01;38;5;208m",
    "LESS_TERMCAP_mb": "\u001b[01;31m",
    "PM2_HOME": "/home/ec2-user/.pm2",
    "oqc": "{}",
    "NODE_ENV": "production",
    "status": "online",
    "pm_uptime": 1481755466342,
    "axm_actions": [],
    "axm_monitor": {
      "Loop delay": {
        "alert": {},
        "agg_type": "avg",
        "value": "84.8ms"
      }
    },
    "axm_options": {
      "default_actions": true,
      "http": false,
      "http_latency": 200,
      "http_code": 500,
      "ignore_routes": [],
      "profiling": true,
      "errors": true,
      "alert_enabled": true,
      "custom_probes": true,
      "network": false,
      "ports": false,
      "module_conf": {},
      "module_name": "oqc",
      "module_version": "2.2.1",
      "pmx_version": "0.6.8",
      "error": true
    },
    "axm_dynamic": {},
    "created_at": 1481755466342,
    "restart_time": 0,
    "unstable_restarts": 0,
    "node_version": "6.9.1",
    "versioning": {
      "type": "git",
      "url": "[git@GITREPO]",
      "revision": "08235a2796d07015aa99b8bf5a364ad142217bd8",
      "update_time": "2016-12-14T22:44:26.457Z",
      "comment": "Add production environment",
      "unstaged": false,
      "branch": "master",
      "remotes": [
        "origin"
      ],
      "remote": "origin",
      "branch_exists_on_remote": true,
      "ahead": false,
      "next_rev": null,
      "prev_rev": "285aed29c0f161ff31522bd7cbf131ecabe2824c",
      "repo_path": "/var/opt/oqc/current"
    }
  }
]

cf CONTRIBUTING.md for more informations

vmarchaud commented 7 years ago

Can you give me the name of the image that you used to spawn this system ?

tademianhess commented 7 years ago

AMI ID: amzn-ami-hvm-2016.09.0.20161028-x86_64-gp2 (ami-b73b63a0) AMI Description: Amazon Linux AMI 2016.09.0.20161028 x86_64 HVM GP2

vmarchaud commented 7 years ago

Spawned an instance 1 hour ago and reproduced the problem, we will publish a fix in the next hours.

Unitech commented 7 years ago

Fixed. Please upgrade to PM2 2.2.2:

$ npm install pm2 -g
$ pm2 update
$ pm2 startup amazon
MrSwitch commented 7 years ago

Thanks @Unitech. My issue was different, and your solution only partly solved it. The platform my company use is defined as "Linux" rather than "Amazon"

I hope the following might help some one in similar situation.

>>>> In-memory PM2 is out-of-date, do:
>>>> $ pm2 update
In memory PM2 version: 2.0.18
Local PM2 version: 2.2.2

After updating pm2 update. The above warning message i get when accessing pm2 disappears, however running pm2 startup linux -u user --hp /home/app still broke with 

Error: Unknown platform / init system name

Changing this to something other than Linux, aka systemd for Debian seems to have done the trick. reference

e.g. pm2 startup systemd -u user --hp /home/app

vmarchaud commented 7 years ago

Don't try to put an plateform, just run pm2 startup, pm2 will determine the system you use (Amazon linux use SystemV), we tested under a fresh vm instance on amazon and it works great for us.

oleksiypavlenko commented 7 years ago

@vmarchaud is this a correct way for setting up the auto startup in bash script?

export NODE_ENV=dev NODE_ENV=dev sudo -E pm2 start node/myApp/build/bin/www --name myApp sudo pm2 startup sudo pm2 save

vmarchaud commented 7 years ago

@oleksiypavlenko If your aim is to setup is as root user, yes.

oleksiypavlenko commented 7 years ago

@vmarchaud it is indeed, thank you. I ran into a problem running application as a regular user with freezes when spawning pm2 daemon through script. When I changed all the pm2 to be running under sudo, it looked as if it resolved the problem.

This is how I ran it originally (led to issue with script freezing at the point when pm2 daemon was spawn):

NODE_ENV=dev pm2 start node/myApp/build/bin/www --name myApp sudo pm2 startup ubuntu -u username pm2 save

vmarchaud commented 7 years ago

@oleksiypavlenko Could you try reproducing with DEBUG=pm2:* at the beginning of all command you've done and paste the content in a gist ?