Unitech / pm2

Node.js Production Process Manager with a built-in Load Balancer.
https://pm2.keymetrics.io/docs/usage/quick-start/
Other
41.48k stars 2.62k forks source link

wrong data in pm2 jlist property versioning #3276

Open KaiSchwarz-cnic opened 6 years ago

KaiSchwarz-cnic commented 6 years ago

I called pm2 jlist. In data property pm2_env.versioning I get data to another repository. In our case: our webhook repository which is used to trigger the deployment process and thus restarts the pm2 apps. For each of our pm2 apps we have this data covered in the above data property which doesn't make a lot sense. The purpose of that object data is imo to cover the versioning data of the app repository itself. I hope I am right here.

Our webhook creates always a new child process and executes the deployment script of the appropriate app. the child process always inherits the environment of the parent process.

===============================================================================
--- PM2 REPORT (Mon Nov 13 2017 12:40:25 GMT+0000 (UTC)) ----------------------
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 2.6.1
node version         : 6.10.2
node path            : /usr/lib/node_modules/pm2/bin/pm2
argv                 : /usr/bin/nodejs,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : node
uid                  : 1000
gid                  : 1000
uptime               : 36450min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 2.6.1
node version         : 6.11.5
node path            : /usr/bin/pm2
argv                 : /home/node/.nvm/versions/node/v6.11.5/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : node
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz
cpus nb              : 8
freemem              : 532135936
totalmem             : 33694265344
home                 : /home/node
===============================================================================
--- PM2 list -----------------------------------------------
┌───────────┬────┬─────────┬───────┬────────┬─────────┬────────┬─────┬────────────┬──────┬──────────┐
│ App name  │ id │ mode    │ pid   │ status │ restart │ uptime │ cpu │ mem        │ user │ watching │
├───────────┼────┼─────────┼───────┼────────┼─────────┼────────┼─────┼────────────┼──────┼──────────┤
│ CP3PRD    │ 6  │ cluster │ 26650 │ online │ 68      │ 2D     │ 0%  │ 421.4 MB   │ node │ disabled │
│ CP3PRD    │ 3  │ cluster │ 26161 │ online │ 68      │ 2D     │ 0%  │ 337.4 MB   │ node │ disabled │
│ CP3PRD    │ 4  │ cluster │ 26406 │ online │ 68      │ 2D     │ 0%  │ 407.8 MB   │ node │ disabled │
│ CP3PRD    │ 5  │ cluster │ 26615 │ online │ 68      │ 2D     │ 0%  │ 407.9 MB   │ node │ disabled │
│ CP3PRD    │ 2  │ cluster │ 26371 │ online │ 68      │ 2D     │ 0%  │ 326.8 MB   │ node │ disabled │
│ CP3PRD    │ 7  │ cluster │ 27008 │ online │ 67      │ 2D     │ 0%  │ 392.7 MB   │ node │ disabled │
│ CP3PRD    │ 8  │ cluster │ 27042 │ online │ 66      │ 2D     │ 0%  │ 441.3 MB   │ node │ disabled │
│ CP3PRD    │ 9  │ cluster │ 27076 │ online │ 66      │ 2D     │ 1%  │ 404.1 MB   │ node │ disabled │
│ PORT80APP │ 15 │ cluster │ 29843 │ online │ 0       │ 4h     │ 0%  │ 39.0 MB    │ node │ disabled │
│ TLDINFO   │ 0  │ cluster │ 27407 │ online │ 9       │ 3D     │ 0%  │ 101.4 MB   │ node │ disabled │
│ WEBHOOK   │ 16 │ cluster │ 560   │ online │ 0       │ 3h     │ 0%  │ 36.5 MB    │ node │ disabled │
└───────────┴────┴─────────┴───────┴────────┴─────────┴────────┴─────┴────────────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/home/node/.pm2/pm2.log last 20 lines:
PM2        | 2017-11-13 08:37:16: 14
PM2        | 2017-11-13 08:37:16: Stopping app:PORT80APP id:14
PM2        | 2017-11-13 08:37:16: App name:PORT80APP id:14 disconnected
PM2        | 2017-11-13 08:37:16: App [PORT80APP] with id [14] and pid [29094], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:37:16: pid=29094 msg=process killed
PM2        | 2017-11-13 08:37:16: Starting execution sequence in -cluster mode- for app name:PORT80APP id:15
PM2        | 2017-11-13 08:37:26: App name:PORT80APP id:15 online
PM2        | 2017-11-13 08:39:07: Stopping app:WEBHOOK id:11
PM2        | 2017-11-13 08:39:07: App name:WEBHOOK id:11 disconnected
PM2        | 2017-11-13 08:39:07: App [WEBHOOK] with id [11] and pid [21068], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:39:07: pid=21068 msg=process killed
PM2        | 2017-11-13 08:39:07: Starting execution sequence in -cluster mode- for app name:WEBHOOK id:11
PM2        | 2017-11-13 08:39:17: App name:WEBHOOK id:11 online
PM2        | 2017-11-13 08:39:49: 11
PM2        | 2017-11-13 08:39:49: Stopping app:WEBHOOK id:11
PM2        | 2017-11-13 08:39:49: App name:WEBHOOK id:11 disconnected
PM2        | 2017-11-13 08:39:49: App [WEBHOOK] with id [11] and pid [31075], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:39:49: pid=31075 msg=process killed
PM2        | 2017-11-13 08:41:53: Starting execution sequence in -cluster mode- for app name:WEBHOOK id:16
PM2        | 2017-11-13 08:41:53: App name:WEBHOOK id:16 online

Have we to change anything in our process or is this indeed a bug?

vmarchaud commented 6 years ago

You must set the cwd correctly when launching your application, could you share your ecosystem ?

KaiSchwarz-cnic commented 6 years ago

So this is the configuration for one of our apps used in the webhook to redeploy:

CP3_PRD: {
    branches: ["master"],
    command: "./deploy.sh master CP3_PRD",
    events: ["push"],
    exit: false,
    group: "node",
    options: {
      cwd: "/opt/fnode",
      env: {
        NODE_ENV: "production",
      },
    },
    token: ".............................. ;o) ................................",
    user: "node",
  },

cwd points to the directory where the repository of that appropriate app got cloned to. That should be correct then for my understanding.

in the webhook logic, we set the process environment then as follows:

 tpl.options.env = Object.assign({}, process.env, tpl.options.env);//tpl corresponds to the config above

and finally execute the command through child_process.exec:

cp.exec(tpl.command, tpl.options, (err, stdout, stderr) => {
    //handling of the callback results        
});

Basically, I don't see there an issue? I mean, everything is restarting and working fine, just not these data fields.

KaiSchwarz-cnic commented 6 years ago

Ehm.. can pm2 reload --update-env be the reason in the appropriate deployment script of the apps?

KaiSchwarz-cnic commented 6 years ago

No, not the case. I've started our port 80 to 443 forwarding app manually: all data in these fields looked fine. triggered redeployment via webhook app -> data broken. --update-env is not part of the deployment script of that port forwarding app.

KaiSchwarz-cnic commented 6 years ago

Do you need anything else @vmarchaud ?

KaiSchwarz-cnic commented 6 years ago

Looks like tpl.options.env = Object.assign({}, process.env, tpl.options.env); is the reason for this. Where the process environment of the existing process (parent) is getting set for the child process, but overwritten by the template env settings. This got introduces as we had problems one day when restarting our apps, I'll keep this issue updated. Let me see if I can solve this. If you have any suggestion, let me know @vmarchaud.

KaiSchwarz-cnic commented 6 years ago

introduced because of: https://github.com/Unitech/pm2/issues/2756

If I change tpl.options.env = Object.assign({}, process.env, tpl.options.env); to tpl.options.env = Object.assign({}, tpl.options.env); necessary paths seem to be missing so that deployment process breaks. e.g. ./deploy.sh: line 5: npm: command not found

I am asking myself what you suggest to be the right way here. I really have no clue.

vmarchaud commented 6 years ago

I'm pretty sure --update-env can't update the cwd of your application, did you set the cwd in the pm2 configuration of your application ?

KaiSchwarz-cnic commented 6 years ago

here's the pm2.json we use for our main app:

{
  "apps": [{
    "name": "CP3PRD",
    "script": "fnode.js",
    "args": ["CP3_PRD"],
    "error_file": "/var/log/frontend/CP3/err.log",
    "out_file": "/var/log/frontend/CP3/out.log",
    "instances": 0,
    "cwd": "/opt/fnode",
    "env": {
      "NODE_ENV": "production"
    },
    "exec_mode": "cluster",
    "exec_interpreter": "node",
    "merge_logs": true,
    "log_date_format": "YYYY-MM-DD HH:mm:ss Z",
    "autorestart": true,
    "restart_delay": 10000,
    "watch": false,
    "wait_ready": true,
    "listen_timeout": 10000,
    "kill_timeout": 20000
  }]
}

That's the config we use to start the application. The options part is reflected again accordingly in the webhook configuration which we use as mentioned:

options: {
      cwd: "/opt/fnode",
      env: {
        NODE_ENV: "production",
      },
    },

So I cannot imagine cwd is the reason behind.

KaiSchwarz-cnic commented 6 years ago

I have the impression that pm2 itself reuses some options settings (which came from parent process and were generated for the webhook app) at least for the versioning part later on. Deployment itself seems to work correctly.

So as, settings from the parent process seem to be required to have paths to npm, et al set correctly, maybe further things I am not aware of currently - I am asking myself which object keys from the options object need to be cleaned up to get this working?

KaiSchwarz-cnic commented 6 years ago

any further suggestions?

vmarchaud commented 6 years ago

No suggestion in this case it will need to be inspected by someone (feel free to do it and open up a PR to fix any issue)

KaiSchwarz-cnic commented 6 years ago

Thanks, as this goes very deep into pm2 architecture, I don't think I am able to cover this (and yes, I am very busy in addition). If I should get the time, yes I'll have an eye on it, but don't expect this to happen.

KaiSchwarz-cnic commented 6 years ago

Are there news on this issue? I would also recommend to get the "S: Open for PR" flag removed, as I won't get that time in near future.

wallet77 commented 6 years ago

@papakai this flag is not only for people who open issue, but for all people who want to contribute.

KaiSchwarz-cnic commented 6 years ago

@wallet77 thanks, good to know. Still leaves me the question when / if this might get reviewed. Or at least when to get some informations which environment data to explicitely drop to avoid this data issue.

wallet77 commented 6 years ago

@papakai we have a lot of work with the next release. I think we can check this one (and many other issues linked to versioning) after the next release.