wrong data in pm2 jlist property versioning

[KaiSchwarz-cnic]

I called pm2 jlist. In data property pm2_env.versioning I get data to another repository. In our case: our webhook repository which is used to trigger the deployment process and thus restarts the pm2 apps. For each of our pm2 apps we have this data covered in the above data property which doesn't make a lot sense. The purpose of that object data is imo to cover the versioning data of the app repository itself. I hope I am right here.

Our webhook creates always a new child process and executes the deployment script of the appropriate app. the child process always inherits the environment of the parent process.

===============================================================================
--- PM2 REPORT (Mon Nov 13 2017 12:40:25 GMT+0000 (UTC)) ----------------------
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 2.6.1
node version         : 6.10.2
node path            : /usr/lib/node_modules/pm2/bin/pm2
argv                 : /usr/bin/nodejs,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : node
uid                  : 1000
gid                  : 1000
uptime               : 36450min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 2.6.1
node version         : 6.11.5
node path            : /usr/bin/pm2
argv                 : /home/node/.nvm/versions/node/v6.11.5/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : node
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz
cpus nb              : 8
freemem              : 532135936
totalmem             : 33694265344
home                 : /home/node
===============================================================================
--- PM2 list -----------------------------------------------
┌───────────┬────┬─────────┬───────┬────────┬─────────┬────────┬─────┬────────────┬──────┬──────────┐
│ App name  │ id │ mode    │ pid   │ status │ restart │ uptime │ cpu │ mem        │ user │ watching │
├───────────┼────┼─────────┼───────┼────────┼─────────┼────────┼─────┼────────────┼──────┼──────────┤
│ CP3PRD    │ 6  │ cluster │ 26650 │ online │ 68      │ 2D     │ 0%  │ 421.4 MB   │ node │ disabled │
│ CP3PRD    │ 3  │ cluster │ 26161 │ online │ 68      │ 2D     │ 0%  │ 337.4 MB   │ node │ disabled │
│ CP3PRD    │ 4  │ cluster │ 26406 │ online │ 68      │ 2D     │ 0%  │ 407.8 MB   │ node │ disabled │
│ CP3PRD    │ 5  │ cluster │ 26615 │ online │ 68      │ 2D     │ 0%  │ 407.9 MB   │ node │ disabled │
│ CP3PRD    │ 2  │ cluster │ 26371 │ online │ 68      │ 2D     │ 0%  │ 326.8 MB   │ node │ disabled │
│ CP3PRD    │ 7  │ cluster │ 27008 │ online │ 67      │ 2D     │ 0%  │ 392.7 MB   │ node │ disabled │
│ CP3PRD    │ 8  │ cluster │ 27042 │ online │ 66      │ 2D     │ 0%  │ 441.3 MB   │ node │ disabled │
│ CP3PRD    │ 9  │ cluster │ 27076 │ online │ 66      │ 2D     │ 1%  │ 404.1 MB   │ node │ disabled │
│ PORT80APP │ 15 │ cluster │ 29843 │ online │ 0       │ 4h     │ 0%  │ 39.0 MB    │ node │ disabled │
│ TLDINFO   │ 0  │ cluster │ 27407 │ online │ 9       │ 3D     │ 0%  │ 101.4 MB   │ node │ disabled │
│ WEBHOOK   │ 16 │ cluster │ 560   │ online │ 0       │ 3h     │ 0%  │ 36.5 MB    │ node │ disabled │
└───────────┴────┴─────────┴───────┴────────┴─────────┴────────┴─────┴────────────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/home/node/.pm2/pm2.log last 20 lines:
PM2        | 2017-11-13 08:37:16: 14
PM2        | 2017-11-13 08:37:16: Stopping app:PORT80APP id:14
PM2        | 2017-11-13 08:37:16: App name:PORT80APP id:14 disconnected
PM2        | 2017-11-13 08:37:16: App [PORT80APP] with id [14] and pid [29094], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:37:16: pid=29094 msg=process killed
PM2        | 2017-11-13 08:37:16: Starting execution sequence in -cluster mode- for app name:PORT80APP id:15
PM2        | 2017-11-13 08:37:26: App name:PORT80APP id:15 online
PM2        | 2017-11-13 08:39:07: Stopping app:WEBHOOK id:11
PM2        | 2017-11-13 08:39:07: App name:WEBHOOK id:11 disconnected
PM2        | 2017-11-13 08:39:07: App [WEBHOOK] with id [11] and pid [21068], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:39:07: pid=21068 msg=process killed
PM2        | 2017-11-13 08:39:07: Starting execution sequence in -cluster mode- for app name:WEBHOOK id:11
PM2        | 2017-11-13 08:39:17: App name:WEBHOOK id:11 online
PM2        | 2017-11-13 08:39:49: 11
PM2        | 2017-11-13 08:39:49: Stopping app:WEBHOOK id:11
PM2        | 2017-11-13 08:39:49: App name:WEBHOOK id:11 disconnected
PM2        | 2017-11-13 08:39:49: App [WEBHOOK] with id [11] and pid [31075], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:39:49: pid=31075 msg=process killed
PM2        | 2017-11-13 08:41:53: Starting execution sequence in -cluster mode- for app name:WEBHOOK id:16
PM2        | 2017-11-13 08:41:53: App name:WEBHOOK id:16 online

Have we to change anything in our process or is this indeed a bug?

[vmarchaud]

You must set the cwd correctly when launching your application, could you share your ecosystem ?

[KaiSchwarz-cnic]

So this is the configuration for one of our apps used in the webhook to redeploy:

CP3_PRD: {
    branches: ["master"],
    command: "./deploy.sh master CP3_PRD",
    events: ["push"],
    exit: false,
    group: "node",
    options: {
      cwd: "/opt/fnode",
      env: {
        NODE_ENV: "production",
      },
    },
    token: ".............................. ;o) ................................",
    user: "node",
  },

cwd points to the directory where the repository of that appropriate app got cloned to. That should be correct then for my understanding.

in the webhook logic, we set the process environment then as follows:

 tpl.options.env = Object.assign({}, process.env, tpl.options.env);//tpl corresponds to the config above

and finally execute the command through child_process.exec:

cp.exec(tpl.command, tpl.options, (err, stdout, stderr) => {
    //handling of the callback results        
});

Basically, I don't see there an issue? I mean, everything is restarting and working fine, just not these data fields.

[KaiSchwarz-cnic]

Ehm.. can pm2 reload --update-env be the reason in the appropriate deployment script of the apps?

[KaiSchwarz-cnic]

No, not the case. I've started our port 80 to 443 forwarding app manually: all data in these fields looked fine. triggered redeployment via webhook app -> data broken. --update-env is not part of the deployment script of that port forwarding app.

[KaiSchwarz-cnic]

Do you need anything else @vmarchaud ?

[KaiSchwarz-cnic]

Looks like tpl.options.env = Object.assign({}, process.env, tpl.options.env); is the reason for this. Where the process environment of the existing process (parent) is getting set for the child process, but overwritten by the template env settings. This got introduces as we had problems one day when restarting our apps, I'll keep this issue updated. Let me see if I can solve this. If you have any suggestion, let me know @vmarchaud.

[KaiSchwarz-cnic]

introduced because of: https://github.com/Unitech/pm2/issues/2756

If I change tpl.options.env = Object.assign({}, process.env, tpl.options.env); to tpl.options.env = Object.assign({}, tpl.options.env); necessary paths seem to be missing so that deployment process breaks. e.g. ./deploy.sh: line 5: npm: command not found

I am asking myself what you suggest to be the right way here. I really have no clue.

[vmarchaud]

I'm pretty sure --update-env can't update the cwd of your application, did you set the cwd in the pm2 configuration of your application ?

[KaiSchwarz-cnic]

here's the pm2.json we use for our main app:

{
  "apps": [{
    "name": "CP3PRD",
    "script": "fnode.js",
    "args": ["CP3_PRD"],
    "error_file": "/var/log/frontend/CP3/err.log",
    "out_file": "/var/log/frontend/CP3/out.log",
    "instances": 0,
    "cwd": "/opt/fnode",
    "env": {
      "NODE_ENV": "production"
    },
    "exec_mode": "cluster",
    "exec_interpreter": "node",
    "merge_logs": true,
    "log_date_format": "YYYY-MM-DD HH:mm:ss Z",
    "autorestart": true,
    "restart_delay": 10000,
    "watch": false,
    "wait_ready": true,
    "listen_timeout": 10000,
    "kill_timeout": 20000
  }]
}

That's the config we use to start the application. The options part is reflected again accordingly in the webhook configuration which we use as mentioned:

options: {
      cwd: "/opt/fnode",
      env: {
        NODE_ENV: "production",
      },
    },

So I cannot imagine cwd is the reason behind.

[KaiSchwarz-cnic]

I have the impression that pm2 itself reuses some options settings (which came from parent process and were generated for the webhook app) at least for the versioning part later on. Deployment itself seems to work correctly.

So as, settings from the parent process seem to be required to have paths to npm, et al set correctly, maybe further things I am not aware of currently - I am asking myself which object keys from the options object need to be cleaned up to get this working?

[KaiSchwarz-cnic]

any further suggestions?

[vmarchaud]

No suggestion in this case it will need to be inspected by someone (feel free to do it and open up a PR to fix any issue)

[KaiSchwarz-cnic]

Thanks, as this goes very deep into pm2 architecture, I don't think I am able to cover this (and yes, I am very busy in addition). If I should get the time, yes I'll have an eye on it, but don't expect this to happen.

[KaiSchwarz-cnic]

Are there news on this issue? I would also recommend to get the "S: Open for PR" flag removed, as I won't get that time in near future.

[wallet77]

@papakai this flag is not only for people who open issue, but for all people who want to contribute.

[KaiSchwarz-cnic]

@wallet77 thanks, good to know. Still leaves me the question when / if this might get reviewed. Or at least when to get some informations which environment data to explicitely drop to avoid this data issue.

[wallet77]

@papakai we have a lot of work with the next release. I think we can check this one (and many other issues linked to versioning) after the next release.