Failed to send PM2 status

[Fchen48]

Hi,

I installed pm2-zabbix on a fresh Ubuntu 16.04 LTS LXC container (privileged), after install I tried pm2-zabbix --monitor using user root and I receive

{"name":"pm2-zabbix","hostname":"app01","pid":26409,"level":50,"event":"PM2ZabbixMonitor#processListSendingError","error":{"killed":false,"code":2,"signal":null,"cmd":"/usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -"},"msg":"Failed to send process list with stats to server: Error: Command failed: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -\n","time":"2018-04-11T18:20:52.257Z","v":0}

The same error appears if I will perform the command with user zabbix.

Next error I receive is if I will start pm2-zabbix using systemd systemctl status pm2-zabbix

● pm2-zabbix.service - PM2 monitor for Zabbix
   Loaded: loaded (/etc/systemd/system/pm2-zabbix.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-04-11 18:29:40 UTC; 4s ago
  Process: 26966 ExecStart=/usr/local/bin/pm2-zabbix --monitor (code=exited, status=217/USER)
 Main PID: 26966 (code=exited, status=217/USER)

Apr 11 18:29:40 app01 systemd[1]: Started PM2 monitor for Zabbix.
Apr 11 18:29:40 app01 systemd[26966]: pm2-zabbix.service: Failed at step USER spawning /usr/local/bin/pm2-zabbix: No such process
Apr 11 18:29:40 app01 systemd[1]: pm2-zabbix.service: Main process exited, code=exited, status=217/USER
Apr 11 18:29:40 app01 systemd[1]: pm2-zabbix.service: Unit entered failed state.
Apr 11 18:29:40 app01 systemd[1]: pm2-zabbix.service: Failed with result 'exit-code'.

PM2 is running under user root and sudoers file is installed.

Sending with zabbix_sender works fine!

Anything else I can check?

[rkaw92]

There seem to be two separate issues here:

1. Server rejecting some keys - the sender process is communicating with the server, but the server refuses to accept some keys for this hostname. Try running LOG_LEVEL=trace pm2-zabbix --monitor and observe what it's sending to the server. Usually, the keys will not exist on the server (missing LLD configuration step) or the hostname will be wrong.
2. systemd unable to launch the executable - normally, npm install -g pm2-zabbix installs packages in the system's global node_modules, but some Node version managers like nvm can override your npm config and cause it to install globals elsewhere. Make sure you perform a real global install, not a local one for your user. If unable, you will need to change the unit file to match your custom paths for Node.js and the script.

[Fchen48]

I rechecked the installation but doesn't found anything unusual.

root@app01 ~ # npm i -g pm2-zabbix
/usr/local/bin/pm2-zabbix -> /usr/local/lib/node_modules/pm2-zabbix/monitor.js
+ pm2-zabbix@0.3.2
updated 2 packages in 3.373s

---

Here is the trace log:

root@app01 ~ # LOG_LEVEL=trace pm2-zabbix --monitor
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":30,"event":"PM2ZabbixMonitor#starting","monitor":true,"msg":"Starting PM2ZabbixMonitor (monitoring enabled: true)","time":"2018-04-12T13:42:52.675Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":20,"event":"PM2ZabbixMonitor#processMapGenerated","processMap":{"pm2-hook-1":{"name":"pm2-hook","status":"online","resources":{"memory":52666368,"cpu":0},"restarts":0},"EPKW-4 {"name":"EPKW","status":"stopped","resources":{"memory":0,"cpu":0},"restarts":4}},"msg":"Process map generated","time":"2018-04-12T13:42:52.690Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":30,"event":"PM2ZabbixMonitor#started","msg":"PM2ZabbixMonitor started","time":"2018-04-12T13:42:52.690Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":10,"event":"PM2ZabbixMonitor#gotProcessMap","processMap":{"pm2-hook-1":{"name":"pm2-hook","status":"online","resources":{"memory":51863552,"cpu":0},"restarts":0},"EPKW-4":{"name":"EPKW","status":"stopped","resources":{"memory":0,"cpu":0},"restarts":4}},"msg":"","time":"2018-04-12T13:43:07.707Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":20,"event":"PM2ZabbixMonitor#sendProcessList","processList":{"pm2.processes[pm2-hook-1,status]":"online","pm2.processes[pm2-hook-1,cpu]":0,"pm2.processes[pm2-hook-1,memory]":51863552,"pm2.processes[pm2-hook-1,restarts]":0,"pm2.processes[EPKW-4,status]":"stopped","pm2.processes[EPKW-4,cpu]":0,"pm2.processes[EPKW-4,memory]":0,"pm2.processes[EPKW-4,restarts]":4},"msg":"Sending process list with stats","time":"2018-04-12T13:43:07.707Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":10,"event":"ZabbixSender#send","bin":"/usr/bin/zabbix_sender","args":["--config","/etc/zabbix/zabbix_agentd.conf","--input-file","-"],"stdIn":"app01 pm2.processes[pm2-hook-1,status] online\napp01 pm2.processes[pm2-hook-1,cpu] 0\napp01 pm2.processes[pm2-hook-1,memory] 51863552\napp01 pm2.processes[pm2-hook-1,restarts] 0\napp01 pm2.processes[EPKW-4,status] stopped\napp01 pm2.processes[EPKW-4,cpu] 0\napp01 pm2.processes[EPKW-4,memory] 0\napp01 pm2.processes[EPKW-4,restarts] 4\n","debug":false,"msg":"Executing: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -","time":"2018-04-12T13:43:07.707Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":50,"event":"PM2ZabbixMonitor#processListSendingError","error":{"killed":false,"code":2,"signal":null,"cmd":"/usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -"},"msg":"Failed to send process list with stats to server: Error: Command failed: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -\n","time":"2018-04-12T13:43:07.715Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":10,"event":"PM2ZabbixMonitor#gotPM2State","processState":{"name":"PM2","status":"online","resources":{"cpu":0.036499984644320554,"memory":60280832},"restarts":0,"pid":8510},"msg":"","time":"2018-04-12T13:43:07.719Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":20,"event":"PM2ZabbixMonitor#sendPM2Status","status":{"pm2.status":"online","pm2.cpu":0.036499984644320554,"pm2.memory":60280832,"pm2.pid":8510},"msg":"Sending PM2 status","time":"2018-04-12T13:43:07.719Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":10,"event":"ZabbixSender#send","bin":"/usr/bin/zabbix_sender","args":["--config","/etc/zabbix/zabbix_agentd.conf","--input-file","-"],"stdIn":"app01 pm2.status online\napp01 pm2.cpu 0\napp01 pm2.memory 60280832\napp01 pm2.pid 8510\n","debug":false,"msg":"Executing: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -","time":"2018-04-12T13:43:07.719Z","v":0}
{"name":"pm2-zabbix","hostname":"app01","pid":13571,"level":50,"event":"PM2ZabbixMonitor#PM2StatusSent","error":{"killed":false,"code":2,"signal":null,"cmd":"/usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -"},"msg":"Failed to send PM2 status: Error: Command failed: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -\n","time":"2018-04-12T13:43:07.721Z","v":0}

[rkaw92]

As for the systemd issue, it seems that status=217/USER is commonly caused by a missing or malformed user specification. Note this section in the default unit file we provide:

[Service]
Type=simple
User=debian

Clearly, if user debian does not exist on your system, the service must fail to start. You will need to change the user to the one that you intend to run your PM2 under.

Now, the trace log shows you which keys are being sent to the Zabbix server. Can you confirm that those keys exist in the Zabbix server management GUI? The line with ZabbixSender#send contains the complete text that is being sent to zabbix_sender in the stdIn field, so you should be able to see whether the host name and the key names match.

[Fchen48]

Hi,

I changed the user to zabbix but no I get another error, pm2-zabbix will create the folder nonexistent under / but have no permission for it.

Apr 14 11:43:00 app01 sudo[3244]:   zabbix : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/pm2-zabbix --discover
Apr 14 11:43:00 app01 sudo[3244]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 11:43:00 app01 sudo[3244]: pam_unix(sudo:session): session closed for user root
Apr 14 11:43:10 app01 systemd[1]: Stopped PM2 monitor for Zabbix.
Apr 14 11:43:19 app01 systemd[1]: Failed to reset devices.list on /system.slice/pm2-zabbix.service: Operation not permitted
Apr 14 11:43:19 app01 systemd[1]: Started PM2 monitor for Zabbix.
Apr 14 11:43:19 app01 pm2-zabbix[3260]: Error: EACCES: permission denied, mkdir '/nonexistent'
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.fs.mkdirSync (fs.js:902:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:71:13)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:77:24)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Function.sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:77:24)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at module.exports.Client.initFileStructure (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:132:14)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at new module.exports (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:38:8)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at new module.exports (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/API.js:99:17)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.<anonymous> (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/index.js:11:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Module._compile (internal/modules/cjs/loader.js:654:30)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:665:10)
Apr 14 11:43:19 app01 pm2-zabbix[3260]: Error: EACCES: permission denied, mkdir '/nonexistent'
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.fs.mkdirSync (fs.js:902:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:71:13)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:77:24)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Function.sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:77:24)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at module.exports.Client.initFileStructure (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:140:14)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at new module.exports (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:38:8)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at new module.exports (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/API.js:99:17)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.<anonymous> (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/index.js:11:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Module._compile (internal/modules/cjs/loader.js:654:30)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:665:10)
Apr 14 11:43:19 app01 pm2-zabbix[3260]: Error: ENOENT: no such file or directory, open '/nonexistent/.pm2/module_conf.json'
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.fs.openSync (fs.js:660:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.fs.writeFileSync (fs.js:1319:33)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at module.exports.Client.initFileStructure (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:148:10)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at new module.exports (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:38:8)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at new module.exports (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/API.js:99:17)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.<anonymous> (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/index.js:11:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Module._compile (internal/modules/cjs/loader.js:654:30)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:665:10)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Module.load (internal/modules/cjs/loader.js:566:32)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at tryModuleLoad (internal/modules/cjs/loader.js:506:12)
Apr 14 11:43:19 app01 pm2-zabbix[3260]: Error: EACCES: permission denied, mkdir '/nonexistent'
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.fs.mkdirSync (fs.js:902:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:71:13)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:77:24)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Function.sync (/usr/local/lib/node_modules/pm2-zabbix/node_modules/mkdirp/index.js:77:24)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at module.exports.Client.initFileStructure (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:156:14)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at new module.exports (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:38:8)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at new module.exports (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/API.js:99:17)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.<anonymous> (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/index.js:11:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Module._compile (internal/modules/cjs/loader.js:654:30)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:665:10)
Apr 14 11:43:19 app01 pm2-zabbix[3260]: fs.js:660
Apr 14 11:43:19 app01 pm2-zabbix[3260]:   return binding.open(pathModule.toNamespacedPath(path),
Apr 14 11:43:19 app01 pm2-zabbix[3260]:                  ^
Apr 14 11:43:19 app01 pm2-zabbix[3260]: Error: ENOENT: no such file or directory, open '/nonexistent/.pm2/pm2.log'
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at Object.fs.openSync (fs.js:660:18)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at module.exports.Client.launchDaemon (/usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:219:14)
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at /usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:103:10
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at /usr/local/lib/node_modules/pm2-zabbix/node_modules/pm2/lib/Client.js:307:14
Apr 14 11:43:19 app01 pm2-zabbix[3260]:     at process._tickCallback (internal/process/next_tick.js:176:11)
Apr 14 11:43:19 app01 systemd[1]: pm2-zabbix.service: Main process exited, code=exited, status=1/FAILURE

and for the other problem, zabbix_sender send only the hostname app01 and not the FQDN which is set in the zabbix config file and in the zabbix server configuration. Is there a way that the hostname will be taken from the zabbix configuration instead of /etc/hostname, because I'm not able to change it, Proxmox will reset it on container start.

[trenta]

I seem to be having this problem also. I have had it working in the past and am not sure when it stopped.
pm2-zabbix --monitor {"name":"pm2-zabbix","hostname":"cameras01","pid":1479,"level":50,"event":"PM2ZabbixMonitor#PM2StatusSent","error":{"killed":false,"code":2,"signal":null,"cmd":"/usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -"},"msg":"Failed to send PM2 status: Error: Command failed: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -\n","time":"2018-10-21T23:25:00.934Z","v":0}

[faissy2018]

can any one help me out i am still unable to receive any notfication on zabbix related to pm2 status

{"name":"pm2-zabbix","hostname":"ip-172-31-43-89","pid":25685,"level":50,"event":"PM2ZabbixMonitor#processListSendingError","error":{"killed":false,"code":1,"signal":null,"cmd":"/usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -"},"msg":"Failed to send process list with stats to server: Error: Command failed: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -\n","time":"2018-11-04T22:02:01.473Z","v":0} {"name":"pm2-zabbix","hostname":"ip-172-31-43-89","pid":25685,"level":50,"event":"PM2ZabbixMonitor#PM2StatusSent","error":{"killed":false,"code":1,"signal":null,"cmd":"/usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -"},"msg":"Failed to send PM2 status: Error: Command failed: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -\n","time":"2018-11-04T22:02:01.473Z","v":0} {"name":"pm2-zabbix","hostname":"ip-172-31-43-89","pid":25685,"level":10,"event":"PM2ZabbixMonitor#gotPM2State","processState":{"name":"PM2","status":"online","resources":{"cpu":0,"memory":48480256},"restarts":0,"pid":13550},"msg":"","time":"2018-11-04T22:02:01.474Z","v":0}

i tried pm2 stop cron but didnt receive any notification.

^C root@ip-172-31-43-89:/lib/systemd/system# pm2 stop cron [PM2] Applying action stopProcessId on app [cron](ids: 0) [PM2] cron ✓ ┌──────────┬────┬─────────┬──────┬─────┬─────────┬─────────┬────────┬─────┬────────┬──────┬──────────┐ │ App name │ id │ version │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │ ├──────────┼────┼─────────┼──────┼─────┼─────────┼─────────┼────────┼─────┼────────┼──────┼──────────┤ │ cron │ 0 │ N/A │ fork │ 0 │ stopped │ 60 │ 0 │ 0% │ 0 B │ root │ disabled │ └──────────┴────┴─────────┴──────┴─────┴─────────┴─────────┴────────┴─────┴────────┴──────┴──────────┘ Use pm2 show <id|name> to get more details about an app

[Dave10win]

Hi Im having the same problem, if tried everthing i can find on google but nuthing helps..

I made a replica of two of my servers (nginx + varnish and nodejs), on the original server everthing is working fine with zabbix, however on my replicas i'm havig problems with pm2-zabbix and zabbix-varnish but no problem with system info or nginx info.

This is the error i get wen i run (on the node server) "sudo pm2-zabbix --monitor --debug": {"name":"pm2-zabbix","hostname":"nodejs61replica","pid":98546,"level":50,"event":"PM2ZabbixMonitor#PM2StatusSent","error":{"kilt-file -"},"msg":"Failed to send PM2 status: Error: Command failed: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.c {"name":"pm2-zabbix","hostname":"nodejs61replica","pid":98546,"level":50,"event":"PM2ZabbixMonitor#PM2StatusSent","error":{"killed":false,"code":1,"signal":null,"cmd":"/usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -"},"msg":"Failed to send PM2 status: Error: Command failed: /usr/bin/zabbix_sender --config /etc/zabbix/zabbix_agentd.conf --input-file -\n","time":"2020-05-03T10:10:03.944Z","v":0}

on my varnish server i run "sudo /usr/local/bin/zabbix-varnish-cache.py -i '' send -c /etc/zabbix/zabbix_agentd.conf" and it seems evrthing is OK but zabbix server dos not recive the info:

Help please!