Sudden PM2 core dump. Asking for assistance in finding the root cause, if possible.

[JuriStefanovski]

What's going wrong?

host syslog extract: Mar 15 03:30:10 Center1 systemd[1]: Started Process Core Dump (PID 3079673/UID 0). Mar 15 03:30:12 Center1 systemd-coredump[3079674]: Core file was truncated to 2147483648 bytes. Mar 15 03:30:17 Center1 systemd-coredump[3079674]: Process 391241 (PM2 v5.3.0: God) of user 1000 dumped core.#012#012Stack trace of thread 391241:#012#0 0x00007f204686a00b n/a (n/a + 0x0) Mar 15 03:30:17 Center1 systemd[1]: systemd-coredump@1-3079673-0.service: Succeeded. Mar 15 03:30:17 Center1 systemd[1]: pm2-user.service: New main PID 391249 does not belong to service, and PID file is not owned by root. Refusing. Mar 15 03:30:17 Center1 systemd[1]: pm2-user.service: Main process exited, code=dumped, status=6/ABRT Mar 15 03:30:17 Center1 systemd[1]: pm2-user.service: Failed with result 'core-dump'.

How could we reproduce this issue?

Not sure if this can be easily replicated, this PM2 worked great for a long time, but perhaps the information contained in the core dump could be of interest to developers? The core dump file is available, but its size is around 299 MB. Ready to upload/send it wherever you say, if necessary. Please close this case if it is not relevant to you. I apologize for any inconvenience.

Supporting information

Linux Mint 20.3 Una Linux Center1 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

PM2 report Date : Mon Mar 18 2024 14:16:49 GMT+0200 (Eastern European Standard Time)

Daemon pm2d version : 5.3.0 node version : 10.19.0 node path : not found argv : /usr/bin/node,/usr/local/lib/node_modules/pm2/lib/Daemon.js argv0 : node user : user uid : 1000 gid : 1000 uptime : 4747min

--- CLI ---------------------------------------------------- local pm2 : 5.3.0 node version : 10.19.0 node path : /usr/local/bin/pm2 argv : /usr/bin/node,/usr/local/bin/pm2,report argv0 : node user : user uid : 1000 gid : 1000

--- System info -------------------------------------------- arch : x64 platform : linux type : Linux cpus : Intel(R) Xeon(R) E-2246G CPU @ 3.60GHz cpus nb : 12 freemem : 20717719552 totalmem : 33243213824 home : /home/user

--- PM2 list ----------------------------------------------- ┌────┬────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐ │ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │ ├────┼────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤ │ 2 │ db │ default │ N/A │ fork │ 10845 │ 3D │ 0 │ online │ 0.7% │ 68.8mb │ user │ disabled │ │ 3 │ line │ default │ N/A │ fork │ 10901 │ 3D │ 0 │ online │ 38.1% │ 88.2mb │ user │ disabled │ │ 4 │ linecleanup │ default │ N/A │ fork │ N/A │ 0 │ 0 │ stopped │ 0% │ 0b │ user │ disabled │ │ 1 │ registry_server │ default │ N/A │ fork │ 10821 │ 3D │ 5 │ online │ 1.1% │ 20.2mb │ user │ disabled │ └────┴────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘ Module ┌────┬──────────────────────────────┬───────────────┬──────────┬──────────┬──────┬──────────┬──────────┬──────────┐ │ id │ module │ version │ pid │ status │ ↺ │ cpu │ mem │ user │ ├────┼──────────────────────────────┼───────────────┼──────────┼──────────┼──────┼──────────┼──────────┼──────────┤ │ 0 │ pm2-logrotate │ 2.7.0 │ 10694 │ online │ 0 │ 0.3% │ 100.0mb │ user │ └────┴──────────────────────────────┴───────────────┴──────────┴──────────┴──────┴──────────┴──────────┴──────────┘

--- Daemon logs -------------------------------------------- /home/user/.pm2/pm2.log last 20 lines: PM2 | 2024-03-15T07:09:07: PM2 log: Application log path : /home/user/.pm2/logs PM2 | 2024-03-15T07:09:07: PM2 log: Worker Interval : 30000 PM2 | 2024-03-15T07:09:07: PM2 log: Process dump file : /home/user/.pm2/dump.pm2 PM2 | 2024-03-15T07:09:07: PM2 log: Concurrent actions : 2 PM2 | 2024-03-15T07:09:07: PM2 log: SIGTERM timeout : 1600 PM2 | 2024-03-15T07:09:07: PM2 log:

PM2 | 2024-03-15T07:09:08: PM2 log: App [pm2-logrotate:0] starting in -fork mode- PM2 | 2024-03-15T07:09:08: PM2 log: App [pm2-logrotate:0] online PM2 | 2024-03-15T07:09:12: PM2 log: App [registry_server:1] starting in -fork mode- PM2 | 2024-03-15T07:09:12: PM2 log: App [registry_server:1] online PM2 | 2024-03-15T07:09:13: PM2 log: App [db:2] starting in -fork mode- PM2 | 2024-03-15T07:09:13: PM2 log: App [db:2] online PM2 | 2024-03-15T07:09:14: PM2 log: App [line:3] starting in -fork mode- PM2 | 2024-03-15T07:09:14: PM2 log: App [line:3] online PM2 | 2024-03-16T07:09:08: PM2 log: [PM2] This PM2 is not UP TO DATE PM2 | 2024-03-16T07:09:08: PM2 log: [PM2] Upgrade to version 5.3.1 PM2 | 2024-03-17T07:09:08: PM2 log: [PM2] This PM2 is not UP TO DATE PM2 | 2024-03-17T07:09:08: PM2 log: [PM2] Upgrade to version 5.3.1 PM2 | 2024-03-18T07:09:08: PM2 log: [PM2] This PM2 is not UP TO DATE PM2 | 2024-03-18T07:09:08: PM2 log: [PM2] Upgrade to version 5.3.1

[ultimate-tester]

The first thing that comes to mind is that your system was out of memory, could that have been the case?

[JuriStefanovski]

I looked at the statistics of the monitoring system - at the time of the crash, ~40% of 32 GB of RAM remained free in the system.

Update: Having studied the statistics a little more carefully, I noticed that about two days before the failure, memory consumption began to increase, slowly but surely, from the usual ~48% to 58% It looks like a memory leak, but it is not possible to identify the culprit process; my monitoring, unfortunately, is very basic.

[arasmussen]

Running into the same / a similar issue. Also posted an issue here: https://github.com/Unitech/pm2/issues/5797