search

Unitech / pm2

Node.js Production Process Manager with a built-in Load Balancer.
https://pm2.keymetrics.io/docs/usage/quick-start/
Other
41.59k stars 2.62k forks source link

Heap Out of Memory Error While Running Python Application #5897

Open nauyan opened 1 month ago

nauyan commented 1 month ago

I have deployed a Python Flask application using PM2. The deployment works smoothly under light traffic, but when the application experiences 10-15 concurrent users running heavier workloads, the PM2 service crashes. However, the underlying Python process continues to run.

Can anyone advise on the potential causes of this issue and suggest how I can prevent it from recurring?

Supporting information

Error from pm2.log

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xca5430 node::Abort() [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
 2: 0xb7807d  [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
 3: 0xeca0b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
 4: 0xeca397 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
 5: 0x10dc0e5  [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
 6: 0x10f3f68 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
 7: 0x114cb3c v8::internal::MinorGCJob::Task::RunInternal() [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
 8: 0xd201a6  [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
 9: 0xd22d5f node::PerIsolatePlatformData::FlushForegroundTasksInternal() [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
10: 0x188aa33  [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
11: 0x189f41b  [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
12: 0x188b757 uv_run [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
13: 0xbbbb83 node::SpinEventLoopInternal(node::Environment*) [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
14: 0xcf4d25  [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
15: 0xcf56ed node::NodeMainInstance::Run() [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
16: 0xc5f367 node::Start(int, char**) [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
17: 0x7f350ae3feb0  [/lib64/libc.so.6]
18: 0x7f350ae3ff60 __libc_start_main [/lib64/libc.so.6]
19: 0xbb8d7e _start [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
2024-09-26T14:23:21: PM2 log: ===============================================================================
2024-09-26T14:23:21: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2024-09-26T14:23:21: PM2 log: Time                 : Thu Sep 26 2024 14:23:21 GMT+0000 (Coordinated Universal Time)
2024-09-26T14:23:21: PM2 log: PM2 version          : 5.4.2
2024-09-26T14:23:21: PM2 log: Node.js version      : 20.11.0
2024-09-26T14:23:21: PM2 log: Current arch         : x64
2024-09-26T14:23:21: PM2 log: PM2 home             : /home/ec2-user/.pm2
2024-09-26T14:23:21: PM2 log: PM2 PID file         : /home/ec2-user/.pm2/pm2.pid
2024-09-26T14:23:21: PM2 log: RPC socket file      : /home/ec2-user/.pm2/rpc.sock
2024-09-26T14:23:21: PM2 log: BUS socket file      : /home/ec2-user/.pm2/pub.sock
2024-09-26T14:23:21: PM2 log: Application log path : /home/ec2-user/.pm2/logs
2024-09-26T14:23:21: PM2 log: Worker Interval      : 30000
2024-09-26T14:23:21: PM2 log: Process dump file    : /home/ec2-user/.pm2/dump.pm2
2024-09-26T14:23:21: PM2 log: Concurrent actions   : 2
2024-09-26T14:23:21: PM2 log: SIGTERM timeout      : 1600
2024-09-26T14:23:21: PM2 log: ===============================================================================

PM2 Report

--- PM2 report ----------------------------------------------------------------
Date                 : Thu Sep 26 2024 14:49:48 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.4.2
node version         : 20.11.0
node path            : /home/ec2-user/.nvm/versions/node/v20.11.0/bin/pm2
argv                 : /home/ec2-user/.nvm/versions/node/v20.11.0/bin/node,/home/ec2-user/.nvm/versions/node/v20.11.0/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : ec2-user
uid                  : 1000
gid                  : 1000
uptime               : 26min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.4.2
node version         : 20.11.0
node path            : /home/ec2-user/.nvm/versions/node/v20.11.0/bin/pm2
argv                 : /home/ec2-user/.nvm/versions/node/v20.11.0/bin/node,/home/ec2-user/.nvm/versions/node/v20.11.0/bin/pm2,report
argv0                : node
user                 : ec2-user
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
cpus nb              : 4
freemem              : 14047399936
totalmem             : 16758525952
home                 : /home/ec2-user
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬───────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name      │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
└────┴───────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
[PM2][WARN] Current process list is not synchronized with saved list. App cdr differs. Type 'pm2 save' to synchronize.
===============================================================================
--- Daemon logs --------------------------------------------
/home/ec2-user/.pm2/pm2.log last 20 lines:
PM2        | 16: 0xc5f367 node::Start(int, char**) [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
PM2        | 17: 0x7f350ae3feb0  [/lib64/libc.so.6]
PM2        | 18: 0x7f350ae3ff60 __libc_start_main [/lib64/libc.so.6]
PM2        | 19: 0xbb8d7e _start [PM2 v5.4.2: God Daemon (/home/ec2-user/.pm2)]
PM2        | 2024-09-26T14:23:21: PM2 log: ===============================================================================
PM2        | 2024-09-26T14:23:21: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
PM2        | 2024-09-26T14:23:21: PM2 log: Time                 : Thu Sep 26 2024 14:23:21 GMT+0000 (Coordinated Universal Time)
PM2        | 2024-09-26T14:23:21: PM2 log: PM2 version          : 5.4.2
PM2        | 2024-09-26T14:23:21: PM2 log: Node.js version      : 20.11.0
PM2        | 2024-09-26T14:23:21: PM2 log: Current arch         : x64
PM2        | 2024-09-26T14:23:21: PM2 log: PM2 home             : /home/ec2-user/.pm2
PM2        | 2024-09-26T14:23:21: PM2 log: PM2 PID file         : /home/ec2-user/.pm2/pm2.pid
PM2        | 2024-09-26T14:23:21: PM2 log: RPC socket file      : /home/ec2-user/.pm2/rpc.sock
PM2        | 2024-09-26T14:23:21: PM2 log: BUS socket file      : /home/ec2-user/.pm2/pub.sock
PM2        | 2024-09-26T14:23:21: PM2 log: Application log path : /home/ec2-user/.pm2/logs
PM2        | 2024-09-26T14:23:21: PM2 log: Worker Interval      : 30000
PM2        | 2024-09-26T14:23:21: PM2 log: Process dump file    : /home/ec2-user/.pm2/dump.pm2
PM2        | 2024-09-26T14:23:21: PM2 log: Concurrent actions   : 2
PM2        | 2024-09-26T14:23:21: PM2 log: SIGTERM timeout      : 1600
PM2        | 2024-09-26T14:23:21: PM2 log: ===============================================================================