PM2 stopped cause of Error: write EPIPE error

markbrinkman commented 1 year ago

What's going wrong?

Last night our pm2 randomly stopped running after two weeks without any problems. After running pm2 resurrect the process restarted without any issues. When searching through the logs i found the following errors:

2023-01-25T20:00:17: PM2 log: App [app.compiled:13] starting in -cluster mode-
2023-01-25T20:00:17: PM2 log: App [app.compiled:10] starting in -cluster mode-
2023-01-25T20:00:17: PM2 log: App [app.compiled:1] starting in -cluster mode-
2023-01-25T20:00:17: PM2 error: Error: write EPIPE
    at ChildProcess.target._send (node:internal/child_process:874:20)
    at ChildProcess.<anonymous> (node:internal/child_process:677:16)
    at ChildProcess.emit (node:events:525:35)
    at ChildProcess.emit (node:domain:552:15)
    at emit (node:internal/child_process:946:14)
    at processTicksAndRejections (node:internal/process/task_queues:84:21)
2023-01-25T20:00:17: PM2 log: ===============================================================================
2023-01-25T20:00:17: PM2 log: --- PM2 global error caught ---------------------------------------------------
2023-01-25T20:00:17: PM2 log: Time                 : Wed Jan 25 2023 20:00:17 GMT+0000 (Coordinated Universal Time)
2023-01-25T20:00:17: PM2 error: write EPIPE
2023-01-25T20:00:17: PM2 error: Error: write EPIPE
    at ChildProcess.target._send (node:internal/child_process:874:20)
    at ChildProcess.target.send (node:internal/child_process:747:19)
    at sendHelper (node:internal/cluster/utils:28:15)
    at send (node:internal/cluster/primary:363:10)
    at Worker.disconnect (node:internal/cluster/primary:369:3)
    at Worker.destroy (node:internal/cluster/primary:382:10)
    at Worker.<anonymous> (/usr/lib/node_modules/pm2/lib/God.js:248:30)
    at Object.onceWrapper (node:events:628:26)
    at Worker.emit (node:events:513:28)
    at Worker.emit (node:domain:489:12)
2023-01-25T20:00:17: PM2 log: ===============================================================================
2023-01-25T20:00:17: PM2 error: [PM2] Resurrecting PM2
2023-01-25T20:00:17: PM2 log: App name:app.compiled id:24 disconnected
2023-01-25T20:00:17: PM2 log: App [app.compiled:32] exited with code [0] via signal [SIGINT]
2023-01-25T20:00:17: PM2 log: App [app.compiled:24] exited with code [0] via signal [SIGINT]
2023-01-25T20:00:17: PM2 log: App [app.compiled:32] starting in -cluster mode-
2023-01-25T20:00:17: PM2 log: App [app.compiled:24] starting in -cluster mode-
2023-01-25T20:00:17: PM2 log: App [app.compiled:13] online
2023-01-25T20:00:17: PM2 log: App [app.compiled:1] online
node:internal/modules/cjs/loader:988
  throw err;
  ^

Error: Cannot find module '/undefined'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:985:15)
    at Function.Module._load (node:internal/modules/cjs/loader:833:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:22:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}
2023-01-25T20:00:17: PM2 log: App [app.compiled:32] online
2023-01-25T20:00:17: PM2 log: App [app.compiled:10] online
2023-01-25T20:00:17: PM2 log: PM2 successfully forked
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
2023-01-25T20:00:18: PM2 log: ===============================================================================
2023-01-25T20:00:18: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2023-01-25T20:00:18: PM2 log: Time                 : Wed Jan 25 2023 20:00:18 GMT+0000 (Coordinated Universal Time)
2023-01-25T20:00:18: PM2 log: PM2 version          : 5.2.2
2023-01-25T20:00:18: PM2 log: Node.js version      : 16.18.1
2023-01-25T20:00:18: PM2 log: Current arch         : x64
2023-01-25T20:00:18: PM2 log: PM2 home             : /root/.pm2
2023-01-25T20:00:18: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2023-01-25T20:00:18: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2023-01-25T20:00:18: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2023-01-25T20:00:18: PM2 log: Application log path : /root/.pm2/logs
2023-01-25T20:00:18: PM2 log: Worker Interval      : 30000
2023-01-25T20:00:18: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2023-01-25T20:00:18: PM2 log: Concurrent actions   : 2
2023-01-25T20:00:18: PM2 log: SIGTERM timeout      : 1600
2023-01-25T20:00:18: PM2 log: ===============================================================================
2023-01-25T20:00:18: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2023-01-25T20:00:18: PM2 log: App [pm2-logrotate:0] online
2023-01-25T20:00:18: PM2 log: Stopping app:pm2-logrotate id:0
2023-01-25T20:00:18: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGINT]
2023-01-25T20:00:19: PM2 log: pid=1679090 msg=process killed
2023-01-25T20:00:19: PM2 log: PM2 successfully stopped

How could we reproduce this issue?

I am not quite sure how to produce/avoid this issue.

Supporting information

$ pm2 report

--- PM2 report ----------------------------------------------------------------
Date                 : Thu Jan 26 2023 11:03:27 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.2.2
node version         : 16.18.1
node path            : not found
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
uptime               : 208min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.2.2
node version         : 16.18.1
node path            : not found
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
cpus nb              : 32
freemem              : 63062401024
totalmem             : 67530924032
home                 : /root
===============================================================================
--- PM2 list -----------------------------------------------
**Redacted**
Module
┌────┬──────────────────────────────┬───────────────┬──────────┬──────────┬──────┬──────────┬──────────┬──────────┐
│ id │ module                       │ version       │ pid      │ status   │ ↺    │ cpu      │ mem      │ user     │
├────┼──────────────────────────────┼───────────────┼──────────┼──────────┼──────┼──────────┼──────────┼──────────┤
│ 0  │ pm2-logrotate                │ 2.7.0         │ 1689056  │ online   │ 0    │ 0.6%     │ 57.4mb   │ root     │
└────┴──────────────────────────────┴───────────────┴──────────┴──────────┴──────┴──────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/root/.pm2/pm2.log last 20 lines:
PM2        | 2023-01-26T11:00:14: PM2 log: App name:app.compiled id:33 disconnected
PM2        | 2023-01-26T11:00:14: PM2 log: App [app.compiled:33] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:00:14: PM2 log: App [app.compiled:33] starting in -cluster mode-
PM2        | 2023-01-26T11:00:15: PM2 log: App [app.compiled:33] online
PM2        | 2023-01-26T11:00:17: PM2 log: App name:app.compiled id:35 disconnected
PM2        | 2023-01-26T11:00:17: PM2 log: App [app.compiled:35] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:00:17: PM2 log: App [app.compiled:35] starting in -cluster mode-
PM2        | 2023-01-26T11:00:17: PM2 log: App [app.compiled:35] online
PM2        | 2023-01-26T11:00:19: PM2 log: App name:app.compiled id:36 disconnected
PM2        | 2023-01-26T11:00:19: PM2 log: App [app.compiled:36] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:00:19: PM2 log: App [app.compiled:36] starting in -cluster mode-
PM2        | 2023-01-26T11:00:19: PM2 log: App [app.compiled:36] online
PM2        | 2023-01-26T11:00:22: PM2 log: App name:app.compiled id:3 disconnected
PM2        | 2023-01-26T11:00:22: PM2 log: App [app.compiled:3] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:00:22: PM2 log: App [app.compiled:3] starting in -cluster mode-
PM2        | 2023-01-26T11:00:22: PM2 log: App [app.compiled:3] online
PM2        | 2023-01-26T11:01:03: PM2 log: App name:app.compiled id:32 disconnected
PM2        | 2023-01-26T11:01:03: PM2 log: App [app.compiled:32] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:01:03: PM2 log: App [app.compiled:32] starting in -cluster mode-
PM2        | 2023-01-26T11:01:03: PM2 log: App [app.compiled:32] online

xiaoqiang1999 commented 1 year ago

I also encountered this error when using "child_process": write EPIPE. The functions I call are "child_process.spawn" and "child_process.exec". There are no errors running the program directly using "node main.js".

Please, have you solved this problem?

CatchTheDog commented 1 year ago

any help, please！I encountered the same error after my cron job start two days ,and it seems running well after restart;and again, it crash with the error after about two days.

CatchTheDog commented 1 year ago

2023-06-18T14:01:11: PM2 error: Error: write EPIPE at ChildProcess.target._send (internal/child_process.js:839:20) at ChildProcess.target.send (internal/child_process.js:710:19) at God.sendDataToProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:720:12) at Server.onmessage (/usr/lib/node_modules/pm2/node_modules/pm2-axon-rpc/lib/server.js:104:6) at RepSocket.emit (events.js:376:20) at RepSocket.emit (domain.js:470:12) at Parser. (/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/rep.js:51:15) at Parser.emit (events.js:376:20) at Parser.emit (domain.js:532:15) at Parser._write (/usr/lib/node_modules/pm2/node_modules/amp/lib/stream.js:91:16) 2023-06-18T14:01:11: PM2 log: =============================================================================== 2023-06-18T14:01:11: PM2 log: --- PM2 global error caught --------------------------------------------------- 2023-06-18T14:01:11: PM2 log: Time : Sun Jun 18 2023 14:01:11 GMT+0800 (China Standard Time) 2023-06-18T14:01:11: PM2 error: write EPIPE 2023-06-18T14:01:11: PM2 error: Error: write EPIPE at ChildProcess.target._send (internal/child_process.js:839:20) at ChildProcess.target.send (internal/child_process.js:710:19) at God.sendDataToProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:720:12) at Server.onmessage (/usr/lib/node_modules/pm2/node_modules/pm2-axon-rpc/lib/server.js:104:6) at RepSocket.emit (events.js:376:20) at RepSocket.emit (domain.js:470:12) at Parser. (/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/rep.js:51:15) at Parser.emit (events.js:376:20) at Parser.emit (domain.js:532:15) at Parser._write (/usr/lib/node_modules/pm2/node_modules/amp/lib/stream.js:91:16)

xiaoqiang1999 commented 1 year ago

any help, please！I encountered the same error after my cron job start two days ,and it seems running well after restart;and again, it crash with the error after about two days.

兄弟，这个问题我解决了，先说一下我的程序。我的操作系统是Linux，程序使用child_process的exec执行goaccess命令(goaccess是一个Nginx日志分析工具)，大概命令是：

cat xxx.log | goaccess xxx

当我在dev模式下运行程序时，一切正常；当我使用PM2运行程序时，会报此错误：write EPIPE error 我在好多网站上查找了好久，费了好大力气，才意外的发现这个问题跟goaccess有关，当我使用PM2运行程序时，goaccess并不知道数据会从pipe管道输入，所以报了这个错。我查找了goaccess的issue，作者提供了一个标识符，可以告诉goaccess数据是从pipe管道输入的，当我把命令改成这样，程序就正常运行了：

cat xxx.log | goaccess xxx -

注意，比之前的命令多了个“-”，这就是goaccess管道输入的标识符。

我的问题就这样解决了，希望能帮到你。

The following is the English translation of ChatGPT：

Brother, I solved this problem. Let me first explain my program. My operating system is Linux, and the program uses the exec function of child_process to execute the goaccess command (which is an Nginx log analysis tool). The command is roughly as follows:

cat xxx.log | goaccess xxx

When I run the program in dev mode, everything is normal. But when I use PM2 to run the program, it reports an error: write EPIPE error. I searched many websites for a long time and finally found out that this problem is related to goaccess. When I use PM2 to run the program, goaccess does not know that data will be input from the pipe, so it reports this error. I searched the goaccess issue tracker and found that the author provided an identifier that can tell goaccess that data is being input from a pipe. When I changed the command to the following, the program ran normally:

cat xxx.log | goaccess xxx -

Note that there is an extra hyphen compared to the previous command, which is the identifier for goaccess to indicate that data is being piped in. That's how I solved my problem. I hope it can help you.

Unitech / pm2