Rocket.Chat not responding - Error: accept EMFILE

vv9k commented 4 years ago

Description:

Rocket.Chat suddenly stopped responding. Mobile clients kept loading and desktop client responded with 502 Bad Gateway. The error came up 600000 times in ~3 minutes. After systemctl restart rocketchat it went back to working as usual.

Expected behavior:

Rocket.Chat working as expected.

Actual behavior:

Service unavailable.

Server Setup Information:

Version of Rocket.Chat Server: 3.3.0
Operating System: Debian 9.12
Deployment Method: tar
Number of Running Instances: 1
DB Replicaset Oplog:
NodeJS Version: 12.18.0
MongoDB Version: 4.0.19

Relevant logs:

Jun 16 17:14:07 chat rocketchat[1818]: Exception in callback of async function: Error: accept EMFILE
Jun 16 17:14:07 chat rocketchat[1818]:     at TCP.onconnection (net.js:1534:24) {
Jun 16 17:14:07 chat rocketchat[1818]:   errno: 'EMFILE',
Jun 16 17:14:07 chat rocketchat[1818]:   code: 'EMFILE',
Jun 16 17:14:07 chat rocketchat[1818]:   syscall: 'accept'
Jun 16 17:14:07 chat rocketchat[1818]: }

root@chat:~# rg "errno: 'EMFILE'" /var/log/syslog | head -n 1
Jun 16 17:14:06 chat rocketchat[1818]:   errno: 'EMFILE',
root@chat:~# rg "errno: 'EMFILE'" /var/log/syslog | tail -n 1
Jun 16 17:17:26 chat rocketchat[1818]:   errno: 'EMFILE',
root@chat:~# rg "errno: 'EMFILE'" /var/log/syslog | wc -l
583658

kaffeeknecht commented 4 years ago

Same Issue here

Version of Rocket.Chat Server: 3.2.2
Operating System: Ubuntu 16.04.4
Deployment Method: snap
Number of Running Instances: 1
DB Replicaset Oplog:
NodeJS Version: 12.16.1
MongoDB Version: 3.6.14

There is a second error with errno: 'EMFILE' which occurred some time (~3-4 hours) prior to the crash:

Jun 29 10:28:43 chat rocketchat-server.rocketchat-server[18333]: Push ➔ info Trying sending push to gateway again in 100 milliseconds
Jun 29 10:28:43 chat rocketchat-server.rocketchat-server[18333]: server.js:204 Push ➔ error Error sending push to gateway (0 try) -> Error: getaddrinfo EMFILE gateway.rocket.chat
Jun 29 10:28:43 chat rocketchat-server.rocketchat-server[18333]:     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26) {
Jun 29 10:28:43 chat rocketchat-server.rocketchat-server[18333]:   errno: 'EMFILE',
Jun 29 10:28:43 chat rocketchat-server.rocketchat-server[18333]:   code: 'EMFILE',
Jun 29 10:28:43 chat rocketchat-server.rocketchat-server[18333]:   syscall: 'getaddrinfo',
Jun 29 10:28:43 chat rocketchat-server.rocketchat-server[18333]:   hostname: 'gateway.rocket.chat'
Jun 29 10:28:43 chat rocketchat-server.rocketchat-server[18333]: }

RAurelian commented 4 years ago

We get this error too, every other day. We see the following errors: Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: errno: 'EMFILE', Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: code: 'EMFILE', Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: syscall: 'accept' Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: } Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: Exception in callback of async function: Error: accept EMFILE Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: at TCP.onconnection (net.js:1532:24) { Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: errno: 'EMFILE', Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: code: 'EMFILE', Aug 03 18:07:47 rockchat rocketchat-server.rocketchat-server[13165]: syscall: 'accept'

github-actions[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

RAurelian commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Well, the issue still occurs in 3.6.2 (Ubuntu Snap)..

LifeIsAParadox commented 4 years ago

we get this error too

bombaata commented 3 years ago

got this error too

+---------------------------------------------------+
|                   SERVER RUNNING                  |
+---------------------------------------------------+
|                                                   |
|  Rocket.Chat Version: 3.9.4                       |
|       NodeJS Version: 12.18.4 - x64               |
|      MongoDB Version: 4.0.22                      |
|       MongoDB Engine: mmapv1                      |
|             Platform: linux                       |
|         Process Port: 3000                        |
|             Site URL: https://xxxxxxxxxxxxxxxxxxx |
|     ReplicaSet OpLog: Enabled                     |
|          Commit Hash: 889669024f                  |
|        Commit Branch: HEAD                        |
|                                                   |
+---------------------------------------------------+

Feb  2 12:10:52 server rocketchat[15406]: Exception in callback of async function: Error: accept EMFILE
Feb  2 12:10:52 server rocketchat[15406]:     at TCP.onconnection (net.js:1533:24)
Feb  2 12:10:52 server rocketchat[15406]:     at TCP.callbackTrampoline (internal/async_hooks.js:120:14) {
Feb  2 12:10:52 server rocketchat[15406]:   errno: 'EMFILE',
Feb  2 12:10:52 server rocketchat[15406]:   code: 'EMFILE',
Feb  2 12:10:52 server rocketchat[15406]:   syscall: 'accept'
Feb  2 12:10:52 server rocketchat[15406]: }

this error occured 4389717 times and produced 2GB+ of log before we restart the service.

We got DNS problems today, I think it's related.

RocketChat / Rocket.Chat