search

RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
40.35k stars 10.48k forks source link

Memory leak #13602

Open bazzilio opened 5 years ago

bazzilio commented 5 years ago

Description:

I see memory leak on the latest rocket chat version 0.74.3 How can i debug the root cause of the issue ?

Expected behavior:

No memory leak :)

Actual behavior:

image

Server Setup Information:

Relevant logs:

I see some strange errors, that can be related to the issue:

[34mI20190302-14:54:21.610(0) Exception in onLogin callback: TypeError: Cannot redefine property: loginAt
at Function.defineProperty (<anonymous>)
at Object.keys.forEach.p (packages/rocketchat:statistics/server/lib/SAUMonitor.js:279:12)
at Array.forEach (<anonymous>)
at SAUMonitorClass._updateConnectionInfo (packages/rocketchat:statistics/server/lib/SAUMonitor.js:278:22)
at Accounts.onLogin.info (packages/rocketchat:statistics/server/lib/SAUMonitor.js:120:9)
at packages/callback-hook/hook.js:130:22
at _onLoginHook.each.callback (packages/accounts-base/accounts_server.js:178:7)
at Hook.each (packages/callback-hook/hook.js:108:15)
at AccountsServer._successfulLogin (packages/accounts-base/accounts_server.js:177:23)
at AccountsServer._attemptLogin (packages/accounts-base/accounts_server.js:350:12)
at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:520:23)
at MethodInvocation.methodMap.(anonymous function) (packages/rocketchat_monitoring.js:2731:30)
at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1767:12)
at DDP._CurrentMethodInvocation.withValue (packages/ddp-server/livedata_server.js:719:19)
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1304:12)
at DDPServer._CurrentWriteFence.withValue (packages/ddp-server/livedata_server.js:717:46)
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1304:12)
at Promise (packages/ddp-server/livedata_server.js:715:46)
at new Promise (<anonymous>)
at Session.method (packages/ddp-server/livedata_server.js:689:23)
at packages/ddp-server/livedata_server.js:559:43 
DDP RATE LIMIT: limit by connectionId per method
{
  "allowed": false,
  "timeToReset": 644,
  "numInvocationsLeft": 0,
  "userId": "oetxdmjGiBewXf34T",
  "clientAddress": "172.20.0.2",
  "type": "method",
  "name": "stream-notify-room",
  "connectionId": "rpkiRALqTkoMpSwjz",
  "broadcastAuth": false
}

Full logs: Logs from the server interface: server-rocket.log

Logs from docker container: docker-rocket.log

tassoevan commented 5 years ago

The Exception in onLogin callback: TypeError: Cannot redefine property: loginAt was solved by #13553, but I'm not sure it is connected somehow.

bazzilio commented 5 years ago

Thank you, for quick answer, i'll try update to the master branch and return later.

rasos commented 5 years ago

Same observation here that memory usage is increasing every day with RC 0.74.1

Duplicate of https://github.com/RocketChat/Rocket.Chat/issues/12880 ?

bazzilio commented 5 years ago

In #12880 says, that Leak rate seems to correlate with message volume" In my case i don't see any correlation.

izenk commented 5 years ago

Faced with same situation on 0.73.2 (mongo 4.0.5) docker-compose deployment 50 users, 30 active concurently Every time after a week of stable work - memory usage increases up 4GB (2GB mongo and 2GB for rchat app). At the same time Recommendation says that 2GB should be enough for my situation

May it related to this #10714 Because this warning is present in logs

dhoeld commented 5 years ago

Same problem here. There is a restart for any of our 3 Rocket.Chat replicas on our Kubernetes Cluster every night due to "FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory"

We also got these strange errors mentioned above in our logs.

cortex3 commented 5 years ago

same here, using CentOS 7, rocketchat 0.73.2, mongodb 4.0.5, node 8.11.4 and 2 instances on the same host. Restarting the service resolves the problem for a while. rock

ninghao commented 5 years ago

i had the same problem. also runs on a centos server with docker.

reetp commented 5 years ago

Please test on latest v1.1.1

engelgabriel commented 5 years ago

HI @bazzilio @rasos @ninghao are you still experiencing this issue with the latest versions?

fcoppolani commented 5 years ago

We have the same problem with 1.2.3, we are going to upgrade to 2.1 soon to check this issue.

UAnton commented 4 years ago

I had the same problem with Snap 2.4.2

pawadski commented 4 years ago

Potentially same problem with 3.0.3, NodeJS memory usage really high. Below Grafana screenshot of NodeJS memory exported by Prometheus exporter (enabled in Logs). We have 4 RocketChat instances and below picture shows one of them using up considerably more memory.

image

Bengrunt commented 3 years ago

Hello there, still seeing the memory leak on our 3.12 instances.

I know this kind of issue is always hard to fix but is there any hope in this getting fixed sometime soon or do we invest time in countermeasures on the deployment side? If anyone has feedback on this I'd be glad to read it :)

ThrowAwayDan commented 1 year ago

Issue persists. Just noticed that RC server takes an insane amount of memory. Found out about this when I tried to activate jitsi on the marketplace, which pushed it over the edge.

image