Perfomance issues when upgrading from 6.3.3 to 6.4.2

[shiryov]

Description:

Version 6.4.2 loads servers heavily. In the grafana we see many requests rate that were not that before. Docker instances are restarting with OOM.

Steps to reproduce:

1. Update your RC from 6.3.3 to 6.4.2
2. Look at the production Graphаna
3. See docker logs

Expected behavior:

Performance and stability are no worse than the previous version 6.3.3

Actual behavior:

High CPU load on 5 servers, instance restarts

Server Setup Information:

• Version of Rocket.Chat Server: 6.4.2 Enterprise ~6000 active sessions
• Operating System: Oracle Linux Server release 8.6
• Deployment Method: docker
• Number of Running Instances: 16
• DB Replicaset Oplog: yes
• NodeJS Version: v14.21.3
• MongoDB Version: 5.0.15 / wiredTiger (oplog Enabled)

Client Setup Information

• Desktop App or Browser Version: Chrome 117 Rocket.Chat Electron 3.9.9, 3.9.6, 3.8.17
• Operating System: Windows, Mac, Linux

Additional context

there are a lot of calls to these API methods:

action.buttons
commands.list

In a few hours without disconnections, I saw in my Google Chrome(117) - 50 calling of API method commands.list

RC logs chart with API section for one hour:

before update

after update

There is also a dependence of this issue #30717, but it is not significant. We found out that there are not many such clients in our network.

Relevant logs:

[ankar84]

Stability of Rocket.Chat service decreased heavily. Please help us to improve it.

[ankar84]

That could be related fix https://github.com/RocketChat/Rocket.Chat/pull/30819 that can help a bit.