Rocket chat from time to time is frozen

[jankogaga]

Hi,

Rocket chat from time to time is frozen. The issue occurs among many versions (the latest I have tried 0.71.1). Rocket chat is running in docker container with Centos 7. I am not able to find anything useful in docker logs. When doing curl, I am able to find the content of a login page.

Thanks, Dragan

[reetp]

Do you use a proxy ? A lot of issues are to do with proxy setups (I 've had a few !) If there are no errors in Rocket, then it is unlikely to be Rocket itself. Please go check your server logs - httpd etc.

[jankogaga]

Yes, we use Nginx proxy. But, the issue never existed last year, but in newer versions. Regarding logs, all I can find: /var/log/httpd/ssl_error_log /var/log/httpd/error_log They are empty.

Thanks, Dragan

[reetp]

Lots of things have changed since last year, not just Rocket....

You probably ought to go and turn on some nginx logging. That may help you. Start here: https://docs.nginx.com/nginx/admin-guide/monitoring/logging/

[jankogaga]

Thank you. We have already set Nginx logging, so I will back to you once I have another Rocket freezing.

[jankogaga]

Just it has been frozen, no records in nginx log.

[geekgonecrazy]

Anything happening at that time on your server when it freezes? How many users connected? How many instances of rocket.chat?

[jankogaga]

I can't find any log at that frozen time. We have 12 users in total. We have 1 instance.

[MathiasGmeiner]

I have the same problem. Around 10 users, one instance. If I‘m correct the log says something about mongodb, but I have the log not handy right now.

[geekgonecrazy]

Do you have message pruning or any thing like that setup?

Do you see CPU spikes in relation to this freezing?

[jankogaga]

I forgot to mention, Rocket chat is dockerized and uses RAM and CPU from the host server and the resources there are not the issue.

[jankogaga]

Just noticed, when Rocket chat is freezing Process Janus takes 6% of CPU in comparison when it is working well (1-2%)

[sharkmax88]

Greetings, we had such a problem when there was a shortage of RAM, in this case SWAP began to be used, and its volume was too small initially, and when it ended the rocket was frozen. First, I discovered a pattern that this happens when you enter the administration panel from a mobile device. And then it turned out that this was due to a lack of RAM. After rasherieniya RAM and swap everything became good.

[reetp]

Can people advise what amount of RAM they are talking about please?

[MathiasGmeiner]

I had one with 2GB, with that one Rocket.chat was frozen 1x a week. Now it's running on 4GB, seems fine since the last 10 days.

[sharkmax88]

The situation is exactly the same as in the previous commentator, with a 2 GB rocket stably frozen once a week, with 3 GB and increased to 1 GB swap, everything is OK!

[jankogaga]

I must stress out that my Rocket running in docker has a lot of available RAM on the host node. However, I don't know how to make Rocket to use it. Does any configuration exist for Rocket in order to determine how much RAM to use?

[jankogaga]

Do we have any recommendation from Rocket Chat team?

[geekgonecrazy]

The problem is we really don’t have enough information to go on.

If small amount of memory with mongo and rocket.chat in same system. It could be a memory struggle.

Would be interesting to see stats from Prometheus or system graphed so we could see if a pattern associated with freezes

[jankogaga]

Before installation of any SW on Rocket Chat, could you please tell me if you know how to configure a memory usage of Rocket Chat? Here is the output of free command (one more time Rocket chat is installed as a Docker and it uses all host's resources - there is no HW resource isolation) [root@rocket /]# free -h total used free shared buff/cache available Mem: 62G 17G 376M 101M 44G 44G Swap: 63G 9.3G 54G

Thanks, Dragan

[jankogaga]

Hi, Here is a graph from Munin monitoring SW, taken at the time of freezing. Please take a look at rocket. The other docker containers are covered.

[geekgonecrazy]

Is anyone still getting this? If so if you open a new incognito tab.. does the issue persist? Are you able to login from that new tab?

[jankogaga]

Hi geekgonecrazy,

I am able to log in from an incognito tab. However, freezing doesn't persist just on one session, but for all logged users.

[Nikk-0]

Hi 👋 I get the same issu with rocket behind proxy (Apache2). We're not with a docker instance, it's legacy installation (manual) We only have one instance. We don't have too many users.

We get noting in the logs :

• mongodb seems OK (slow query but nothing at the time of frozen)
• apache2 is ok
• syslog (mode debug for rocket log.... )
• CPU / RAM is ok (with htop)
• We monitor network output and input and it's not too much

It freeze for all users... send and receipt for clients. and some user get "houston we have a problem..." . Web page is very slow to show up a this moment.

It freeze frome 1-2 minutes. But multiple time in 10-30 minutes...

At the moment when it freeze rocket chat seem to have a log hole in log :

EDIT : They get sent at 13h36 so after 36 minutes waiting ...

[Nikk-0]

After moving proxy from apache to nginx no more freeze is happening. Maybe a websocket timeout... Don't really know.