Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
4.11k stars 552 forks source link

MeshCentral freeze after couple of days - Allocation failed - JavaScript heap out of memory #5584

Open Janiaking opened 10 months ago

Janiaking commented 10 months ago

After couple of days/week of working MeshCentral in Docker container suddenly it's stop responding via web interface. In logs there are so many info of JavaScript , NodeJS errors. I'm noticed 2 things: 1 - Size of meshcentral-powerdb is reached 300MB and stop growing 2 - After delete 3 files ( meshcentral-power.db , meshcentral-stats.db , meshcentral-events.db ) and reboot it works fine for couple of days/weeks again Here are attachment with logs mesherrors.txt

NiceGuyIT commented 10 months ago

The mesherrors.txt clearly shows the problem:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

You need to troubleshoot why it's running out of memory. If you expect us to help you, you need to follow the issue template that you deliberately removed. How many clients do you have? What is your host configuration? How much memory does your host have? Since you're using Docker, do have any Docker specific limitations, specifically with regard to memory? Are you using the official Docker image? What is your MeshCentral config (redact sensitive info)?

si458 commented 10 months ago

Out of memory seems to show ur server is low on memory (self explanatory)

I run dual core, 4gb ram for 100 devices, but I've know others run alot more devices on the same sorta spec, What's ur container spec?

Janiaking commented 10 months ago

Server is Synology 1619xs+ , 16GB RAM, Intel Xeeon 4-core 2.2GHz In Container Manager I limited meshcentral container to 8GB of RAM but i think it should be enough. On another machine with the same container and Synology i have 4 GB of RAM and there are no problems I'v got about 90 clients

For now i disable limitation of RAM for this container, will see what happen. but from what I remember, I did it a few weeks ago with the hope that it would be a problem. I'm using typhonragewind/meshcentral:latest.

silversword411 commented 10 months ago

I'm using typhonragewind/meshcentral:latest.

You're using someone else's docker config

NiceGuyIT commented 10 months ago

After delete 3 files ( meshcentral-power.db , meshcentral-stats.db , meshcentral-events.db ) and reboot it works fine for couple of days/weeks

Those databases seem self-explanatory. Have you checked how much history is being stored with the power, stats and events in MeshCentral? Have you changed the dbExpire settings in your config? The issue template specifically asks for a redacted version of your config.json and since you chose not to provide it, I'm choosing not to provide any more help than this. Good luck and I hope you find the problem.

        "dbExpire": {
          "type": "object",
          "properties": {
            "events": {
              "type": "integer",
              "default": 1728000,
              "description": "Amount of time in seconds that events are kept in the database."
            },
            "powerevents": {
              "type": "integer",
              "default": 864000,
              "description": "Amount of time in seconds that device power events are kept in the database."
            },
            "statsevents": {
              "type": "integer",
              "default": 2592000,
              "description": "Amount of time in seconds that server statistics are kept in the database."
            }
          }
        },
Janiaking commented 10 months ago

I dont know how much history because when container stops with this errors i have to cut this 3 files out until container will start properly

Thanks for help for everyone, i will try this config schema if issue come back to me.