Very high memory on customized agent after long time

[tomsik-radek]

Describe the bug I have encountered this on both Linux (Debian and Fedora) and Windows Server 2022. Right now I only see it on Windows, so I'm reporting this as a Windows issue. Agent is customized with my name

To Reproduce Steps to reproduce the behavior:

1. Install MeshCentral agent
2. Let the server run for a few days
3. See Agent memory usage slowly creeping up. Starts at like 200MB, can go up to a gig
4. After restarting the service it goes back down

Screenshots

Server Software (please complete the following information):

• OS: Debian 12
• Virtualization: Docker
• Network: LAN, MeshCentral in WAN mode behind NGINX Proxy Manager
• Version: typhonragewind/meshcentral:mongodb-1.1.20
• Node: Node.js v20.11.0

Remote Device (please complete the following information):

• Device: KVM in Proxmox
• OS: Windows Server 2022 Standard Eval
• Network: LAN, MeshCentral in WAN mode
• Current Core Version (if known): Current Core: Dec 9 2022, 1992526869

Additional context Add any other context about the problem here.

Your config.json file

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "_maintenanceMode": true,
    "cert": "[redacted]",
    "_SQLite3": true,
    "mongodb": "mongodb://mongodb:27017/mesh",
    "mongodbcol": "mesh",
    "WANonly": true,
    "sessionKey": "null",
    "port": 443,
    "aliasPort": 443,
    "redirPort": 80,
    "redirAliasPort": 80,
    "AgentPong": 300,
    "TLSOffload": "127.0.0.1",
    "_ignoreAgentHashCheck": true,
    "SelfUpdate": false,
    "AllowFraming": true,
    "WebRTC": false,
    "nice404": true,
    "allowHighQualityDesktop": true,
    "publicPushNotifications": false,
    "trustedProxy": "nginx-proxy-manager"
  },
  "domains": {
    "": {
      "_siteStyle": 2,
      "title": "MeshCentral",
      "title2": "[redacted]",
      "_titlePicture": "title-sample.png",
      "_loginPicture": "title-sample.png",
      "mobileSite": true,
      "maxDeviceView": 200,
      "_unknownUserRootRedirect": "https://www.youtube.com/watch?v=2Q_ZzBGPdqE",
      "nightMode": 0,
      "ipkvm": false,
      "minify": true,
      "newAccounts": false,
      "_welcomeText": "Sample Welcome Test.",
      "_welcomePicture": "mainwelcome.jpg",
      "_welcomePictureFullScreen": false,
      "meshMessengerTitle": "MeshMessenger",
      "_meshMessengerPicture": "messenger.png",
      "___hide__": "Sum of: 1 = Hide header, 2 = Hide tab, 4 = Hide footer, 8 = Hide title, 16 = Hide left bar, 32 = Hide back buttons",
      "hide": 4,
      "footer": "<a href='f'>Homepage</a>",
      "loginfooter": "This is not a public service.",
      "allowSavingDeviceCredentials": true,
      "guestDeviceSharing": true,
      "_AutoRemoveInactiveDevices": 37,
      "_DeviceSearchBarServerAndClientName": false,
      "_agentSelfGuestSharing": {
        "expire": 120
      },
      "certUrl": "https://meshcentral.[redacted]:443",
      "deviceMeshRouterLinks": {
        "rdp": true,
        "ssh": true,
        "scp": true
      },
      "_PreconfiguredScripts": [
        {
          "name": "Run NotePad as user",
          "file": "scripts/notepad.bat",
          "type": "bat",
          "runas": "user"
        },
        {
          "name": "Run NotePad as agent",
          "cmd": "notepad.exe",
          "type": "bat",
          "runas": "agent"
        },
        {
          "name": "Run echo",
          "cmd": "echo \"hello world\"",
          "type": "sh",
          "runas": "agent"
        },
        {
          "name": "Agent Update",
          "cmd": "agentupdate",
          "type": "agent"
        }
      ],
      "_PreconfiguredRemoteInput": [
        {
          "name": "CompanyUrl",
          "value": "[redacted]"
        },
        {
          "name": "Any Text",
          "value": "Any text\r"
        },
        {
          "name": "Welcome",
          "value": "Default welcome text"
        }
      ],
      "myServer": {
        "Backup": false,
        "Restore": false,
        "Upgrade": false,
        "ErrorLog": false,
        "Console": false,
        "Trace": false
      },
      "passwordRequirements": {
        "min": 8,
        "max": 128,
        "upper": 1,
        "lower": 1,
        "numeric": 1,
        "nonalpha": 1,
        "reset": 3650,
        "force2factor": false,
        "skip2factor": "127.0.0.1",
        "oldPasswordBan": 5,
        "banCommonPasswords": false,
        "twoFactorTimeout": 30
      },
      "twoFactorCookieDurationDays": 180,
      "agentInviteCodes": false,
      "_agentNoProxy": true,
      "geoLocation": false,
      "novnc": true,
      "mstsc": true,
      "ssh": true,
      "_WebEmailsPath": "/myserver/email-templates",
      "consentMessages": {
        "title": "MeshCentral Remote Access",
        "desktop": "{0} requesting remote desktop access. Grant access?",
        "terminal": "{0} requesting remote terminal access. Grant access?",
        "files": "{0} requesting remote files access. Grant access?",
        "consentTimeout": 60,
        "autoAcceptOnTimeout": false
      },
      "notificationMessages": {
        "title": "MeshCentral Remote Access",
        "desktop": "{0} started a remote desktop session.",
        "terminal": "{0} started a remote terminal session.",
        "files": "{0} started a remote files session."
      },
      "agentCustomization": {
        "displayName": "Tomsikr MeshCentral Agent",
        "description": "MeshCentral agent for remote monitoring, management and assistance.",
        "_companyName": "tomsikr llc",
        "serviceName": "tomsikrMeshAgent",
        "_image": "agent-logo.png",
        "_fileName": "_tomsikrMeshAgent",
        "filename": "MeshAgent"
      },
      "_agentFileInfo": {
        "_icon": "agent.ico",
        "_filedescription": "sample_filedescription",
        "fileversion": "1.0",
        "_internalname": "sample_internalname",
        "_legalcopyright": "sample_legalcopyright",
        "_originalfilename": "sample_originalfilename",
        "productname": "sample_productname",
        "productversion": "v0.1.2.3"
      },
      "assistantCustomization": {
        "title": "Tomsikr Mesh Assistant",
        "_image": "assistant-logo.png",
        "fileName": "tomsikrAssist"
      },
      "androidCustomization": {
        "title": "Tomsikr Mesh Android™",
        "_subtitle": "Product Subtitle™",
        "_image": "assistant-logo.png"
      },
      "_userAllowedIP": "127.0.0.1,192.168.1.0/24",
      "_userBlockedIP": "127.0.0.1,::1,192.168.0.100",
      "_agentAllowedIP": "192.168.0.100/24",
      "_agentBlockedIP": "127.0.0.1,::1",
      "___userSessionIdleTimeout__": "Number of user idle minutes before auto-disconnect",
      "_userSessionIdleTimeout": 60,
      "_userConsentFlags": {
        "desktopnotify": true,
        "terminalnotify": true,
        "filenotify": true,
        "desktopprompt": true,
        "terminalprompt": true,
        "fileprompt": true,
        "desktopprivacybar": true
      },
      "urlSwitching": true,
      "_desktopPrivacyBarText": "Privacy bar: {0}, {1}",
      "_limits": {
        "maxDevices": 3000,
        "maxUserAccounts": 100,
        "maxUserSessions": 100,
        "maxAgentSessions": 100,
        "maxSingleUserSessions": 10
      },
      "terminal": {
        "linuxshell": "login",
        "launchCommand": {
          "linux": "clear\necho \"Hello Linux\"\n",
          "darwin": "clear\necho \"Hello MacOS\"\n",
          "freebsd": "clear\necho \"Hello FreeBSD\"\n"
        }
      },
      "redirects": {
        "homepage": "[redacted]"
      },
      "_yubikey": {
        "id": "0000",
        "secret": "xxxxxxxxxxxxxxxxxxxxx",
        "_proxy": "http://myproxy.domain.com:80"
      },
      "_httpHeaders": {
        "Strict-Transport-Security": "max-age=360000",
        "x-frame-options": "SAMEORIGIN"
      },
      "_agentConfig": [ "webSocketMaskOverride=1", "coreDumpEnabled=1" ],
      "_assistantConfig": [ "disableUpdate=1" ],
      "_sessionRecording": {
        "_onlySelectedUsers": true,
        "_onlySelectedUserGroups": true,
        "_onlySelectedDeviceGroups": true,
        "_filepath": "C:\\temp",
        "_index": true,
        "_maxRecordings": 10,
        "_maxRecordingDays": 15,
        "_maxRecordingSizeMegabytes": 3,
        "__protocols__": "Is an array: 1 = Terminal, 2 = Desktop, 5 = Files, 100 = Intel AMT WSMAN, 101 = Intel AMT Redirection, 200 = Messenger",
        "protocols": [ 1, 2, 101 ]
      }
    }
  }
}

"Custom NGINX Configuration" in NGINX Proxy Manager

proxy_set_header CF-Connecting-IP $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

[si458]

yes we are aware, no idea why it happens, im going to try and look into it now i can build the agents myself for all platforms it happens on linux too, a simple fix is to monitor its usage and if above say 500mb, then restart the service

[tomsik-radek]

yes we are aware, no idea why it happens, im going to try and look into it now i can build the agents myself for all platforms it happens on linux too, a simple fix is to monitor its usage and if above say 500mb, then restart the service

Would there be a way to do this internally? I don't have any monitoring software I could use. At most restart the service every x days. And I assume there is no way to "remote into" the agent? One of the dependencies has to be leaking, right?

[tomsik-radek]

Would a ProcessExplorer memory dump help? Do you know anyone who can read it? The file is almost a gig btw.

[si458]

you can run service restart from the console tab of the device in meshcentral, this will restart the service 👍

[rawring-loser]

I got same problem with high memory usage of meshagent on all platform: Linux, Windows, VPS... I haven't heard problem from Mesh server hosting on Public IP. From the post of @krayon007 and Meshcentral documentation:

The “AgentPong” line instructs the server to send data to the agent each 300 seconds and the agent by default will send data to the server every 120 seconds. As long as NGINX timeouts are longer than this, connections should remain open.

So agent try to send data every 120 sec to server with only 60 sec (default value of NGINX) timeout of Nginx Proxy Manager (NPM), making connection still open, maybe a bug. Meshagent will create more new connections and memory leaked. My Meshcentral server put behind NPM, I tried to put this in advanced tab, greater than 120 sec:

• proxy_read_timeout 200s;

• proxy_connect_timeout 200s;

• proxy_send_timeout 200s;

Seem memory problem gone. I haven't tried on other reversed proxy but I think all are same problem. Hope it help.

[silversword411]

I have don't have either ping or pong enabled.

{
  "settings": {
    "Cert": "snip",
    "WANonly": true,
    "Port": 4430,
    "AliasPort": 443,
    "RedirPort": 800,
    "AllowLoginToken": true,
    "AllowFraming": true,
    "AllowHighQualityDesktop": true,
    "TlsOffload": "127.0.0.1",
    "agentCoreDump": false,
    "Compression": true,
    "WsCompression": true,
    "AgentWsCompression": true,
    "PublicPushNotifications": true,
    "WebRTC": true,
    "webrtcConfig": {
      "iceServers": [
        { "urls": "stun:stun.services.mozilla.com" },
        { "urls": "stun:stun.l.google.com:19302" }
      ]
    },
    "MaxInvalidLogin": {
      "time": 5,
      "count": 5,
      "coolofftime": 30
    },
    "postgres": {
      "user": "snip",
      "password": "snip",
      "port": "5432",
      "host": "localhost"
    }
  },
  "domains": {
    "": {
      "Title": "snip Support",
      "NewAccounts": false,
      "CertUrl": "https://snip.com:443/",
      "GeoLocation": true,
      "allowSavingDeviceCredentials": true,
      "CookieIpCheck": false,
      "mstsc": true,
      "altmessenging": {
        "name": "Jitsi",
        "url": "https://meet.jit.si/myserver-{0}"
      },
      "SessionRecording": {
        "onlySelectedUsers": true,
        "onlySelectedUserGroups": true,
        "onlySelectedDeviceGroups": true,
        "index": true,
        "maxRecordingDays": 120,
        "protocols": [
          1,
          2,
          200
        ]
      }
    }
  }

I do use this to monitor and restart mesh agent. https://github.com/amidaware/community-scripts/blob/main/scripts_staging/Win_Mesh_CheckForAgentProblems.ps1

Agreed it's something to do with agents disconnecting from the mesh a lot, either sleep, stuttering network connect or the link

[silversword411]

"_unknownUserRootRedirect": "https://www.youtube.com/watch?v=2Q_ZzBGPdqE",

I see you copied the advanced config example for the most part...I thought that was a good addition :)

[tomsik-radek]

"_unknownUserRootRedirect": "https://www.youtube.com/watch?v=2Q_ZzBGPdqE",

I see you copied the advanced config example for the most part...I thought that was a good addition :)

How did I miss that one? It needs an XcQ link.

[tomsik-radek]

I got same problem with high memory usage of meshagent on all platform: Linux, Windows, VPS... I haven't heard problem from Mesh server hosting on Public IP. From the post of @krayon007 and Meshcentral documentation:

The “AgentPong” line instructs the server to send data to the agent each 300 seconds and the agent by default will send data to the server every 120 seconds. As long as NGINX timeouts are longer than this, connections should remain open.

So agent try to send data every 120 sec to server with only 60 sec (default value of NGINX) timeout of Nginx Proxy Manager (NPM), making connection still open, maybe a bug. Meshagent will create more new connections and memory leaked. My Meshcentral server put behind NPM, I tried to put this in advanced tab, greater than 120 sec:

• proxy_read_timeout 200s;

• proxy_connect_timeout 200s;

• proxy_send_timeout 200s;

Seem memory problem gone. I haven't tried on other reversed proxy but I think all are same problem. Hope it help.

Will try and report back, thanks

[silversword411]

It needs an XcQ link.

I considered that...but decided to go another way 🤣

[tomsik-radek]

you can run service restart from the console tab of the device in meshcentral, this will restart the service 👍

yes I'm aware, but there doesn't seem to be a way to do it automatically other than a script

Linux is easy, just a cron to restart systemtl, but Windows Task Scheduler can't restart a service or a process.

[silversword411]

Sure it can, just script the powershell above. I use TRMM to monitor/fix for now till we find all the leaks.

but Windows Task Scheduler can't restart a service or a process.

[tomsik-radek]

Sure it can, just script the powershell above. I use TRMM to monitor/fix for now till we find all the leaks.

but Windows Task Scheduler can't restart a service or a process.

TRMM? By the name I'm guessing it's an RMM software, but it's too generic for google

[silversword411]

TRMM?

https://github.com/amidaware/tacticalrmm

[tomsik-radek]

Ah, TRMM. Yeah, I have been eyeing that for a while. Shame it's hardcoded for port 443. I will need to try this out. Hopefully the RDP experience is less crap than MeshCentral

[si458]

@tomsik-radek tacticalrmm use meshcentral for its remote control and i believe also the rdp, if meshcentrals RDP is CRAP, then plz do explain or open an issue if you are having issues!

[tomsik-radek]

@tomsik-radek tacticalrmm use meshcentral for its remote control and i believe also the rdp, if meshcentrals RDP is CRAP, then plz do explain or open an issue if you are having issues!

I apologize. I shouldn't have said it like that. I do have many issues with the RDP implementation, but as you said, those belong in a separate tickets. Meaningless insults like this are... meaningless. There is no use in saying "this sucks" without saying why. I'm aware TacticalRMM is built on top of MeshCentral.

[silversword411]

UI is always difficult to perfect.

Mesh working in the constraints of a browser makes it even harder.

TRMM isn't built on mesh. TRMM uses and integrates with mesh via its software interface. It uses it for the desktop, file and remote terminal features, everything else is completely separate via a separate agent.

[si458]

@Shivangraj yes don't worry about that, if u check with ps aux u can see it did restart, the is just a warning

Need to look into why it shows that error?

[tomsik-radek]

service restart

getting error for ubuntu

`> service restart Command returned an exception error: waitExit() aborted because thread is exiting

service restart Command returned an exception error: waitExit() aborted because thread is exiting sudo service restart`

Will it fail if you restart it with systemctl?