Large File Downloads Fail At Random States of Completition

[randominternetuser]

In an almost identical situation to #2911 according to the trace logs the FileRelay will randomly disconnect without error sometimes 7MB into the file, sometimes hundreds of MBs into a file. It does not seem to be effected by WebRTC or not, local network or via a reverse proxy.

Steps to reproduce the behavior:

1. Open file manager and download any decently sized file.

Server Ubuntu 22.04 LAN/WAN HAProxy with SSL offloading Full AMT 1.1.21

Device

Desktop/Laptop OS Windows 10/11 Local & Remote WAN

Config File - Sanitized - Bare metal install

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "Cert": "meshcentral.example.com",
    "Port": 4430,
    "AliasPort": 443,
    "RedirPort": 0,
    "AgentPong": 300,
    "TlsOffload": "ROUTER_IP",
    "TrustedProxy": "ROUTER_IP",
    "MpsPort": 44330,
    "MpsAliasHost": "meshcentral.example.com",
    "MpsAliasPort": 4433,
    "MpsTlsOffload": false,
    "MpsHighSecurity": true,
    "AgentInviteCodes": true,
    "MongoDb": "mongodb://127.0.0.1:27017/meshcentral",
    "RelayPort": 453,
    "desktopMultiplex": true,
    "Compression": false,
    "WsCompression": false,
    "AgentWsCompression": false,
    "CookieIpCheck": false,
    "Minify": true,
    "LANonly": false,
    "WebRTC": false,
    "nice404": true,
    "clickOnce": false,
    "publicPushNotifications": true,
    "allowHighQualityDesktop": true,
    "plugins": { "enabled": true },
    "NpmPath": "/usr/bin/node /usr/bin/npm",
    "maxInvalidLogin": {
      "time": 10,
      "count": 5,
      "coolofftime": 30
    }
  },
  "domains": {
    "": {
      "CertUrl": "https://ROUTER_IP:443/",
      "AgentConfig": ["webSocketMaskOverride=1"],
    }
  }
}

[troymroberts]

I had this issue previously. I chased it forever with no real resolution.

I ended up upgrading (via re-install and backup/restore config) my MC server (same cloud provider) to one with more resources and it went away. My performance issues with client devices also went away. I initially did not think the host was the issue because top never showed more than 50% cpu or memory utilization for the prior server.

Add more resources to MC and it went away

[Otsegox]

I had this issue previously. I chased it forever with no real resolution.

I ended up upgrading (via re-install and backup/restore config) my MC server (same cloud provider) to one with more resources and it went away. My performance issues with client devices also went away. I initially did not think the host was the issue because top never showed more than 50% cpu or memory utilization for the prior server.

Add more resources to MC and it went away

How many more? You provided no context. If you had 1 core and 2 gigs of Ram, I get it. What if you have 12 cores and 64GB of RAM? I have this issue, but your comment doesn't tell me how much I should increase.

[ghplw]

I'm seeing this problem as well. Client, server and all the networks that connect them seem stable and not even close to overloaded.

[si458]

@ghplw one thing to check is make sure webrtc is set as false or removed from config.json as it causes issues i will have to have alook at this issue again at some point as ive never been able to replicate the bug? whats the sorta size file you are copying? is the device local or remote? is ur meshcentral in the cloud or on site? do you use a reverse proxy of some form (nginx, cloudflare, etc)

[Otsegox]

Size - Think ISO 4 gigs. Or maybe a large zip file - 15 gigs Local server is local. Could be on the same VLAN and or Switch. Does not matter. No reverse proxy.

The agent just crashes after attempting. It takes time, but it crashes. I haven't tried in a long time, but the agent also hasn't been updated in ages. It also struggles with making zip files of multiple files... If there are a lot of them. It also struggles to list a folder with thousands of files, too.

[si458]

@Otsegox oh wow 15GB zip is a big chunky file haha have you tried using the batch file upload method but on a single device? with the files tab, we send the file over the websocket to your meshcentral server and it then relays it to the remote machine, is something happens while its happening it can cause issues

if you use the batch file upload method, it will upload the file in the background to meshcentral server using a POST request (NOT WEBSOCKETS), then once its finished uploading, it will tell the remote device, YOH DOWNLOAD THIS FILE FROM THIS URL ON ME, and it makes the download a little bit quicker, only downside to this method is you would need lot of memory on your meshcentral server for a 15GB file because it has to store the file on its disk somewhere temporarily then ask the remote devices to download that file

[Otsegox]

@si458 I have not. The files tab is... Confusing. I see how I can upload to it, and it does it well (400-500Mbps)(drag and drop into the files tab yields about 40Mbps), but what to do after that? I don't understand how I can tell an agent to download it to the computer. Being the agent doesn't have the interface, how does the agent know to download it? And if there are 50 computers in that group, how do they know which computer is supposed to download it?