Multiple devices file upload results in a 1KB file

catalinchertes commented 2 years ago

When we upload a 19MB file to multiple devices using the batch feature. The folder is created and the file is uploaded, but when we check the size, it changes to 1KB, so something is happening with the file after upload since the original size is 19MB.

When we do it from the single device file upload option, it works fine and the file keeps it's original size.

We could really use this feature as we have over 700 remote desktops with mesh agents. This tool is so amazing and we are very thankful for it, thanks for your hard work.

This is the server log upon upload: (I removed the host name for privacy).

POST

/uploadfilebatch.ashx

{"host":"","connection":"keep-alive","content-length":"19764126","cache-control":"max-age=0","sec-ch-ua":"\"Google Chrome\";v=\"95\", \"Chromium\";v=\"95\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","origin":"null","content-type":"multipart/form-data; boundary=----WebKitFormBoundary6gIonySgFlG8ltar","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"iframe","accept-encoding":"gzip, deflate, br","accept-language":"es-ES,es;q=0.9,ca;q=0.8,ro;q=0.7,it;q=0.6","cookie":"_ga=GA1.2.781225311.1628789010; xid=eyJ1c2VyaWQiOiJ1c2VyLy9hY2F0YWxpbiIsImlwIjoiMTkyLjE2OC4xMC4xNiIsInQiOjI3Mjg2MjE4fQ==; xid.sig=Cchi5tgZBMBWkAkqO3gCbYX7W0w"}

AntonAndreevichMoroz commented 2 years ago

@Ylianst could this be part of a common problem uploading files to multiple devices? #3259 Our mesh instance crashes with this action.

catalinchertes commented 2 years ago

I tried to upload a powershell script through the action group. The original script is as follows:

$taskName = "Shutdown Computer"

$description = "Shuts computer down daily at 2300 MTN"

$taskAction = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument 'Stop-Computer -Force'

$taskTrigger = New-ScheduledTaskTrigger -Daily -At 9PM

Register-ScheduledTask -TaskName $taskName -Action $taskAction -Trigger $taskTrigger -Description $description`

The file once uploaded has this content:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /agentdownload.ashx</pre>
</body>
</html>

Ylianst commented 2 years ago

Hi. Sorry for the delay. The way this feature works is that the MeshCentral server will cache your file and then generate a bunch of URL's that are given to all agents. The agents will then download the URL and save the file at the desired target location.

As a result, your server should see a bunch of HTTP GET requests from the agents on the URL /agentdownload.ashx.

One obvious possibility here is that you have a reverse proxy of firewall in front of your MeshCentral server that is blocking the URL from being downloaded. Whatever is blocking the URL is returning a web page saying the file can't be downloaded and that is what the agent is saving. So, that is why the resulting file is 1k.

I just did a test and uploaded a file to 4 computers at once. I also enabled tracing for "Web Server Requests" in the "My Server" / "Trace" tab. In my case, it worked. The files got sent and you can see 4 requests on the HTTP server.

Each request has an encrypted time-limited cookie, so it's not possible for just anyone to get you files.

If you try to access https://yourserver/agentdownload.ashx from a regular browser, you should see this:

I am thinking you have something blocking this URL from the agents, but not knowing your network setup I will stop here and see if I am in the right ballpark.

Ylianst commented 2 years ago

@AntonAndreevichMoroz Thanks for the reminder, I just fixed #3259.

catalinchertes commented 2 years ago

Thanks for the quick reply @Ylianst!

I've turned on the logs and sent a bunch of files, in this case an .msi installer which is 18mb in size. The webrequest shows correctly and generates no errors, but when I check the file uploaded on the remote computer, they are 750kb in size, it seems like it doesn't download the entire file.

We don't have anything blocking the requests and we tried with localhost machines on the same local network, but it still does the same thing. I will investigate on this issue anyways to see if there's something on our end that might be blocking the request.

Have you tried copying a file, like an executable, and running it on the remote computer? Or checking the size to see if it's the same? If you could confirm that to me.

Thank you very much, This project never ceases to amaze me!

catalinchertes commented 2 years ago

We did two tests:

The first test was two computers on different lans separated by the firewall, we checked the traffic which is encrypted and we could see it passing through the port 777 but the client fails to complete the get request.

The second test was two computers on the same lan, with the same results (no firewall in between), in both cases, the files we send as a batch upload to various machines, end up being a html file with the failed get request.

Ylianst commented 2 years ago

1 - Here is a simple test, initiate a batch upload to one computer and look in the meshcentral-files/tmp folder. You should see a file with an odd name and it should be the same size as the file you uploaded. If it's not, the problem is the file upload to the server.

2 - If the file in tmp is ok, since all of the URL's are time limited, you can initiate a download, capture one of the URL's using the tracing like you did above and click on one, copy it and paste it in the browser. See what you download.

Since all your files where 750kb in size, I am hoping that the first test fails. I will test the upload to the server now, maybe there is a problem there.

catalinchertes commented 2 years ago

I tried both cases:

1 - I uploaded a batch file, the content in the TMP folder is the same size as the original (ok).

2 - I copied the URL (/agentdownload.ashx ...) and pasted it in my browser, it downloaded a agentdownload.ashx file the same size as the original file. (This was tested on the computer where I sent the batch file upload).

For testing purposes I changed the name of the file and extension to the original file name and extension and it worked properly and had the same size.

Doing it manually as you just shown me works perfectly as intended.

Ylianst commented 2 years ago

I just tested a 117 megabyte file batch uploaded to a single computer and it worked for me. The file was fully into meshcentral-files/tmp and the remote device downloaded it correctly. Certainly interested in your results.

(Ok, you just posted your results)

Ylianst commented 2 years ago

Thank you for your test results. Looks like the problem is the agent downloading the URL. I was hoping one of the two tests would fail (arg!).

catalinchertes commented 2 years ago

Yeah, it seems like the problem lays in the agent not being able to download the file.

Thanks for the quick reply!

catalinchertes commented 2 years ago

Hey, I'm sure you are busy with all the other issues and feature requests, but is there any update on this issue? This feature would be really useful. Thanks.

silversword411 commented 2 years ago

Sounds like an in-network AV sliced the connection.

catalinchertes commented 2 years ago

That's not the case for me, I've done tests without anything blocking the connection in between the server and the client and still no success. I can manually download the file on the client by copying the get URL and pasting it on the client browser, it downloads the file completely. But doing it automatically fails for some reason.

si458 commented 9 months ago

issue seems to be resolved now in latest build 1.1.20 so closing, if still happening, plz reply with full details of whats now happening

catalinchertes commented 9 months ago

I updated the server to the latest build and the issue is still happening for me.

I try to upload a file using the batch upload feature and the result is a 1KB size file that If I open with notepad the content is a "Cannot Get" html file.

I tried bypassing all firewall and antivirus between the server and the agent and the issue still happens.

I would really like to have this feature working as we have many computers that we manage through meshcentral and every now and then we need to upload the same file to various computers and instead we need to go one by one.

If I enable the server requests trace and I copy the /agentdownload.ashx url manually and paste it in the browser on the computer that has the meshagent installed, it successfully manages to download that file, but when it does it through the agent it fails.

si458 commented 9 months ago

That to me screams a permission issue? Does the user account on MC have full admin rights or the upload files rights?

silversword411 commented 9 months ago

Can't have a 1kb file if you don't have permissions. More like a sliced connection by security appliance/av

AntonAndreevichMoroz commented 9 months ago

I'm sure it's not an antivirus or firewall. If you individually send a file for 1 device, then everything works. But if you do it en masse, it doesn’t work. This means with permission and blocking and everything is in order.

si458 commented 9 months ago

i cant seem to replicate this issue, so the must be something thats special about the remote device, do you have the agent installed in a different folder? is the agent defo installed and not connected? remote OS version/build number? do you have a reverse proxy infront of meshcentral?

catalinchertes commented 9 months ago

I tried it on linux, windows 7, windows 10 and windows 11 devices, they all fail, I tried it on devices that are on the local LAN and also on devices that are connecting from the WAN and neither work.

The agent install location is the default one and the account running the service has sufficient permissions on.

There's no reverse proxy between the clients and the server and no antivirus.

I'm running out of ideas and things to try.

si458 commented 9 months ago

@catalinchertes can you share ur config.json? (hide sensitive) as the MUST be an issue with your meshcentral setup are you also using the latest nodejs? 18 or above? whats meshcentral running on server side?

catalinchertes commented 9 months ago

Meshcentral server is running on a Ubuntu 22.04.3 LTS virtual machine.

Meshcentral version:

Here is the NodeJS Version:

Here is the config.json file:

``` { "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json", "__comment__": "This is a sample configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.", "settings": { "_noagentupdate": true, "cert": "censored", --> removed because private info "mongoDb": "mongodb://127.0.0.1:27017", "mongoDbName": "meshcentral", "_mongoDbChangeStream": true, "mongoDbBulkOperations": true, "_WANonly": true, "_LANonly": true, "_sessionKey": "MyReallySecretPassword1", "_sessionSameSite": "strict", "_dbEncryptKey": "MyReallySecretPassword2", "_dbRecordsEncryptKey": "MyReallySecretPassword", "_dbRecordsDecryptKey": "MyReallySecretPassword", "__dbExpire": "Amount of time to keep various events in the database, in seconds. Below are the default values.", "_dbExpire": { "events": 1728000, "powerevents": 864000, "statsevents": 2592000 }, "port": 443, "_portBind": "127.0.0.1", "_aliasPort": 4430, "_redirPort": 80, "_redirPortBind": "127.0.0.1", "_redirAliasPort": 80, "agentPort": 777, "_agentPortBind": "127.0.0.1", "_agentAliasPort": 777, "_agentAliasDNS": "agents.myserver.mydomain.com", "_agentPortTls": true, "_exactPorts": true, "_allowLoginToken": true, "_allowFraming": true, "_cookieIpCheck": false, "_cookieEncoding": "hex", "_compression": true, "_wscompression": false, "_agentwscompression": true, "_agentsInRam": false, "_webRTC": true, "_nice404": false, "_selfUpdate": true, "_browserPing": 60, "_browserPong": 60, "_agentPing": 60, "_agentPong": 60, "_agentIdleTimeout": 150, "_meshErrorLogPath": "c:\\tmp", "_npmPath": "c:\\npm.exe", "_npmProxy": "http://1.2.3.4:80", "_allowHighQualityDesktop": true, "_webPush": { "email": "xxxxx@xxxxx.com" }, "_publicPushNotifications": true, "_desktopMultiplex": true, "_userAllowedIP": "127.0.0.1,192.168.1.0/24", "_userBlockedIP": "127.0.0.1,::1,192.168.0.100", "_agentAllowedIP": "192.168.0.100/24", "_agentBlockedIP": "127.0.0.1,::1", "_authLog": "c:\\temp\\auth.log", "_manageAllDeviceGroups": [ "user//admin" ], "_manageCrossDomain": [ "user//admin" ], "_localDiscovery": { "name": "Local server name", "info": "Information about this server" }, "_tlsOffload": "127.0.0.1::1", "_trustedProxy": "127.0.0.1::1", "mpsPort": 4433, "_mpsPortBind": "127.0.0.1", "_mpsAliasPort": 4433, "_mpsAliasHost": "mps.mydomain.com", "_mpsTlsOffload": true, "_no2FactorAuth": true, "_log": "main,web,webrequest,cert", "_syslog": "meshcentral", "_syslogauth": "meshcentral-auth", "_syslogjson": "meshcentral-json", "_webrtcConfig": { "iceServers": [ { "urls": "stun:stun.services.mozilla.com" }, { "urls": "stun:stun.l.google.com:19302" } ] }, "_autoBackup": { "_mongoDumpPath": "C:\\Program Files\\MongoDB\\Server\\4.2\\bin\\mongodump.exe", "backupIntervalHours": 24, "keepLastDaysBackup": 10, "zipPassword": "MyReallySecretPassword3", "_backupPath": "C:\\backups", "_googleDrive": { "folderName": "MeshCentral-Backups", "maxFiles": 10 }, "_webdav": { "_url": "https://server/remote.php/dav/files/xxxxx@server.com/", "_username": "user", "_password": "pass", "_folderName": "MeshCentral-Backups", "_maxFiles": 10 } }, "_redirects": { "meshcommander": "https://www.meshcommander.com/" }, "maxInvalidLogin": "Time in minutes, max amount of bad logins from a source IP in the time before logins are rejected.", "maxInvalidLogin": { "time": 10, "count": 10, "coolofftime": 10 }, "watchDog": { "interval": 100, "timeout": 400 }, "_AmtProvisioningServer": { "port": 9971, "deviceGroup": "mesh//xxxxxxxxxxxxxxxxxxxxx", "newMebxPassword": "amtpassword", "trustedFqdn": "sample.com", "ip": "192.168.1.1" }, "plugins": { "enabled": true } }, "_domaindefaults": { "__comment__": "Any settings in this section is used as default setting for all domains", "title": "Remote Desktop Server [MESH]", "footer": "Default page footer", "newAccounts": false }, "domains": { "": { "siteStyle": 2, "title": "censored", --> removed because private info "title2": "", "titlePicture": "default.png", "agentInviteCodes": true, "_loginPicture": "title-sample.png", "_userQuota": 1048576, "_meshQuota": 248576, "_minify": true, "showNotesPanel": true, "_guestDeviceSharing" : false, "_loginKey": [ "abc", "123" ], "_newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "_userNameIsEmail": true, "_newAccountEmailDomains": [ "sample.com" ], "_newAccountsRights": [ "nonewgroups", "notools" ], "_welcomeText": "Sample Text on Login Page.", "welcomePicture": "censored", --> removed because private info "PasswordRequirements": censored", --> removed because private info "_maxDeviceView": 100, "_welcomePictureFullScreen": false, "_meshMessengerTitle": "MeshMessenger", "_meshMessengerPicture": "messenger.png", "___hide__": "Sum of: 1 = Hide header, 2 = Hide tab, 4 = Hide footer, 8 = Hide title, 16 = Hide left bar", "_hide": 4, "_footer": "Twitter", "_loginfooter": "This is a private server.", "_certUrl": "https://192.168.2.106:443/", "_altMessenging": { "name": "Jitsi", "url": "https://meet.jit.si/myserver-{0}" }, "_deviceMeshRouterLinks": { "rdp": true, "ssh": true, "scp": true, "extralinks": [ { "name": "HTTP", "protocol": "http", "port": 80, "_ip": "192.168.1.100", "_filter": [ "mesh//xxxx", "node//xxxx" ] }, { "name": "HTTPS", "protocol": "https", "port": 443 } ] }, "myServer": { "Backup": true, "Restore": true, "Upgrade": true, "ErrorLog": true, "Console": true, "Trace": true }, "_passwordRequirements": { "min": 8, "max": 128, "upper": 1, "lower": 1, "numeric": 1, "nonalpha": 1, "reset": 90, "force2factor": true, "skip2factor": "127.0.0.1,192.168.2.0/24", "oldPasswordBan": 5, "banCommonPasswords": true }, "twoFactorCookieDurationDays": 7, "_agentInviteCodes": true, "_agentNoProxy": true, "_geoLocation": true, "_novnc": false, "_mstsc": true, "_ssh": true, "_WebEmailsPath": "/myserver/email-templates", "consentMessages": { "title": "censored", --> removed because private info "desktop": "censored", --> removed because private info "terminal": "censored", --> removed because private info "files": "censored", --> removed because private info }, "notificationMessages": { "title": "censored", --> removed because private info "desktop": "censored", --> removed because private info "terminal": "censored", --> removed because private info "files": "censored", --> removed because private info }, "agentCustomization": { "displayName": "censored", --> removed because private info "description": "censored", --> removed because private info "companyName": "censored", --> removed because private info "_serviceName": "Mesh Agent", "_fileName": "meshagent" }, "_userAllowedIP": "127.0.0.1,192.168.1.0/24", "_userBlockedIP": "127.0.0.1,::1,192.168.0.100", "_agentAllowedIP": "192.168.0.100/24", "_agentBlockedIP": "127.0.0.1,::1", "_orphanAgentUser": "admin", "___userSessionIdleTimeout__": "Number of user idle minutes before auto-disconnect", "_userSessionIdleTimeout": 30, "_userConsentFlags": { "desktopnotify": true, "terminalnotify": true, "filenotify": true, "desktopprompt": true, "terminalprompt": true, "fileprompt": true, "desktopprivacybar": true }, "_urlSwitching": false, "_desktopPrivacyBarText": "Privacy bar: {0}, {1}", "_limits": { "_maxDevices": 100, "_maxUserAccounts": 100, "_maxUserSessions": 100, "_maxAgentSessions": 100, "maxSingleUserSessions": 10 }, "_amtScanOptions": [ "LabNetwork 192.168.15.0/23", "SalesNetwork 192.168.8.0/24" ], "_amtAcmActivation": { "log": "amtactivation.log", "certs": { "mycertname": { "certfiles": [ "amtacm-leafcert.crt", "amtacm-intermediate1.crt", "amtacm-intermediate2.crt", "amtacm-rootcert.crt" ], "keyfile": "amtacm-leafcert.key" } } }, "_amtManager": { "adminAccounts": [{ "user": "admin", "pass": "MyP@ssw0rd" }], "environmentDetection": [ "domain1.com", "domain2.com", "domain3.com", "domain4.com" ], "wifiProfiles": [ { "name": "Profile1", "ssid": "MyStation1", "authentication": "wpa2-psk", "encryption": "ccmp-aes", "password": "MyP@ssw0rd" } ] }, "_redirects": { "meshcommander": "https://www.meshcommander.com/" }, "_yubikey": { "id": "0000", "secret": "xxxxxxxxxxxxxxxxxxxxx", "_proxy": "http://myproxy.domain.com:80" }, "_httpHeaders": { "Strict-Transport-Security": "max-age=360000", "x-frame-options": "SAMEORIGIN" }, "_agentConfig": [ "webSocketMaskOverride=1", "coreDumpEnabled=1" ], "_sessionRecording": { "_onlySelectedDeviceGroups": true, "_filepath": "C:\\temp", "index": true, "_maxRecordings": 10, "_maxRecordingSizeMegabytes": 1000, "__protocols__": "Is an array: 1 = Terminal, 2 = Desktop, 5 = Files, 100 = Intel AMT WSMAN, 101 = Intel AMT Redirection, 200 = Messenger", "protocols": [ 1, 2, 101 ] }, "__authStrategies": { "__comment__": "This section is used to allow users to login using other accounts. You will need to get an API key from the services and register callback URL's", "_twitter": { "_callbackurl": "https://server/auth-twitter-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxx", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, "_google": { "_callbackurl": "https://server/auth-google-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxx" }, "_github": { "_callbackurl": "https://server/auth-github-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxx", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, "_reddit": { "_callbackurl": "https://server/auth-reddit-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxx", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, "_azure": { "_callbackurl": "https://server/auth-azure-callback", "newAccounts": false, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "tenantid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, "_jumpcloud": { "_callbackurl": "https://server/auth-jumpcloud-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "entityid": "meshcentral", "idpurl": "https://sso.jumpcloud.com/saml2/saml2", "cert": "jumpcloud-saml.pem" }, "_saml": { "_callbackurl": "https://server/auth-saml-callback", "_disableRequestedAuthnContext": true, "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "_newAccountsRights": [ "nonewgroups", "notools" ], "entityid": "meshcentral", "idpurl": "https://server/saml2", "cert": "saml.pem" } } }, "_customer1": { "_dns": "customer1.myserver.com", "_title": "Customer1", "_title2": "TestServer", "_newAccounts": 1, "_auth": "sspi", "__auth": "ldap", "_LDAPUserName": "gecos", "_LDAPUserKey": "uid", "_LDAPUserEmail": "otherMail", "_LDAPPptions": { "url": "test", "anne": { "gecos": "Anne O'Nyme", "displayName": "O Nyme anne", "uid": "anneonyme", "mail": "anneonyme@example.com", "email": "anneonyme@example.com", "otherMail": [ "other.anneonyme@example.com", "anneonyme@example.com" ] }, "so": { "displayName": "Sticker Sophie", "gecos": "Sophie Sticker", "uid": "ssticker", "mail": "ssticker@example.com", "email": "ssticker@example.com", "otherMail": [ "other.ssticker@example.com", "ssticker@example.com" ] } }, "__LDAPOptions": { "URL": "ldap://1.2.3.4:389", "BindDN": "CN=svc_meshcentral,CN=Users,DC=meshcentral,DC=local", "BindCredentials": "Password.1", "SearchBase": "DC=meshcentral,DC=local", "SearchFilter": "(sAMAccountName={{username}})" }, "_footer": "Test", "_certUrl": "https://192.168.2.106:443/" }, "_info": { "_share": "C:\\ExtraWebSite" } }, "_letsencrypt": { "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.", "email": "censored", --> removed because private info "names": "censored", --> removed because private info "production": true }, "_peers": { "serverId": "server1", "servers": { "server1": { "url": "wss://192.168.2.133:443/" }, "server2": { "url": "wss://192.168.1.106:443/" } } }, "_smtp": { "host": "smtp.myserver.com", "port": 25, "from": "myemail@myserver.com", "__tls__": "When 'tls' is set to true, TLS is used immidiatly when connecting. For SMTP servers that use TLSSTART, set this to 'false' and TLS will still be used.", "tls": false, "___tlscertcheck__": "When set to false, the TLS certificate of the SMTP server is not checked.", "_tlscertcheck": false, "__tlsstrict__": "When set to true, TLS cypher setup is more limited, SSLv2 and SSLv3 are not allowed.", "_tlsstrict": true }, "_sendgrid": { "from": "myemail@myserver.com", "apikey": "***********" }, "_sms": { "provider": "twilio", "sid": "ACxxxxxxxxx", "auth": "xxxxxxx", "from": "+1-555-555-5555" }, "__sms": { "provider": "plivo", "id": "xxxxxxx", "token": "xxxxxxx", "from": "1-555-555-5555" } } ```

si458 commented 9 months ago

@catalinchertes all looks ok? but you do have plugins enabled and an unknown value watchdog ? you could try removing these those values see if it works? also you arent using letsencrypt? so do you have ur own SSL certificates set somewhere else? also the function for agentdownload.ashx is very basic,

checkUserIpAddress isnt in userblockedip
check domain is valid
decode cookie and check its within 5mins
check if filename valid you asked for
if any above is invalid then 404 otherwise send file

one thing you can try is running meshcentral with just web debug and watch the request for agentdownload.ashx node node_modules/meshcentral --debug web OR if adventurious run in FULL DEBUG mode node node_modules/meshcentral --debug

si458 commented 4 months ago

this has now been fixed by https://github.com/Ylianst/MeshCentral/pull/6155 you can test this by using the docker master tag image OR npm install Ylianst/MeshCentral to install the master branch repo which includes the fix

inzi commented 2 months ago

FWIW, I'm having this same issue and have yet to resolve. Is there a way to log this request on the agent side to see exactly where it's trying to make the call to?

si458 commented 2 months ago

@inzi this should of been fixed already. What version of meshcentral are you running? Can u share ur config.json? What nodejs version?

inzi commented 2 months ago

MC: version 1.1.27 Node: v18.17.1

my config.json:

``` { "settings": { "cert": "meshcentral.mydomain.org", "wanonly": true, "_minify": true, "port": 480, "aliasport": 443, "redirport": 88, "rediraliasport": 80, "agentporttls": true, "agentport": 481, "tlsoffload": "127.0.0.1,10.50.5.10,10.50.5.97,10.50.5.1,::1", "trustedproxy": "127.0.0.1,10.50.5.10,10.50.5.97,10.50.5.1,::1", "cookieipcheck": false, "cookieencoding": "hex", "authlog": "C:\\Program Files\\Open Source\\MeshCentral\\meshcentral-logs\\auth.log", "log": [ "cookie", "dispatch", "main", "peer", "web", "webrequest", "relay", "webrelaydata", "webrelay", "mps", "mpscmd", "swarm", "swarmcmd", "agentupdate", "agent", "cert", "db", "email", "amt", "httpheaders", "websocket" ], "agentlogdump": true, "compression": false, "wscompression": false, "agentWsCompression": true, "_NoAgentUpdate": 1, "selfUpdate": true, "plugins": { "enabled": true }, "allowlogintoken": true, "allowframing": true, "webrtc": true, "ignoreagenthashcheck": false }, "domains": { "": { "Title": "MYORG Mesh", "Title2": "MYORG Mesh", "_TitlePicture": "title-sample.png", "_loginPicture": "logintitle-sample.png", "_welcomeText": "This is sample text", "_welcomePicture": "mainwelcome-04.jpg", "_welcomePictureFullScreen": true, "_siteStyle": "1", "_nightMode": "1", "certUrl": "https://meshcentral.mydomain.org", "meshMessengerTitle": "MYORG Mesh Chat", "_meshMessengerPicture": "chatimage.png", "_footer": "This is a HTML string displayed at the bottom of the web page when a user is logged in.", "_loginfooter": "This is a HTML string displayed at the bottom of the web page when a user is not logged in." }, "agentFileInfo": { "title": "MYORG Mesh", "agentconfig": ["coreDumpEnabled=1"] } }, "smtp": { "host": "smtp.office365.com", "port": 587, "from": "dummy@example.com", "user": "dummy@example.com", "pass": "ThisIsNotARealPasswordIn...2024", "tls": false } } ```

I'm using behind ARR in IIS.

If I do the troubleshooting listed, such as taking the agentdownload.ashx and going to https://server/agentdownload.ashx?c= I get the file without issue.

This is why I was asking about better agent side debugging.

In trace, I see the request come in. I opened meshcentral in VS Code and put a breakpoint in handleAgentDownloadFile but it never hits it.

I put a breakpointe within the obj.agentapp.use loop for functions - and I can see (webserver.js at approximately line 6493) parent.debug('webrequest', '(' + req.clientIp + ') AgentPort: ' + req.url);

The req.clientIp is my test agent, and req.url is indeed agentdownload.ashx

Everything works except this, which made me wonder if I could debug the agent side to see what it's actually hitting. I know it's hitting my server because trace shows it and if I run it in debug, the request hits.

I verified handleAgentDownloadFile is registered in setupHTTPHandlers.

I am using a self signed agent. I have to wonder if my agent is an older version or something. It's my understanding that updating meshcentral will update the agent, and it signs the agents on startup.

Is that correct? Is there something additional I should do to ensure my agent is the latest version besides updating the server?

Thanks for the response.

si458 commented 2 months ago

@inzi you have webrtc set as true!!!

The is a problem with using webrtc and file transfer!

You must set webrtc to false and file transfer will work

Will tag other issue shortly which someone discovered about this exact problem

Edit. https://github.com/Ylianst/MeshCentral/issues/6309

inzi commented 2 months ago

Thank you for the reply. (I sent you a donation via paypal, by the way, for all your hard work - so Thank you, for real)

Done, restarted, still getting 1k file:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /agentdownload.ashx</pre>
</body>
</html>

I see the agents making the request:

Not sure what else to check?

si458 commented 2 months ago

I did see the donation thank you ❤️

Try removing cookieencoding and restart?

catalinchertes commented 2 months ago

Hello guys,

First of all I want to thank you all for the work you are putting into this project and for trying to help us out.

I must say that 3 years later we still face this issue in our environment, we have the latest version 1.1.27 and both options (webrtc and cookiencryption) commented out but it still fails to download the files.

si458 commented 2 months ago

@catalinchertes please can you share your full config.json? (hide password etc)

catalinchertes commented 2 months ago

@catalinchertes please can you share your full config.json? (hide password etc)

Sure thing, here's the config

``` { "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json", "__comment__": "This is a sample configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.", "settings": { "_noagentupdate": true, "cert": "censored", --> removed because private info "mongoDb": "mongodb://127.0.0.1:27017", "mongoDbName": "meshcentral", "_mongoDbChangeStream": true, "mongoDbBulkOperations": true, "_WANonly": true, "_LANonly": true, "_sessionKey": "MyReallySecretPassword1", "_sessionSameSite": "strict", "_dbEncryptKey": "MyReallySecretPassword2", "_dbRecordsEncryptKey": "MyReallySecretPassword", "_dbRecordsDecryptKey": "MyReallySecretPassword", "__dbExpire": "Amount of time to keep various events in the database, in seconds. Below are the default values.", "_dbExpire": { "events": 1728000, "powerevents": 864000, "statsevents": 2592000 }, "port": 443, "_portBind": "127.0.0.1", "_aliasPort": 4430, "_redirPort": 80, "_redirPortBind": "127.0.0.1", "_redirAliasPort": 80, "agentPort": 777, "_agentPortBind": "127.0.0.1", "_agentAliasPort": 777, "_agentAliasDNS": "agents.myserver.mydomain.com", "_agentPortTls": true, "_exactPorts": true, "_allowLoginToken": true, "_allowFraming": true, "_cookieIpCheck": false, "_cookieEncoding": "hex", "_compression": true, "_wscompression": false, "_agentwscompression": true, "_agentsInRam": false, "_webRTC": true, "_nice404": false, "_selfUpdate": true, "_browserPing": 60, "_browserPong": 60, "_agentPing": 60, "_agentPong": 60, "_agentIdleTimeout": 150, "_meshErrorLogPath": "c:\\tmp", "_npmPath": "c:\\npm.exe", "_npmProxy": "http://1.2.3.4:80", "_allowHighQualityDesktop": true, "_webPush": { "email": "xxxxx@xxxxx.com" }, "_publicPushNotifications": true, "_desktopMultiplex": true, "_userAllowedIP": "127.0.0.1,192.168.1.0/24", "_userBlockedIP": "127.0.0.1,::1,192.168.0.100", "_agentAllowedIP": "192.168.0.100/24", "_agentBlockedIP": "127.0.0.1,::1", "_authLog": "c:\\temp\\auth.log", "_manageAllDeviceGroups": [ "user//admin" ], "_manageCrossDomain": [ "user//admin" ], "_localDiscovery": { "name": "Local server name", "info": "Information about this server" }, "_tlsOffload": "127.0.0.1::1", "_trustedProxy": "127.0.0.1::1", "mpsPort": 4433, "_mpsPortBind": "127.0.0.1", "_mpsAliasPort": 4433, "_mpsAliasHost": "mps.mydomain.com", "_mpsTlsOffload": true, "_no2FactorAuth": true, "_log": "main,web,webrequest,cert", "_syslog": "meshcentral", "_syslogauth": "meshcentral-auth", "_syslogjson": "meshcentral-json", "_webrtcConfig": { "iceServers": [ { "urls": "stun:stun.services.mozilla.com" }, { "urls": "stun:stun.l.google.com:19302" } ] }, "_autoBackup": { "_mongoDumpPath": "C:\\Program Files\\MongoDB\\Server\\4.2\\bin\\mongodump.exe", "backupIntervalHours": 24, "keepLastDaysBackup": 10, "zipPassword": "MyReallySecretPassword3", "_backupPath": "C:\\backups", "_googleDrive": { "folderName": "MeshCentral-Backups", "maxFiles": 10 }, "_webdav": { "_url": "https://server/remote.php/dav/files/xxxxx@server.com/", "_username": "user", "_password": "pass", "_folderName": "MeshCentral-Backups", "_maxFiles": 10 } }, "_redirects": { "meshcommander": "https://www.meshcommander.com/" }, "maxInvalidLogin": "Time in minutes, max amount of bad logins from a source IP in the time before logins are rejected.", "maxInvalidLogin": { "time": 10, "count": 10, "coolofftime": 10 }, "watchDog": { "interval": 100, "timeout": 400 }, "_AmtProvisioningServer": { "port": 9971, "deviceGroup": "mesh//xxxxxxxxxxxxxxxxxxxxx", "newMebxPassword": "amtpassword", "trustedFqdn": "sample.com", "ip": "192.168.1.1" }, "plugins": { "enabled": true } }, "_domaindefaults": { "__comment__": "Any settings in this section is used as default setting for all domains", "title": "Remote Desktop Server [MESH]", "footer": "Default page footer", "newAccounts": false }, "domains": { "": { "siteStyle": 2, "title": "censored", --> removed because private info "title2": "", "titlePicture": "default.png", "agentInviteCodes": true, "_loginPicture": "title-sample.png", "_userQuota": 1048576, "_meshQuota": 248576, "_minify": true, "showNotesPanel": true, "_guestDeviceSharing" : false, "_loginKey": [ "abc", "123" ], "_newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "_userNameIsEmail": true, "_newAccountEmailDomains": [ "sample.com" ], "_newAccountsRights": [ "nonewgroups", "notools" ], "_welcomeText": "Sample Text on Login Page.", "welcomePicture": "censored", --> removed because private info "PasswordRequirements": censored", --> removed because private info "_maxDeviceView": 100, "_welcomePictureFullScreen": false, "_meshMessengerTitle": "MeshMessenger", "_meshMessengerPicture": "messenger.png", "___hide__": "Sum of: 1 = Hide header, 2 = Hide tab, 4 = Hide footer, 8 = Hide title, 16 = Hide left bar", "_hide": 4, "_footer": "Twitter", "_loginfooter": "This is a private server.", "_certUrl": "https://192.168.2.106:443/", "_altMessenging": { "name": "Jitsi", "url": "https://meet.jit.si/myserver-{0}" }, "_deviceMeshRouterLinks": { "rdp": true, "ssh": true, "scp": true, "extralinks": [ { "name": "HTTP", "protocol": "http", "port": 80, "_ip": "192.168.1.100", "_filter": [ "mesh//xxxx", "node//xxxx" ] }, { "name": "HTTPS", "protocol": "https", "port": 443 } ] }, "myServer": { "Backup": true, "Restore": true, "Upgrade": true, "ErrorLog": true, "Console": true, "Trace": true }, "_passwordRequirements": { "min": 8, "max": 128, "upper": 1, "lower": 1, "numeric": 1, "nonalpha": 1, "reset": 90, "force2factor": true, "skip2factor": "127.0.0.1,192.168.2.0/24", "oldPasswordBan": 5, "banCommonPasswords": true }, "twoFactorCookieDurationDays": 7, "_agentInviteCodes": true, "_agentNoProxy": true, "_geoLocation": true, "_novnc": false, "_mstsc": true, "_ssh": true, "_WebEmailsPath": "/myserver/email-templates", "consentMessages": { "title": "censored", --> removed because private info "desktop": "censored", --> removed because private info "terminal": "censored", --> removed because private info "files": "censored", --> removed because private info }, "notificationMessages": { "title": "censored", --> removed because private info "desktop": "censored", --> removed because private info "terminal": "censored", --> removed because private info "files": "censored", --> removed because private info }, "agentCustomization": { "displayName": "censored", --> removed because private info "description": "censored", --> removed because private info "companyName": "censored", --> removed because private info "_serviceName": "Mesh Agent", "_fileName": "meshagent" }, "_userAllowedIP": "127.0.0.1,192.168.1.0/24", "_userBlockedIP": "127.0.0.1,::1,192.168.0.100", "_agentAllowedIP": "192.168.0.100/24", "_agentBlockedIP": "127.0.0.1,::1", "_orphanAgentUser": "admin", "___userSessionIdleTimeout__": "Number of user idle minutes before auto-disconnect", "_userSessionIdleTimeout": 30, "_userConsentFlags": { "desktopnotify": true, "terminalnotify": true, "filenotify": true, "desktopprompt": true, "terminalprompt": true, "fileprompt": true, "desktopprivacybar": true }, "_urlSwitching": false, "_desktopPrivacyBarText": "Privacy bar: {0}, {1}", "_limits": { "_maxDevices": 100, "_maxUserAccounts": 100, "_maxUserSessions": 100, "_maxAgentSessions": 100, "maxSingleUserSessions": 10 }, "_amtScanOptions": [ "LabNetwork 192.168.15.0/23", "SalesNetwork 192.168.8.0/24" ], "_amtAcmActivation": { "log": "amtactivation.log", "certs": { "mycertname": { "certfiles": [ "amtacm-leafcert.crt", "amtacm-intermediate1.crt", "amtacm-intermediate2.crt", "amtacm-rootcert.crt" ], "keyfile": "amtacm-leafcert.key" } } }, "_amtManager": { "adminAccounts": [{ "user": "admin", "pass": "MyP@ssw0rd" }], "environmentDetection": [ "domain1.com", "domain2.com", "domain3.com", "domain4.com" ], "wifiProfiles": [ { "name": "Profile1", "ssid": "MyStation1", "authentication": "wpa2-psk", "encryption": "ccmp-aes", "password": "MyP@ssw0rd" } ] }, "_redirects": { "meshcommander": "https://www.meshcommander.com/" }, "_yubikey": { "id": "0000", "secret": "xxxxxxxxxxxxxxxxxxxxx", "_proxy": "http://myproxy.domain.com:80" }, "_httpHeaders": { "Strict-Transport-Security": "max-age=360000", "x-frame-options": "SAMEORIGIN" }, "_agentConfig": [ "webSocketMaskOverride=1", "coreDumpEnabled=1" ], "_sessionRecording": { "_onlySelectedDeviceGroups": true, "_filepath": "C:\\temp", "index": true, "_maxRecordings": 10, "_maxRecordingSizeMegabytes": 1000, "__protocols__": "Is an array: 1 = Terminal, 2 = Desktop, 5 = Files, 100 = Intel AMT WSMAN, 101 = Intel AMT Redirection, 200 = Messenger", "protocols": [ 1, 2, 101 ] }, "__authStrategies": { "__comment__": "This section is used to allow users to login using other accounts. You will need to get an API key from the services and register callback URL's", "_twitter": { "_callbackurl": "https://server/auth-twitter-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxx", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, "_google": { "_callbackurl": "https://server/auth-google-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxx" }, "_github": { "_callbackurl": "https://server/auth-github-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxx", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, "_reddit": { "_callbackurl": "https://server/auth-reddit-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxx", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, "_azure": { "_callbackurl": "https://server/auth-azure-callback", "newAccounts": false, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "clientid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "clientsecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "tenantid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, "_jumpcloud": { "_callbackurl": "https://server/auth-jumpcloud-callback", "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "entityid": "meshcentral", "idpurl": "https://sso.jumpcloud.com/saml2/saml2", "cert": "jumpcloud-saml.pem" }, "_saml": { "_callbackurl": "https://server/auth-saml-callback", "_disableRequestedAuthnContext": true, "newAccounts": true, "_newAccountsUserGroups": [ "ugrp//xxxxxxxxxxxxxxxxx" ], "_newAccountsRights": [ "nonewgroups", "notools" ], "entityid": "meshcentral", "idpurl": "https://server/saml2", "cert": "saml.pem" } } }, "_customer1": { "_dns": "customer1.myserver.com", "_title": "Customer1", "_title2": "TestServer", "_newAccounts": 1, "_auth": "sspi", "__auth": "ldap", "_LDAPUserName": "gecos", "_LDAPUserKey": "uid", "_LDAPUserEmail": "otherMail", "_LDAPPptions": { "url": "test", "anne": { "gecos": "Anne O'Nyme", "displayName": "O Nyme anne", "uid": "anneonyme", "mail": "anneonyme@example.com", "email": "anneonyme@example.com", "otherMail": [ "other.anneonyme@example.com", "anneonyme@example.com" ] }, "so": { "displayName": "Sticker Sophie", "gecos": "Sophie Sticker", "uid": "ssticker", "mail": "ssticker@example.com", "email": "ssticker@example.com", "otherMail": [ "other.ssticker@example.com", "ssticker@example.com" ] } }, "__LDAPOptions": { "URL": "ldap://1.2.3.4:389", "BindDN": "CN=svc_meshcentral,CN=Users,DC=meshcentral,DC=local", "BindCredentials": "Password.1", "SearchBase": "DC=meshcentral,DC=local", "SearchFilter": "(sAMAccountName={{username}})" }, "_footer": "Test", "_certUrl": "https://192.168.2.106:443/" }, "_info": { "_share": "C:\\ExtraWebSite" } }, "_letsencrypt": { "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.", "email": "censored", --> removed because private info "names": "censored", --> removed because private info "production": true }, "_peers": { "serverId": "server1", "servers": { "server1": { "url": "wss://192.168.2.133:443/" }, "server2": { "url": "wss://192.168.1.106:443/" } } }, "_smtp": { "host": "smtp.myserver.com", "port": 25, "from": "myemail@myserver.com", "__tls__": "When 'tls' is set to true, TLS is used immidiatly when connecting. For SMTP servers that use TLSSTART, set this to 'false' and TLS will still be used.", "tls": false, "___tlscertcheck__": "When set to false, the TLS certificate of the SMTP server is not checked.", "_tlscertcheck": false, "__tlsstrict__": "When set to true, TLS cypher setup is more limited, SSLv2 and SSLv3 are not allowed.", "_tlsstrict": true }, "_sendgrid": { "from": "myemail@myserver.com", "apikey": "***********" }, "_sms": { "provider": "twilio", "sid": "ACxxxxxxxxx", "auth": "xxxxxxx", "from": "+1-555-555-5555" }, "__sms": { "provider": "plivo", "id": "xxxxxxx", "token": "xxxxxxx", "from": "1-555-555-5555" } } ```

si458 commented 2 months ago

only 1 thing wrong with your config ive spotted you have 2 maxInvalidLogin i think one was suppose to be a commented out _maxInvalidLogin but all looks ok? will do some more digging what are the remote OS you are trying to send too? all windows? all linux? mix?

inzi commented 2 months ago

Hi Simon.

Removed cookie encoding, still no joy. When the agent attempts to download, does it connect on HTTP or HTTPS or on the agent port? I'm starting to wonder because the server traced to "agentport:".

Does that mean the agent connected directly to MC on the agent port (in my config, 481), or does it make an HTTPS call?

I'm going through the agent source now to see if I can find answers.

inzi commented 2 months ago

Could this entry in the agent log be related?

"Certificate loaded from DB was not signed by our root cert in the Cert Store"

Would that potentially prevent the agent from connecting?

When I run the server in debug mode in VSCode, the handleAgentDownloadFile function is never called.

si458 commented 2 months ago

@inzi let me have another look tomorrow and let you know sir!

Im trying to think what's causing it?

General, it should should do a https request to the agent port with the /agentdownload.ashx url at the end as u are seeing in the screenshot

So it's something else?

inzi commented 2 months ago

Thanks. That gives me some idea on what to test. Thanks again for your help!

si458 commented 2 months ago

One other thing to try is try using the batch upload to a single device first, then try multiple devices Also check a normal file transfer works without any issues

inzi commented 2 months ago

That solved it. Based on this post: https://github.com/Ylianst/MeshCentral/issues/1767#issuecomment-684316894

I moved the agent port, then set reverse proxy for the original port and restarted.

After restarting, the agents began reconnecting, I then did a file upload and the file transmitted correctly.

The issue is the the mesh agent tries to do a standard https request on the agent port, and if that is not mapped through the reverse proxy, the SSL cert is invalid and the connection aborts, resulting the 1k file of "cannot GET".

I'm still testing to make sure my websockets connections are stable. I'll update this thread.

For anyone reading this in the future, I'm using IIS ARR and URL Rewrite. I bound the ARR site to 443 and the original agent port, then added the alias port settings in my config for MC.

This did the trick, no the agent can make a standard HTTPS call and will get a valid SSL cert back.

inzi commented 2 months ago

Hello guys,

First of all I want to thank you all for the work you are putting into this project and for trying to help us out.

I must say that 3 years later we still face this issue in our environment, we have the latest version 1.1.27 and both options (webrtc and cookiencryption) commented out but it still fails to download the files. ...

@catalinchertes

See above how I resolved mine - For me, it was because the agent is trying to do a standard https request to the agentport, which in your case is port 777. If you go to https://yourserver:777, do you get a valid SSL cert? If not, that may be the issue.

I'm using IIS ARR in front of my MC server, so my setup is different. It doesn't look like you've got a reverse proxy in front of yours based on the config.json file, so I'd see if your able to hit port 777 from where you need to. If you are, make sure your cert is used on the agent port for standard https requests.

If you resolve that, I bet your agents will start working as expected.

inzi commented 2 months ago

I think I have found the issue with the 1k files and why it works and doesn't work when you have a reverse proxy.

The agent uses the MeshServer url to get the file. So if you have wss://mesh.domain.com:444/agent.ashx in your config, the command sent to the agent is a wget, and it tries to open https://mesh.domain.com:444/agentdownload.ashx?c=....

The problem with this is the communication over the agent port uses an internal certificate, but if that certificate is different than the URL, it won't download.

If you specify an agentport and agentport alias, the agent will be able to successfully download. However, mesh central's certificate going back to the agent won't match, and communication will be lost with the agent. Thus, the agent will "connect", but desktop, files, and console are unavailable.

Mesh central reports the agent as connected.

The best solution for this, IMHO, is an option in the msh file for what URL to use for wget. Or Have it as a server side setting for the agent.

This way, when instructed to execute a wget, it will not use the web socket port to try and download, but will instead do a normal https request, which will succeed.

catalinchertes commented 2 months ago

Hello guys, First of all I want to thank you all for the work you are putting into this project and for trying to help us out. I must say that 3 years later we still face this issue in our environment, we have the latest version 1.1.27 and both options (webrtc and cookiencryption) commented out but it still fails to download the files. ...

@catalinchertes

See above how I resolved mine - For me, it was because the agent is trying to do a standard https request to the agentport, which in your case is port 777. If you go to https://yourserver:777, do you get a valid SSL cert? If not, that may be the issue.

I'm using IIS ARR in front of my MC server, so my setup is different. It doesn't look like you've got a reverse proxy in front of yours based on the config.json file, so I'd see if your able to hit port 777 from where you need to. If you are, make sure your cert is used on the agent port for standard https requests.

If you resolve that, I bet your agents will start working as expected.

Thanks for looking into this @inzi, much appreciated.

If I browse to the https://myserver:777 it says that the certificate is ok but I get the same "Cannot GET /" just like when I batch upload files.

Now I understand why when I copied the /agentdownload.. url and pasted it in the browser using the normal 443 port it worked like a charm but when the agent tries to download the files it goes through the 777 port which returns the cannot get.

The certificate is the same and is ok, I'll look around and mess with the configuration to see if I'm able to fix this knowing the above.

si458 commented 2 months ago

ok so /agentdownload.ashx is only accessable from the main webserver port (or aliasport if set) so when the remote machines does a download it should download from https://meshcentral.mydomain.com/agentdownload.ashx

BUT if its trying to download from https://meshcentral.mydomain.com:777/agentdownload.ashx and 777 is the agentport then the is indeed a bug!

@catalinchertes @inzi can you just verify what URL it thinks its downloading from again for me? too many comments and list track whats what

si458 commented 2 months ago

@inzi @catalinchertes ok i think its now fixed for you! https://github.com/Ylianst/MeshCentral/commit/4e37455471c05c171a88f0e4016447b18d62f421 basically we need to also setup the agentdownload.ashx on the agentonly port! apply the latest commit and restart meshcentral and give it a try for me please!

catalinchertes commented 2 months ago

@si458 That indeed fixed the issue sir!!

I applied the new webserver.js file and restarted meshcentral and everything works perfectly now.

Thank you!!

si458 commented 2 months ago

@catalinchertes what i believe was happening from the looks of meshcore.js it was doing the request, it oversally returns a 404, but was still saving 0 bytes into the file because it returned something this oversally is stupid and shouldnt even attempt to save if its not a 200 but something else so might look into fixing that at a later date but glad its sorted still tho!

inzi commented 2 months ago

So far, it's working, but agent port alias worked for me before, but after 24 hours or so, I lost the ability to manage the clients.

I'll run it with this setting and see.

Thanks Simon.

inzi commented 2 months ago

@si458 Did you ever write up anything for working on the agent's source code? I don't know a lot of C, but I've done assembly so concepts like pointers & structures aren't foreign. But I do a lot of c# & powershell.

I'd like to understand how the agent architecture more so I could help.

Ylianst / MeshCentral

Multiple devices file upload results in a 1KB file #3282