search

amidaware / rmmagent

Tactical RMM Agent
https://github.com/amidaware/tacticalrmm
Other
124 stars 71 forks source link

Mesh Agent Install Failure - "The file or directory is corrupted and unreadable" #2

Closed Can-eh-dian11 closed 2 years ago

Can-eh-dian11 commented 2 years ago

Hi,

I recently updated to 0.12.0 via the provided update.sh script and have discovered that any attempt to install the TacticalRMM agent on a new computer fails during the Mesh Agent installation.

Brief overview of my current setup:

The meshagent.exe file is downloaded on the affected system however it is a 1KB file. I've included the error message presented by the TacticalRMM installer (Switching between installation methods makes no difference). There only seems to be an issue with agent install, the rest of the functionality provided by TacticalRMM is fine for existing agents.

Downloading agent... Extracting files... Installation starting. time="2022-03-23T13:36:49-04:00" level=info msg="Downloading mesh agent..." time="2022-03-23T13:36:49-04:00" level=info msg="Installing mesh agent..."

fork/exec C:\Program Files\TacticalAgent\meshagent.exe: The file or directory is corrupted and unreadable.: [ ] time="2022-03-23T13:36:55-04:00" level=error msg="exec: \"C:\\Program Files\\Mesh Agent\\MeshAgent.exe\": file does not exist: " time="2022-03-23T13:37:00-04:00" level=error msg="exec: \"C:\\Program Files\\Mesh Agent\\MeshAgent.exe\": file does not exist: "

What I've tried:

Any ideas you may have to correct this would be much appreciated!

dinger1986 commented 2 years ago

Is your AV messing it up?

Can-eh-dian11 commented 2 years ago

Not that I've seen. Nothing reported in any logs and I've tried the agent install with A/V disabled as well as completely removed.

dinger1986 commented 2 years ago

Does it work installing mesh directly?

Can-eh-dian11 commented 2 years ago

Just tested that and installing the mesh agent on its own works fine

silversword411 commented 2 years ago

Are you code signed? What agent install method are you using?

Screenshots are good, need to determine what's different between your install and others, since we haven't seen others reporting windows agent issues like this.

Does you installation deviate in any way from the standard install (proxies/alt cert methods etc)

Smaller files than expected are usually signs of in-network av download interruptions

Unstable-Penguin commented 2 years ago

Hi,

I've got a similar issue happening with my install. When the tactical agent try's to install the mesh agent it fails. When running with debugging it generates a HTTP response of 400 from the mesh central server and fails.

I've ruled out being a config issue with the existing setup as I re-install everything from scratch in a test env

It's not the AV as that has been fully disabled and I receive the same results

Debug snippet below

`============================================================================== 2022/03/24 16:28:55.640377 DEBUG RESTY


POST  /api/v3/meshexe/  HTTP/1.1
HOST   : _api.removedurl_
HEADERS:
        Accept: application/json
        Authorization: Token daf8dcdf56b06ae84d8a67c10688203a4a18889a7a6eec9053aaae3e93d3ca80
        Content-Type: application/json
        User-Agent: go-resty/2.7.0 (https://github.com/go-resty/resty)
BODY   :
{
   "arch": "64",
   "plat": "windows"
}
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 400 Bad Request
PROTO        : HTTP/1.1
RECEIVED AT  : 2022-03-24T16:28:55.6403778Z
TIME DURATION: 163.6312ms
HEADERS      :
        Allow: POST, OPTIONS
        Content-Length: 35
        Content-Type: application/json
        Date: Thu, 24 Mar 2022 16:28:55 GMT
        Referrer-Policy: same-origin
        Server: nginx
        Vary: Origin
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
BODY         :
***** RESPONSE WRITTEN INTO FILE *****
==============================================================================`
Can-eh-dian11 commented 2 years ago

Hi,

I've got a similar issue happening with my install. When the tactical agent try's to install the mesh agent it fails. When running with debugging it generates a HTTP response of 400 from the mesh central server and fails.

I've ruled out being a config issue with the existing setup as I re-install everything from scratch in a test env

It's not the AV as that has been fully disabled and I receive the same results

Debug snippet below

`==============================================================================

2022/03/24 16:28:55.640377 DEBUG RESTY

POST  /api/v3/meshexe/  HTTP/1.1
HOST   : _api.removedurl_
HEADERS:
        Accept: application/json
        Authorization: Token daf8dcdf56b06ae84d8a67c10688203a4a18889a7a6eec9053aaae3e93d3ca80
        Content-Type: application/json
        User-Agent: go-resty/2.7.0 (https://github.com/go-resty/resty)
BODY   :
{
   "arch": "64",
   "plat": "windows"
}
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 400 Bad Request
PROTO        : HTTP/1.1
RECEIVED AT  : 2022-03-24T16:28:55.6403778Z
TIME DURATION: 163.6312ms
HEADERS      :
        Allow: POST, OPTIONS
        Content-Length: 35
        Content-Type: application/json
        Date: Thu, 24 Mar 2022 16:28:55 GMT
        Referrer-Policy: same-origin
        Server: nginx
        Vary: Origin
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
BODY         :
***** RESPONSE WRITTEN INTO FILE *****
==============================================================================`

Just want to add on that I've been testing a few things out on my end this morning as well and I'm getting the same 400 error with the same API call. If I download the mesh agent separately and place it on the guest then specify the local mesh option during install effectively bypassing this request everything works as expected.

==============================================================================
2022/03/24 10:57:41.397192 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
POST  /api/v3/meshexe/  HTTP/1.1
HOST   : api.**********
HEADERS:
        Accept: application/json
        Authorization: Token 59172ce88e0965155cbb4e0237f5694f8c049a563a4543161277e7541123fbc3
        Content-Type: application/json
        User-Agent: go-resty/2.7.0 (https://github.com/go-resty/resty)
BODY   :
{
   "arch": "64",
   "plat": "windows"
}
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 400 Bad Request
PROTO        : HTTP/1.1
RECEIVED AT  : 2022-03-24T10:57:41.3971926-04:00
TIME DURATION: 139.5082ms
HEADERS      :
        Allow: POST, OPTIONS
        Content-Length: 55
        Content-Type: application/json
        Date: Thu, 24 Mar 2022 14:57:41 GMT
        Referrer-Policy: same-origin
        Server: nginx
        Vary: Origin
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
BODY         :
***** RESPONSE WRITTEN INTO FILE *****
==============================================================================
time="2022-03-24T10:57:50-04:00" level=fatal msg="Unable to download the mesh agent from the RMM. "
exit status 1
silversword411 commented 2 years ago

Asking again, did I miss your answer?

a 1kb file sounds like in-network av slicing the download and corrupting the file

Are you code signed? What agent install method are you using?

Screenshots are good, need to determine what's different between your install and others, since we haven't seen others reporting windows agent issues like this.

Does you installation deviate in any way from the standard install (proxies/alt cert methods etc)

Smaller files than expected are usually signs of in-network av download interruptions

silversword411 commented 2 years ago

Also did you really say Windows 10 1909...as in you haven't patched in 2 years and have killed windows update somehow?

Can-eh-dian11 commented 2 years ago

Sorry, I completely forgot to include that information in my previous response...

Also did you really say Windows 10 1909...

Windows 10 1909 Enterprise which is still supported (I should have been more specific)

Are you code signed? What agent install method are you using?

We are not using code signed agents. We have A/V exclusions in place per the documentation and A/V was disabled for all testing. Attempts have been made using the generated exe installer, PowerShell, as well as the manual installation option. For the purposes of this test we were logging into the TacticalRMM console from the guest and downloading the agent directly.

Screenshots are good, need to determine what's different between your install and others, since we haven't seen others reporting windows agent issues like this.

Is there something specific you want screenshots of? Config files? Agent install output?

Does you installation deviate in any way from the standard install (proxies/alt cert methods etc)

We have implemented all items in the "Securing nginx" guide. Everything else was a standard install following the documentation provided.

Smaller files than expected are usually signs of in-network av download interruptions

Agreed however I can download the mesh agent and tactical agent standalone without any interruptions. I only run into this issue when the Tactical agent installer attempts to download the mesh agent application. Not saying this can't be the issue, just that I would have expected to run into issues elsewhere as well.

silversword411 commented 2 years ago

As with all detective work, are these the only machines you have in TRMM, or are there others working fine?

Are you using agent install, or deployment links? Download the deployment links with the web browser from different networks. Are the files the same size?

Have you customized your mesh? Tactical installed and managed, or secondard? Have you changed settings in meshcentral? Have you renamed the default "TacticalRMM" group in meshcentral?

silversword411 commented 2 years ago

Troubleshooting is better done thru discord on this stuff.

silversword411 commented 2 years ago

FYI, you can log into mesh.yourdomain.com , and download the installer manually, does that work?

wh1te909 commented 2 years ago

@Can-eh-dian11 please upgrade to 0.12.1 and run the new mesh troubleshooting script (check 0.12.1 release notes on the main repo) and then paste the output of that here (redact sensitive info like domains and tokens)

Unstable-Penguin commented 2 years ago

@wh1te909 @Can-eh-dian11

I've been able to run the update on our install this morning and still no joy. Still getting the same error as yesterday. From my experience the issue is with the request the Tactical agent makes to mesh. This has only happened since the update to version 2

See below for log files and command results.

tactical.exe --log debug

Installer: C:\windows\Temp\winagent-v2.0.1.exe
Tactical Agent: C:\Program Files\TacticalAgent\tacticalrmm.exe
Download URL: https://github.com/amidaware/rmmagent/releases/download/v2.0.1/winagent-v2.0.1.exe
Install command: C:\Program Files\TacticalAgent\tacticalrmm.exe -m install --api https://api.rmm.REDACTED --client-id 1 --site-id 2 --agent-type workstation --auth REDACTED -log debug
Downloading agent...
Extracting files...
Installation starting.
time="2022-03-25T08:35:03Z" level=debug msg="{Hostname:SEC-LAP-00004 Arch:x86_64 AgentID: BaseURL: ApiURL: Token: AgentPK:0 Cert: ProgramDir:C:\\Program Files\\TacticalAgent EXE:C:\\Program Files\\TacticalAgent\\tacticalrmm.exe SystemDrive:C: MeshInstaller:meshagent.exe MeshSystemEXE:C:\\Program Files\\Mesh Agent\\MeshAgent.exe MeshSVC:mesh agent PyBin:C:\\Program Files\\TacticalAgent\\py38-x64\\python.exe Headers:map[] Logger:0xc000238000 Version:2.0.1 Debug:true rClient:0xc000278000 Proxy: LogTo: LogFile:<nil> Platform:windows GoArch:amd64 ServiceConfig:0xc0002440b0}\n"
time="2022-03-25T08:35:03Z" level=debug msg="Agent ID: ZWjTIhHxLbZiXZHqAKmaCQcWRaeJTClXufvdQdwc"
time="2022-03-25T08:35:03Z" level=debug msg="API: api.REDACTED"
time="2022-03-25T08:35:03Z" level=debug msg="Base URL: https://api.REDACTED"
2022/03/25 08:35:03.719956 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
GET  /api/v3/installer/  HTTP/1.1
HOST   : api.REDACTED
HEADERS:
        Accept: application/json
        Authorization: Token REDACTED
        Content-Type: application/json
        User-Agent: go-resty/2.7.0 (https://github.com/go-resty/resty)
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
PROTO        : HTTP/1.1
RECEIVED AT  : 2022-03-25T08:35:03.7199561Z
TIME DURATION: 82.5848ms
HEADERS      :
        Allow: GET, POST, HEAD, OPTIONS
        Content-Length: 4
        Content-Type: application/json
        Date: Fri, 25 Mar 2022 08:35:03 GMT
        Referrer-Policy: same-origin
        Server: nginx
        Vary: Origin
        X-Content-Type-Options: nosniff, nosniff
        X-Frame-Options: DENY
BODY         :
"ok"
==============================================================================
2022/03/25 08:35:03.789151 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
POST  /api/v3/installer/  HTTP/1.1
HOST   : api.REDACTED
HEADERS:
        Accept: application/json
        Authorization: Token REDACTED
        Content-Type: application/json
        User-Agent: go-resty/2.7.0 (https://github.com/go-resty/resty)
BODY   :
{
   "version": "2.0.1"
}
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
PROTO        : HTTP/1.1
RECEIVED AT  : 2022-03-25T08:35:03.7891513Z
TIME DURATION: 68.6578ms
HEADERS      :
        Allow: GET, POST, HEAD, OPTIONS
        Content-Length: 4
        Content-Type: application/json
        Date: Fri, 25 Mar 2022 08:35:03 GMT
        Referrer-Policy: same-origin
        Server: nginx
        Vary: Origin
        X-Content-Type-Options: nosniff, nosniff
time="2022-03-25T08:35:03Z" level=info msg="Downloading mesh agent..."
        X-Frame-Options: DENY
BODY         :
"ok"
==============================================================================
2022/03/25 08:35:03.986780 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
POST  /api/v3/meshexe/  HTTP/1.1
HOST   : api.REDACTED
HEADERS:
        Accept: application/json
        Authorization: Token REDACTED
        Content-Type: application/json
        User-Agent: go-resty/2.7.0 (https://github.com/go-resty/resty)
BODY   :
{
   "arch": "64",
   "plat": "windows"
}
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 400 Bad Request
PROTO        : HTTP/1.1
RECEIVED AT  : 2022-03-25T08:35:03.9867808Z
TIME DURATION: 197.6289ms
HEADERS      :
        Allow: POST, OPTIONS
        Content-Length: 35
        Content-Type: application/json
        Date: Fri, 25 Mar 2022 08:35:04 GMT
        Referrer-Policy: same-origin
        Server: nginx
        Vary: Origin
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
BODY         :
***** RESPONSE WRITTEN INTO FILE *****
==============================================================================

/rmm/api/env/bin/python /rmm/api/tacticalrmm/manage.py check_mesh

Mesh site: https://mesh.REDACTED Mesh username: zteycisq Mesh token: REDACTED Mesh device group: TacticalRMM Auth token ok: REDACTED Mesh url ok: wss://mesh.REDACTED/control.ashx?auth=REDACTED Mesh device id ok: BqgwZd0vOZZ0AEVjTCndMuG1LN2JbTLyKJ92fnDT0nkM7aAF0IYA333JJPTe3SqY

Unstable-Penguin commented 2 years ago

@wh1te909 @Can-eh-dian11

I have found the cause and a work around for my instance. Hopefully this works for you @Can-eh-dian11

In my instance I found the issue relates to the python SSL library using it's own CA store. Not to sure why it is. As a result the SSL for mesh is no longer a valid certificate so the API services errors. I've imported the CA into the Python store and it now working for me

Run the below from you tactical user account to import the CA

source /rmm/api/env/bin/activate
python

import requests
import certifi
import sys

try:
     requests.get('https://meshurl')
     print('Certificate already added to the certifi store')
     sys.exit(0)
 except requests.exceptions.SSLError as err:
     print('SSL Error. Adding custom certs to Certifi store...')
     customca = requests.get('http://URL for CA to be download from ').content
     cafile = certifi.where()
    with open(cafile, 'ab') as outfile:
         outfile.write(b'\n')
         outfile.write(customca)
wh1te909 commented 2 years ago

glad you figured it out, I totally missed the post where you mentioned you were using the "securing nginx" guide which is not officially supported. so that's why you have to do that workaround. I will update the the securing nginx docs with a link to this ticket in case someone comes across this in the future.