Unable to take remote session though asset is online

[Yogesh1190]

Describe the bug Unable to take remote session though asset is online.

I can see below error message in Windows Event Logs.

The Mesh Agent service terminated unexpectedly.  It has done this 6 time(s).  The following corrective action will be taken in 1 milliseconds: Restart the service.

Below errors I can see in mesh Console Logs,

Mesh Console Logs

uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
consent -->undefined
tunnels sessionid ->undefined
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: undefined in method 'handleServerCommand()'
consent -->91
this is relay connection
Inside getUserImage-->p.resolve
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: undefined in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: undefined in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: undefined in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
consent -->undefined
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: undefined in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
tunnels sessionid ->undefined
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: undefined in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: undefined in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: wait() already in progress in method 'handleServerCommand()'
uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'Command' on 'MeshAgent' threw an exception: undefined in method 'handleServerCommand()'
consent -->undefined
tunnels sessionid ->undefined
tunnels sessionid ->undefined

My config.json file

{
    "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
    "settings": {
        "cert": "domain_url_ui.com",
        "_WANonly": false,
        "_LANonly": false,
        "_sessionKey": "MyReallySecretPassword1",
        "sessionSameSite": "lax",
        "port": 4430,
        "aliasPort": 443,
        "redirPort": 800,
        "redirAliasPort": 80,
        "agentPort": 2121,
        "agentAliasPort": 443,
        "agentAliasDNS": "domain_url_agent.com",
        "agentPortTls": false,
        "agentPing": 10,
        "browserPing": 10,
        "agentIdleTimeout": 18000,
        "TLSOffload": "domain_url_ui.com,0.0.0.0/0",
        "_SelfUpdate": false,
        "allowFraming": true,
        "allowLoginToken": true,
        "plugins": {
            "enabled": true
        },
        "postgres": {
            "user": "admin",
            "password": "s8E51NY2YL",
            "port": "5432",
            "host": "database_pod.namespace.svc",
            "database": "meshcentral",
            "ssl": false
        },
        "dbExpire": {
            "events": 0,
            "powerevents": 0,
            "statsevents": 0
        },
        "agentCoreDump": true,
        "agentLogDump": true,
        "minify": true,
        "clipboardGet": true,
        "clipboardSet": true,
        "compression": true,
        "wsCompression": true,
        "agentWsCompression": true,
        "noAgentUpdate": 1,
        "temporaryAgentUpdate": false,
        "amtscanner": false,
        "agentsInRam": true,
        "desktopMultiplex": true
    },
    "domains": {
        "": {
            "title": "MeshCentral",
            "minify": "true",
            "_localSessionRecording": true,
            "_userNameIsEmail": true,
            "certUrl": "https://domain_url_ui.com",
            "notifyimage": "mesh.png",
            "_meshMessengerTitle": "MeshMessenger",
            "nightmode": 1,
            "userSessionIdleTimeout": 90
        }
    }
}

@Ylianst @krayon007 @si458

[si458]

You are missing critical info of the bug report

Please include all info

[Yogesh1190]

You are missing critical info of the bug report

Please include all info

@si458 which critical info you are expecting here. Can you please guide me.

[si458]

Follow the template when u create a bug report

Stuff like what os r u using, on client, server, remote

What's ur config like, etc...

[Yogesh1190]

Follow the template when u create a bug report

Stuff like what os r u using, on client, server, remote

What's ur config like, etc...

Thanks @si458

I have updated the question with config.json which I am using. Also I am facing this issue for all windows operating systems intermittently.

[si458]

This looks like a reverse proxy issue? What's the reverse proxy u r using?

[Yogesh1190]

This looks like a reverse proxy issue? What's the reverse proxy u r using?

@si458 I am using Kubernetes cluster where I have configure Ingress. I have started the Mesh Agent (443) and UI separate port (2121) to improve the performance.

[si458]

i dont think u can seperate the ports like that? ive only every known you have to change the port and redirPort i would start with a blank basic config.json https://github.com/Ylianst/MeshCentral/blob/master/sample-config.json using a single port to check all is working

[Yogesh1190]

@si458 In this mentioned sample-config.json file for agent and ui having 443 port only. Due to which all traffic going through that port. Based on config.json and documentation of meshcentral, I have configured agent and ui on separate port which is working as well.

Though we use common port for both UI and Agent or separate port still I can see intermittent issue of not taking remote session though remote machine is online.

Also We are trying to validate machine is online or not based on pwr field in the mesh node object.

If pwr = 1 then asset is online and if pwr = 0 then asset is offline.

based on this condition we are trying to show below message on the screen, Requested asset is offline

Also sometime, though pwr having value 1 still remote machine not connecting.

[si458]

@Yogesh1190 wow thats an explaination and a half! im sorry i cant help, its way too advanced for my understanding, ive only every used a single port, and helped others with a single port i think the issue is due to u using seperate ports and maybe ur reverse proxy is disconnecting the websockets (i guess?)

[silversword411]

Probably proxy or network equipment changing TCP session timeouts which are prematurely slicing the 24 hr websocket connection

Ping pong will keep things alive, but it's probably better to stop whatever is artificially changing TCP session timeouts...it's probably causing errors all over the place you haven't realized are happening.