search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
3.77k stars 515 forks source link

Can't consistently connect to devices with peering setup #5714

Open kubamaruszczak opened 6 months ago

kubamaruszczak commented 6 months ago

Describe the bug I'm running two MeshCentral v1.1.19 instances configured in peering setup and I have difficulty consistently connecting to devices terminal sessions. I have experienced three different cases. It is worth to mention that my servers connect to external MongoDB and have exact the same configs excepts for serverId in peers section.

In the first case I can connect to device without any issues and there is no logs on the servers.

The second case is that it freezes on Connected and I can't do anything. image

The third case is that it get stucked on Setup... image

In the second and the third case I can see below logs:

PEER: FTunnel meshc2: Start connect to ws://MeshCentralServer2Url:443/meshrelay.ashx?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=fq6bywz056d&rauth=QyH2Ep8vAO4R0i01$zzPFYPno6i1un5A@nqH98UQMeZoXu9GbUUkh3HlkKXNg4@SF7sLbWy9VQ0AYTG1eh20KTjr7rzBmB2JMECdDMhzEZpB3HJjbLSltA==
PEER: FTunnel meshc2: Connected

and when I click on Disconnect button...

PEER: FTunnel disconnect meshc2
PEER: FTunnel1: Soft disconnect
PEER: FTunnel2: Soft disconnect
PEER: FTunnel disconnect meshc2

Expected behavior I would expect not get stucked while trying to connect to the device.

Server Software:

Remote Device:

Your config.json file

{
  "settings": {
    "cert": "my.domain.url",
    "WANonly": true,
    "LANonly": false,
    "sessionKey": "MyReallySecretPassword1",
    "mongoDb": "mongodb://MeshCentralMongoDBUser:MeshCentralMongoDBPassword@MeshCentralMongoDBDNS:27017/meshcentral?replicaSet=rs0",
    "mongoDbChangeStream": true,
    "port": 443,
    "_aliasPort": 443,
    "redirPort": 80,
    "_redirAliasPort": 80,
    "nightMode": 0,
    "tlsOffload": true,
    "_mpsTlsOffload": true,
    "maxInvalidLogin": {
      "time": 10,
      "count": 10,
      "coolofftime": 10
      },
    "rootCertCommonName": "xxx",
    "mpshighsecurity": true,
    "allowLoginToken": true,
    "authLog": "meshcentral-data/auth.log"
  },
  "domains": {
    "": {
      "title": "MeshCentral",
      "_title2": "Servername",
      "_minify": true,
      "newAccounts": false,
      "userNameIsEmail": true,
      "amtAcmActivation": {
        "log": "amtactivation.log",
        "certs": {
          "mycertname": {
            "certfiles": [ "xxx.crt", "XXX.crt" ],
            "keyfile": "xxx.key"
          }
        }
      },
      "ssh": true,
      "authStrategies": {
        "saml": {
          "__callbackurl": "xxx",
          "NewAccounts": true,
          "entityid": "xxx",
          "idpurl": "xxx",
          "cert": "xxx.pem"
        },
        "oidc":{
          "_authorizationURL": "xxx",
          "callbackURL": "xxx",
          "clientid": "xxx",
          "clientsecret": "xxx",
          "issuer": "xxx",
          "tokenURL": "xxx",
          "userInfoURL": "xxx",
          "newAccounts": true,
          "_groups": {
            "required": ["default"],
            "siteadmin": ["xxx"],
            "sync": {
              "enabled": true
            }
          }
        }
      }
    }
  },
  "peers": {
    "serverId": "MeshCentralServerId",
    "servers": {
      "meshc1": {
        "url": "ws://MeshCentralServerUrl:443/"
      },
      "meshc2": {
        "url": "ws://MeshCentralServer2Url:443/"
      }
    }
  }
}
si458 commented 6 months ago

One thing u can try, is see if the is helpful logs, You can run meshcentral on both machines in FULL DEBUG MODE node node_modules/meshcentral --debug This will sprew out logs for almost everything and see if anything jumps out

I'll have to setup a few machine and check peering is working still

kubamaruszczak commented 6 months ago

Well, I've tried to run it in full debug mode before but I could't see anything important there because most of the logs are related to data transfer between machines. Then after enabling only peering logs I was able to observe what is happening. I'll try full debug again and see what happens when it get stucked...

kubamaruszczak commented 6 months ago

So I've done some more investigation in full debug mode and...

These are logs on both instances when I get stucked on Setup...

MeshCentralServer: HTTPHEADERS: GET /meshrelay.ashx/.websocket?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=ln1sjuk1ve&auth=k5cXcI8gI1VAODMwAQEazNxBtYB0N7hq0DzgGio$nVEbSzepEKR3fixum@bencRvBW3YxvX8sNAxKdtJzUhKF@HkEGSnVG@ImZOjp@6NZNW0x8@kTOD8cXPgiPCn2GsQ5jn3MUVgSegyDQ8= {
MeshCentralServer:   host: 'my.domain.url',
MeshCentralServer:   connection: 'Upgrade',
MeshCentralServer:   pragma: 'no-cache',
MeshCentralServer:   'cache-control': 'no-cache',
MeshCentralServer:   'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
MeshCentralServer:   upgrade: 'websocket',
MeshCentralServer:   origin: 'https://my.domain.url',
MeshCentralServer:   'sec-websocket-version': '13',
MeshCentralServer:   'accept-encoding': 'gzip, deflate, br',
MeshCentralServer:   'accept-language': 'en-US,en;q=0.9',
MeshCentralServer:   cookie: '_hjSessionUser_3158721=eyJpZCI6ImY1ZGNjMzQwLTUyNTctNTBiNy04M2U5LTRkM2E2NGUyNDRkMCIsImNyZWF0ZWQiOjE3MDQ5NzYzOTY3MjMsImV4aXN0aW5nIjp0cnVlfQ==; xid=eyJ1c2VyaWQiOiJ1c2VyLy9kZWxhdmFsIiwiaXAiOiI5MS4yMTkuMTE0LjMzIiwieCI6Im5BNUlnWklkIiwidCI6Mjg0MzIwMjN9; xid.sig=xWM-xJ8yU56CPbCAF5jKoj2AkoOXq1splkVS5xA_SgHWtSAtUrHOOE34LqxPSG5H',
MeshCentralServer:   'sec-websocket-key': 'zTuilRSlqNF6ZibUh8YTsw==',
MeshCentralServer:   'sec-websocket-extensions': 'permessage-deflate; client_max_window_bits'
MeshCentralServer: }
MeshCentralServer: WEBREQUEST: (agentMachineIp) /meshrelay.ashx/.websocket?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=ln1sjuk1ve&auth=k5cXcI8gI1VAODMwAQEazNxBtYB0N7hq0DzgGio$nVEbSzepEKR3fixum@bencRvBW3YxvX8sNAxKdtJzUhKF@HkEGSnVG@ImZOjp@6NZNW0x8@kTOD8cXPgiPCn2GsQ5jn3MUVgSegyDQ8=
MeshCentralServer: COOKIE: Decoded AESGCM cookie: {"userid":"user//myUserName","domainid":"","ip":"agentMachineIp","time":1705919757000,"dtime":1633381}
MeshCentralServer: RELAY: Relay holding: ln1sjuk1ve (agentMachineIp) Authenticated
MeshCentralServer: COOKIE: Encoded AESGCM cookie: {"userid":"user//myUserName","domainid":"","ps":1,"time":1705921390}
MeshCentralServer: PEER: FTunnel meshc2: Start connect to ws://MeshCentralServer2Url:443/meshrelay.ashx?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=ln1sjuk1ve&auth=OpjVUAGyGf$K$qlILoGFFt46usjttIw9p@DEB6ev25hucKV2vSlZWYXsQy1kUMTeZyte41fP@iRntfdBGPXNvIQRRVwD6lhbzMfPmXheTjkcCis1vlMBx1cSQ570
MeshCentralServer: PEER: FTunnel meshc2: Connected

MeshCentralServer2: HTTPHEADERS: GET /meshrelay.ashx/.websocket?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=ln1sjuk1ve&rauth=rqvEglr7P208%244R50Ge0uNyZNT0X%24BgSRQ1aPV%24pvfZIavkfsLTJyPpTJ%40IL3RAZQTIeX7eruSW9PUmhFyj7Szir1vRgul4y4ScqYKIC0Vd6uDoXXfCwiQ== {
MeshCentralServer2:   host: 'my.domain.url',
MeshCentralServer2:   upgrade: 'websocket',
MeshCentralServer2:   connection: 'Upgrade',
MeshCentralServer2:   'sec-websocket-key': 'lozIO4Pz1wM0BWGZKcU4y0==',
MeshCentralServer2:   'sec-websocket-version': '13',
MeshCentralServer2:   'content-length': '0'
MeshCentralServer2: }
MeshCentralServer2: WEBREQUEST: (agentMachineIp) /meshrelay.ashx/.websocket?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=ln1sjuk1ve&rauth=rqvEglr7P208%244R50Ge0uNyZNT0X%24BgSRQ1aPV%24pvfZIavkfsLTJyPpTJ%40IL3RAZQTIeX7eruSW9PUmhFyj7Szir1vRgul4y4ScqYKIC0Vd6uDoXXfCwiQ==
MeshCentralServer2: COOKIE: Decoded AESGCM cookie: {"ruserid":"user//myUserName","x":"nA5IgZId","time":1705919757000,"dtime":1633253}
MeshCentralServer2: RELAY: Relay holding: ln1sjuk1ve (agentMachineIp)
MeshCentralServer2: HTTPHEADERS: GET /meshrelay.ashx/.websocket?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=ln1sjuk1ve&auth=OpjVUAGyGf$K$qlILoGFFt46usjttIw9p@DEB6ev25hucKV2vSlZWYXsQy1kUMTeZyte41fP@iRntfdBGPXNvIQRRVwD6lhbzMfPmXheTjkcCis1vlMBx1cSQ570 {
MeshCentralServer2:   'sec-websocket-version': '13',
MeshCentralServer2:   'sec-websocket-key': '+AB7hS3W8A2OBPefKXVG8w==',
MeshCentralServer2:   connection: 'Upgrade',
MeshCentralServer2:   upgrade: 'websocket',
MeshCentralServer2:   'sec-websocket-extensions': 'permessage-deflate; client_max_window_bits',
MeshCentralServer2:   host: 'MeshCentralServer2Url:443'
MeshCentralServer2: }
MeshCentralServer2: WEBREQUEST: (MeshCentralServerUrl) /meshrelay.ashx/.websocket?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=ln1sjuk1ve&auth=OpjVUAGyGf$K$qlILoGFFt46usjttIw9p@DEB6ev25hucKV2vSlZWYXsQy1kUMTeZyte41fP@iRntfdBGPXNvIQRRVwD6lhbzMfPmXheTjkcCis1vlMBx1cSQ570
MeshCentralServer2: COOKIE: ERR: Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data
MeshCentralServer2: COOKIE: ERR: Bad AESSHA cookie due to exception: Error: error:1C80006B:Provider routines::wrong final block length
MeshCentralServer2: COOKIE: Decoded AESGCM cookie: {"userid":"user//myUserName","domainid":"","ps":1,"time":1705921390000,"dtime":386}
MeshCentralServer2: RELAY: Relay connected: ln1sjuk1ve (MeshCentralServerUrl --> agentMachineIp)

These are logs on both instances when I get stucked on Connected

MeshCentralServer: HTTPHEADERS: GET /meshrelay.ashx/.websocket?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=njxonuous8&rauth=Qgd%405GMPMAFi%40ZpJ17OqRYSQo%40fXbeKZDTBjBg69cVxxL9rPPJcBbe0y7%244UWdcWC9jUt6v0UlFgW4OmNFcAqAhIFJyVOiWMMajLovI5MnAchjJaclW0TQ== {
MeshCentralServer:   host: 'my.domain.url',
MeshCentralServer:   upgrade: 'websocket',
MeshCentralServer:   connection: 'Upgrade',
MeshCentralServer:   'sec-websocket-key': 'yhO+8y6CTXY99RJWHcF8cU==',
MeshCentralServer:   'sec-websocket-version': '13',
MeshCentralServer:   'content-length': '0'
MeshCentralServer: }
MeshCentralServer: WEBREQUEST: (agentMachineIp) /meshrelay.ashx/.websocket?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=njxonuous8&rauth=Qgd%405GMPMAFi%40ZpJ17OqRYSQo%40fXbeKZDTBjBg69cVxxL9rPPJcBbe0y7%244UWdcWC9jUt6v0UlFgW4OmNFcAqAhIFJyVOiWMMajLovI5MnAchjJaclW0TQ==
MeshCentralServer: COOKIE: Decoded AESGCM cookie: {"ruserid":"user//myUserName","x":"nA5IgZId","time":1705921559000,"dtime":252178}
MeshCentralServer: RELAY: Relay holding: njxonuous8 (agentMachineIp)
MeshCentralServer: PEER: FTunnel meshc2: Start connect to ws://MeshCentralServer2Url:443/meshrelay.ashx?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=njxonuous8&rauth=Qgd@5GMPMAFi@ZpJ17OqRYSQo@fXbeKZDTBjBg69cVxxL9rPPJcBbe0y7$4UWdcWC9jUt6v0UlFgW4OmNFcAqAhIFJyVOiWMMajLovI5MnAchjJaclW0TQ==
MeshCentralServer: PEER: FTunnel meshc2: Connected

MeshCentralServer2 HTTPHEADERS: GET /meshrelay.ashx/.websocket?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=njxonuous8&auth=xbqkj@miZFAoJiwzkO4VOM$Bj51tDju3iqNRDweq@JEMQUFlnf9qjYFyi@J2rqLu5S6q6xKTGPh@64nneCUIEhNXa7ZPfW4SO87eRVwV8DWnmmeMFGM9iPnQuz@Rg6BkD7e4tu1NGnAvc14= {
MeshCentralServer2   host: 'my.domain.url',
MeshCentralServer2   connection: 'Upgrade',
MeshCentralServer2   pragma: 'no-cache',
MeshCentralServer2   'cache-control': 'no-cache',
MeshCentralServer2   'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
MeshCentralServer2   upgrade: 'websocket',
MeshCentralServer2   origin: 'https://my.domain.url',
MeshCentralServer2   'sec-websocket-version': '13',
MeshCentralServer2   'accept-encoding': 'gzip, deflate, br',
MeshCentralServer2   'accept-language': 'en-US,en;q=0.9',
MeshCentralServer2   cookie: '_hjSessionUser_3158721=eyJpZCI6ImY1ZGNjMzQwLTUyNTctNTBiNy04M2U5LTRkM2E2NGUyNDRkMCIsImNyZWF0ZWQiOjE3MDQ5NzYzOTY3MjMsImV4aXN0aW5nIjp0cnVlfQ==; xid=eyJ1c2VyaWQiOiJ1c2VyLy9kZWxhdmFsIiwiaXAiOiI5MS4yMTkuMTE0LjMzIiwieCI6Im5BNUlnWklkIiwidCI6Mjg0MzIwMzB9; xid.sig=BUhboC4lSKfbmSbkkE_Ua-vVQseilI1JGEH47Lr7MHLYQwEi_ZUA73xwfZ1KQG9J',
MeshCentralServer2   'sec-websocket-key': '8tV39NwHVkNxijhp2RukXQ==',
MeshCentralServer2   'sec-websocket-extensions': 'permessage-deflate; client_max_window_bits'
MeshCentralServer2 }
MeshCentralServer2 WEBREQUEST: (agentMachineIp) /meshrelay.ashx/.websocket?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=njxonuous8&auth=xbqkj@miZFAoJiwzkO4VOM$Bj51tDju3iqNRDweq@JEMQUFlnf9qjYFyi@J2rqLu5S6q6xKTGPh@64nneCUIEhNXa7ZPfW4SO87eRVwV8DWnmmeMFGM9iPnQuz@Rg6BkD7e4tu1NGnAvc14=
MeshCentralServer2 COOKIE: Decoded AESGCM cookie: {"userid":"user//myUserName","domainid":"","ip":"agentMachineIp","time":1705921559000,"dtime":252093}
MeshCentralServer2 RELAY: Relay holding: njxonuous8 (agentMachineIp) Authenticated
MeshCentralServer2 HTTPHEADERS: GET /meshrelay.ashx/.websocket?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=njxonuous8&rauth=Qgd@5GMPMAFi@ZpJ17OqRYSQo@fXbeKZDTBjBg69cVxxL9rPPJcBbe0y7$4UWdcWC9jUt6v0UlFgW4OmNFcAqAhIFJyVOiWMMajLovI5MnAchjJaclW0TQ== {
MeshCentralServer2   'sec-websocket-version': '13',
MeshCentralServer2   'sec-websocket-key': 'TG14jWR53mIBV1PvBIxAQQ==',
MeshCentralServer2   connection: 'Upgrade',
MeshCentralServer2   upgrade: 'websocket',
MeshCentralServer2   'sec-websocket-extensions': 'permessage-deflate; client_max_window_bits',
MeshCentralServer2   host: 'MeshCentralServer2Url:443'
MeshCentralServer2 }
MeshCentralServer2 WEBREQUEST: (MeshCentralServerIp) /meshrelay.ashx/.websocket?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=njxonuous8&rauth=Qgd@5GMPMAFi@ZpJ17OqRYSQo@fXbeKZDTBjBg69cVxxL9rPPJcBbe0y7$4UWdcWC9jUt6v0UlFgW4OmNFcAqAhIFJyVOiWMMajLovI5MnAchjJaclW0TQ==
MeshCentralServer2 COOKIE: Decoded AESGCM cookie: {"ruserid":"user//myUserName","x":"nA5IgZId","time":1705921559000,"dtime":252182}
MeshCentralServer2 RELAY: Relay connected: njxonuous8 (MeshCentralServerIp --> agentMachineIp)

and these are logs on both instances when I'm sucessfully connected...

MeshCentralServer: HTTPHEADERS: GET /meshrelay.ashx/.websocket?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=o89jhh9rhe&auth=LjySBtxIlcQXNuiBCwhkjoPFGUqL@9X5AljVZgjdedj$sYL7Mc$JbF7hPAER1xQPIf8OtEuOlpQErzWMkzBimj8KctE116ZJYpCB9m3erNZp@RdawLHAijIQA7NcHy$9usZ9d5PbO$Z33wc= {
MeshCentralServer:   host: 'my.domain.url',
MeshCentralServer:   connection: 'Upgrade',
MeshCentralServer:   pragma: 'no-cache',
MeshCentralServer:   'cache-control': 'no-cache',
MeshCentralServer:   'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
MeshCentralServer:   upgrade: 'websocket',
MeshCentralServer:   origin: 'https://my.domain.url',
MeshCentralServer:   'sec-websocket-version': '13',
MeshCentralServer:   'accept-encoding': 'gzip, deflate, br',
MeshCentralServer:   'accept-language': 'en-US,en;q=0.9',
MeshCentralServer:   cookie: '_hjSessionUser_3158721=eyJpZCI6ImY1ZGNjMzQwLTUyNTctNTBiNy04M2U5LTRkM2E2NGUyNDRkMCIsImNyZWF0ZWQiOjE3MDQ5NzYzOTY3MjMsImV4aXN0aW5nIjp0cnVlfQ==; xid=eyJ1c2VyaWQiOiJ1c2VyLy9kZWxhdmFsIiwiaXAiOiI5MS4yMTkuMTE0LjMzIiwieCI6Im5BNUlnWklkIiwidCI6Mjg0MzIwNzJ9; xid.sig=XjOEx7GERHfsvFxpYUIyfWRUgg5RdAnT12otqUk5JyslWKkwz_yu6tfyrNclbHvH',
MeshCentralServer:   'sec-websocket-key': 'aFGNsjiJvHX1js2tHHgGeA==',
MeshCentralServer:   'sec-websocket-extensions': 'permessage-deflate; client_max_window_bits'
MeshCentralServer: }
MeshCentralServer: WEBREQUEST: (agentMachineIp) /meshrelay.ashx/.websocket?browser=1&p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=o89jhh9rhe&auth=LjySBtxIlcQXNuiBCwhkjoPFGUqL@9X5AljVZgjdedj$sYL7Mc$JbF7hPAER1xQPIf8OtEuOlpQErzWMkzBimj8KctE116ZJYpCB9m3erNZp@RdawLHAijIQA7NcHy$9usZ9d5PbO$Z33wc=
MeshCentralServer: COOKIE: Decoded AESGCM cookie: {"userid":"user//myUserName","domainid":"","ip":"agentMachineIp","time":1705924350000,"dtime":16591}
MeshCentralServer: RELAY: Relay holding: o89jhh9rhe (agentMachineIp) Authenticated
MeshCentralServer: HTTPHEADERS: GET /meshrelay.ashx/.websocket?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=o89jhh9rhe&rauth=A3BuXW4hIb9CvAxoVpkRwdcHHRfNPav0e%40KLXa9hwabold75uDCrLRR1ZrvYndz1P3iLAhZIl2%24DgMImpr5iWcA%40lg%24zWp1aTsZFSfpPbJlxEy9S0qlvWQ== {
MeshCentralServer:   host: 'my.domain.url',
MeshCentralServer:   upgrade: 'websocket',
MeshCentralServer:   connection: 'Upgrade',
MeshCentralServer:   'sec-websocket-key': 'dJK7xk6TS0pDoeMqVuw4zU==',
MeshCentralServer:   'sec-websocket-version': '13',
MeshCentralServer:   'content-length': '0'
MeshCentralServer: }
MeshCentralServer: WEBREQUEST: (agentMachineIp) /meshrelay.ashx/.websocket?p=1&nodeid=node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ&id=o89jhh9rhe&rauth=A3BuXW4hIb9CvAxoVpkRwdcHHRfNPav0e%40KLXa9hwabold75uDCrLRR1ZrvYndz1P3iLAhZIl2%24DgMImpr5iWcA%40lg%24zWp1aTsZFSfpPbJlxEy9S0qlvWQ==
MeshCentralServer: COOKIE: Decoded AESGCM cookie: {"ruserid":"user//myUserName","x":"nA5IgZId","time":1705924350000,"dtime":16711}
MeshCentralServer: RELAY: Relay connected: o89jhh9rhe (agentMachineIp --> agentMachineIp)

MeshCentralServer2: DISPATCH: DispatchEvent [ '*', 'user//myUserName' ]
MeshCentralServer2: DISPATCH: DispatchEvent [
MeshCentralServer2:   'node//7wJbvpEQlV7PzdgqBlG22Ombfwnam3s2UigRGKo9KI89ZQAKPZkhlbpUPpNj1LqQ',
MeshCentralServer2:   '*',
MeshCentralServer2:   'mesh//GSFsi$3KvISY@z03knKNbO@MdcKimoqJrp4QeLRGFC8rw0BpOD8LvimWi5Tc22LJ',
MeshCentralServer2:   'ugrp//mi$fBKGP1MpMZalXDoUJOy6eo6IA8EuvGgXxg0BOPqUUHSJ1uh28WdlvTrUo8OsN'
MeshCentralServer2: ]
MeshCentralServer2: HTTPHEADERS: GET /images/icon-relay-notify10.png {
MeshCentralServer2:   host: 'my.domain.url',
MeshCentralServer2:   connection: 'keep-alive',
MeshCentralServer2:   'sec-ch-ua': '"Google Chrome";v="113", "Chromium";v="113", "Not-A.Brand";v="24"',
MeshCentralServer2:   'sec-ch-ua-mobile': '?0',
MeshCentralServer2:   'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
MeshCentralServer2:   'sec-ch-ua-platform': '"Linux"',
MeshCentralServer2:   accept: 'image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8',
MeshCentralServer2:   'sec-fetch-site': 'same-origin',
MeshCentralServer2:   'sec-fetch-mode': 'no-cors',
MeshCentralServer2:   'sec-fetch-dest': 'image',
MeshCentralServer2:   'accept-encoding': 'gzip, deflate, br',
MeshCentralServer2:   'accept-language': 'en-US,en;q=0.9',
MeshCentralServer2:   cookie: '_hjSessionUser_3158721=eyJpZCI6ImY1ZGNjMzQwLTUyNTctNTBiNy04M2U5LTRkM2E2NGUyNDRkMCIsImNyZWF0ZWQiOjE3MDQ5NzYzOTY3MjMsImV4aXN0aW5nIjp0cnVlfQ==; xid=eyJ1c2VyaWQiOiJ1c2VyLy9kZWxhdmFsIiwiaXAiOiI5MS4yMTkuMTE0LjMzIiwieCI6Im5BNUlnWklkIiwidCI6Mjg0MzIwNzJ9; xid.sig=XjOEx7GERHfsvFxpYUIyfWRUgg5RdAnT12otqUk5JyslWKkwz_yu6tfyrNclbHvH',
MeshCentralServer2:   'if-none-match': 'W/"129-18d20fca9c4"',
MeshCentralServer2:   'if-modified-since': 'Fri, 19 Jan 2024 09:10:45 GMT'
MeshCentralServer2: }
MeshCentralServer2: WEBREQUEST: (agentMachineIp) /images/icon-relay-notify10.png

Thing that caught my eye are RELAY logs. They are different in above three cases. When I get stucked on setup... relay is holding with Authenticated keyword but remains not connected, there are also some cookies error on the second machine. In stucked on connected relay is holding without Authenticated keyword and also remains not connected. Honestly I don't know how to interpret that...

si458 commented 6 months ago

what sticks out to me is these lines after first glimpes

MeshCentralServer2: COOKIE: ERR: Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data
MeshCentralServer2: COOKIE: ERR: Bad AESSHA cookie due to exception: Error: error:1C80006B:Provider routines::wrong final block length

will look into it for you when i get chance

kubamaruszczak commented 6 months ago

In the Server Peering section of MeshCentral documentation there is this step described:

Shared storage is compulsory to host user files and it must be accessible from all of the servers.

In my setup I have external MongoDB database and two instances of MeshCentral running on two separate machines. I've just replicated meshcentral-data directory on each server and I haven't set up shared storage for them since I use meshcentral to connect to my clients via ssh. I'm wondering if this could be somehow related to my issue.

paolor1965 commented 5 months ago

Good evening I also found that in the latest release the peer does not work correctly, the agent is displayed in the consoles of the 3 nodes, however only from a single node can I perform all the management operations, from the other 2 no Furthermore, by downgrading to the stable version everything works correctly my lab includes 3 vm ubuntu 20.04 lts with mongodb 3.6 replicated

paolor1965 commented 5 months ago

I reinstalled meshcentral on 3 nodes with AlmaLinux release 9.3 (Shamrock Pampas Cat) node v20.11.1 npm 10.4.0 MESHCENTRAL 1.1.21 MongoDB server version: 4.2.25 replicated successfully ( 2 secondary, 1 primary)

I successfully activated the peer, without shared storage for "my files" also in this case the agent is displayed on the three nodes, but from the other 2 when I click for example on connect (rdp) the connection remains on setup... On the node from which I perform remote control I regularly get the following log:

WEBREQUEST: (10.0.0.31) /meshrelay.ashx/.websocket?browser=1&p=2&nodeid=node//WreI5pYQN2sLuvHjaHA5G9RDeweMJCJG$XiQZ839Q9MZBHYTN51zPDBGd2mgKkqt&id=4zw62cat8xq&auth=bnGl6ol ZZYiSsmpzolwkJMMbr2K5eZlE4QNwIWYHwlX@1s3VpAOFdNvmafeNORRqxPb8$@l$saISHoKZhOksHZWL@4HTs30Ri4LgZKgRKSor6ahdc729ANljvQ== COOKIE: ERR: Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data COOKIE: ERR: Bad AESSHA cookie due to exception: Error: error:1C80006B:Provider routines::wrong final block length COOKIE: Decoded AESGCM cookie: {"userid":"user//admin","domainid":"","ps":1,"time":1708595314000,"dtime":658} RELAY: DesktopRelay: Connection (10.0.0.31) DISPATCH: DispatchEvent [

from the node on which it doesn't work I get this log: PEER: FTunnel mongodb3: Start connect to wss://10.0.0.33:443/meshrelay.ashx?browser=1&p=2&nodeid=node//WreI5pYQN2sLuvHjaHA5G9RDeweMJCJG$XiQZ839Q9MZBHYTN51zPDBGd2mgKkqt&id=eibcn7dm 858&auth=B3tH5UxsuCbQmHxvQYRkylo4Py3kU0GNqjItP0t6XtXQyXro4nNoTPM1xXS@IzvO6EfMuZV0GrhHD@DVRq7c5DzK@tLG$5ohWKtmVqqkV3pmCm6ugA0UFT7WKw== PEER: FTunnel mongodb3: Connected DISPATCH: DispatchEvent [

If I downgrade to the stable version the problem does not arise