RunCommand not working consistently in peering environment

[kubamaruszczak]

Describe the bug After this feature have been added https://github.com/Ylianst/MeshCentral/issues/6404 Send Commands seemed to work in peering environment. I started to use it and I still encounter issue when passing commands to agents connected to peer I'm executing commands from. When I execute below command on peer agent is connected to I receive OK in the output. However, command is not executed on the device and if I add --reply flag to the command it gets stucked. If I would do the same on the other peer it will work all fine.

/home/ec2-user/.nvm/versions/node/v18.17.1/bin/node /home/ec2-user/meshcentral/node_modules/meshcentral/meshctrl --loginuser $NAME --loginkey $LoginTokenKey --url ws://localhost:443 RunCommand --id 'node//vCgyHmO$lqTsvaMfSRoHs20mUbg4FDNQu20Twafg20FnSvYjEvCNFZcEEYbqiK@p' --run 'touch /tmp/mesh1' --reply

To Reproduce Steps to reproduce the behavior:

1. connect device to peer1
2. connect to the peer1 and execute RunCommand shell command with --reply flag (in that case it will get stucked and command won't be executed)
3. connect to peer2 and execute RunCommand shell command with --reply flag (in that case mesh_id will be returned and command will be executed)

Expected behavior RunCommand should work no matter which host is used to execute command on a device

Server Software:

• OS: Amazon Linux 2023
• Version: 1.1.31
• Node: 18.17.1

Your config.json file

{
  "settings": {
    "cert": "my.domain.url",
    "WANonly": true,
    "LANonly": false,
    "sessionKey": "MyReallySecretPassword1",
    "mongoDb": "mongodb://MeshCentralMongoDBDNS:27017/meshcentral?replicaSet=rs0",
    "mongoDbChangeStream": true,
    "port": 443,
    "redirPort": 80,
    "nightMode": 0,
    "tlsOffload": true,
    "maxInvalidLogin": {
      "time": 10,
      "count": 10,
      "coolofftime": 10
      },
    "rootCertCommonName": "xxx",
    "mpshighsecurity": true,
    "allowLoginToken": true,
    "authLog": "meshcentral-data/auth.log"
  },
  "domains": {
    "": {
      "title": "MeshCentral",
      "newAccounts": false,
      "userNameIsEmail": true,
      "amtAcmActivation": {
        "certs": {
          "mycertname": {
            "certfiles": [ "xxx.crt", "XXX.crt" ],
            "keyfile": "xxx.key"
          }
        }
      }
    }
  },
  "peers": {
    "serverId": "peer_id",
    "servers": {
      "meshc1": {"url": "ws://peer1_url:443/"},
      "meshc2": {"url": "ws://peer2_url:443/"}
    }
  }
}

[si458]

i have noticed quite a lot of problems with the peering so im working my way through fixing them ill add this to my list to try look at next week! 👍

[kubamaruszczak]

@si458 hi! any progress regarding peering issues?

[si458]

@kubamaruszczak im still looking into this, im a little slow at the moment as i have my 'work' to do first

[ijustw0rkhere]

I can confirm this issue as well. Commands will run fine from one peer, but not necessarily another. I have yet to identify a rhyme or reason.

[kubamaruszczak]

One thing catches my attention. I have one peer running and it has peers section in its config. I've tried to track RunCommand flow and it turns out that when peering is configured commands are sent to agents using the DispatchMessage function: https://github.com/Ylianst/MeshCentral/blob/aa7767f37cb91aeb6f3511c03877c92c9e3dfe62/meshuser.js#L3035

Tracking that further... in multiserver.js the DispatchMessage function iterates over and send commands to all peers in obj.peerServers: https://github.com/Ylianst/MeshCentral/blob/aa7767f37cb91aeb6f3511c03877c92c9e3dfe62/multiserver.js#L412

When only one peer is used this object is empty which results in not sending commands at all. This might be the reason why executing command from another peer works fine. When second peer is connected it has first peer in obj.peerServers and command is sent.