adding a newly created user to a newly created usergroup fails for a mongodb replica set cluster and meshcentral peering

[schories]

summary

When the user is selected to become a member of the selected usergroup, nothing happens. No error message in meshcentral gui, no error message in meshcentral log or meshcentral console (when invoked interactively). Instead a strange document

• "_id": "undefined"
• "action": "createusergroup" is created in the database. Please see document1 below for details.

The process of adding the user to the usergroup can be repeated – as long as the meshcentral server is running – but the user won’t be added to the usergroup.

However, after fully restarting meshcentral server and then repeating to add the same user to the same usergroup, it suddenly works.

But only for users and usergroups existing prior to the restart of meshcentral. Meaning newly created usergroups and users will be affected from the same issue – until meshcentral server is fully restarted and the procedure is repeated one more time.

The strange document remains in the database as garbage forever. It won’t be cleaned up.

main findings

This issue only happens, when a mongodb replica set cluster is used.

If a normal single mongodb connection is used by a single meshcentral instance the issue is NOT occurring AS LONG AS changestream is set to false: "dbChangeStream": false (default).

However, once changestream is enabled ("dbChangeStream": true)

• EITHER for a single mongodb node based connection
• OR – as strictly required for meshcentral server peering – with a mongodb replica set cluster and meshcentral peering

the issue reproducibly occurs.

steps to exactly reproduce the issue

1. Create a MongoDB replica set cluster with 3 nodes.
2. Make sure you run 3 meshcentral nodes in peering configuration.
3. Create a new usergroup called “TESTGROUP1”
4. Create a new user called “testuser1”
5. Open the user page.
6. Click on “Add usegroup”.
7. Pick the NEWLY (important!) created usergroup “TESTGROUP1” from the pull down menu.
8. Click “ok” to confirm.

document1

{
  "_id": "undefined",
  "name": "TESTGROUP1",
  "desc": "",
  "domain": "",
  "links": {
    "user//testuser1": {
      "userid": "user//testuser1",
      "name": "testuser1",
      "rights": 1
    }
  },
  "action": "createusergroup",
  "ugrpid": "ugrp//pv1636bUgp43ti2CqKb4IoIbF2fv8eedfelEeDGQKE@2@xQjMIjwoBZTc7ILglyL",
  "nolog": 1
}

system information:

update: fixed URL typos meshcentral config.json (domains & secrets changed)

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "cert": "sample-backend-test.domain.com",
    "WANonly": true,
    "mpsport": 0,
        "_LANonly": true,
    "sessionKey": "h487gf87h87h87z6r5rd5f",
    "Port": 4443,
    "PortBind": "0.0.0.0",
    "AliasPort": 4443,
    "RedirPort": 4480,
    "_TlsOffload": "127.0.0.1",
    "MongoDB": "mongodb://meshcentral:supersecretpassword@server-meshcentral01:27017,server-meshcentral02:27017,server-meshcentral03:27017/?authSource=admin&replicaSet=meshcentral-replicaset",
    "MongoDbChangeStream": true,
    "MongodbBulkOperations": true,
    "Minify": 1,
    "AgentPing": 180,
    "BrowserPing": 180,
    "plugins": {
      "enabled": true,
      "list": [
        "sample_backend"
      ]
    }
  },
  "domains": {
    "": {
      "_title": "sample-Backend - MeshCentral",
      "_title2": "Servername",
      "minify": true,
      "newAccounts": true,
      "_userNameIsEmail": true
    }
  },
  "peers": {
    "serverId": "sample-backend-test01",
    "servers": {
      "sample-backend-test01": { "url": "wss://sample-backend-test01.domain.com:443/" },
      "sample-backend-test02": { "url": "wss://sample-backend-test02.domain.com:443/" },
      "sample-backend-test03": { "url": "wss://sample-backend-test03.domain.com:443/" }
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "admin@domain.com",
    "names": "sample-backend-test01.domain.com",
    "skipChallengeVerification": true,
    "production": false
  }
}

MongoDB

Version 7.0.5 (stable), 3 node replica cluster, running on Node.js v20.11.0

Meshcentral

> info
{
    "meshVersion": "v1.1.19",
    "nodeVersion": "v20.11.0",
    "runMode": "WAN mode",
    "productionMode": true,
    "database": "MongoDB",
    "dbChangeStream": true,
    "dbBulkOperations": true,
    "serverId": "sample-backend-test02",
    "plugins": [
        "sample_backend"
    ],
    "platform": "linux",
    "arch": "x64",
    "pid": 209578,
    "uptime": 10443.9680998,
    "cpuUsage": {
        "user": 20681063,
        "system": 2523042
    },
    "memoryUsage": {
        "rss": 162508800,
        "heapTotal": 50368512,
        "heapUsed": 43758304,
        "external": 52345040,
        "arrayBuffers": 50050745
    },
    "warnings": [],
    "allDevGroupManagers": []
}

[si458]

just wondering if you have watched the peering video and set it up correctly at all? https://www.youtube.com/watch?v=j8xIRGScju0 i dont think some of your config.json settings are correct? i would have to look into it

[schories]

Update: fixed URL typos

---

Thank you for the the quick reply. Yes, we watched the video.

Our config is based on it, as shown at 5:27 https://youtu.be/j8xIRGScju0?si=JAHvXuiSVf5542NZ&t=327

  "peers": {
    "serverId": "sample-backend-test01",
    "servers": {
      "sample-backend-test01": { "url": "wss://sample-backend-test01.domain.com:443/" },
      "sample-backend-test02": { "url": "wss://sample-backend-test02.domain.com:443/" },
      "sample-backend-test03": { "url": "wss://sample-backend-test03.domain.com:443/" }
    }
  },

Please, let me know where the setup may be wrong - I'd be happy if this would "just" be a configuration issue. Your advice is greatly appreciated!

Btw. Same issue also with MongoDB 6.x - so it seems not to be related to MongoDB.

[si458]

sorry was just a quick glimpse im just busy doing other stuff so will get round to looking at it properly later also wss://sample-backend-test02.domain:443/ is this a mistake in your copy+paste? shouldnt it be domain.com:443 ?

[schories]

Excellent catch! That wasn't the issue sadly. I fixed the typo above, I apologize.