New agents are not added when sessionrecording is enabled

[AntonAndreevichMoroz]

Describe the bug When we activate the functionality of recording connection sans, then when installing a new agent, it does not appear in the MeshCentral interface. If you restart the server service, the agent appears.

To Reproduce Steps to reproduce the behavior:

1. Enable session recording in the config file (sessionRecording).
2. We check the operation of the recording, it works
3. Install the agent on a new PC (maybe more than one)
4. New agents do not appear on the server
5. Restart the MeshCentral Server
6. Seeing new agents

Expected behavior Agents immediately appear on the server as usual

Server Software (please complete the following information):

• OS: Debian 10
• Virtualization: KVM
• Network: hybrid mode, reverse proxy (NGINX) with ssl offload
• Version: 1.0.70
• Node: v16.14.1

Client Device (please complete the following information):

• Device: PC, notebook
• OS: Windows 10/11
• Network: Remote over WAN
• Browser: Google Chrome

Remote Device (please complete the following information):

• Device: PC, notebook, server
• OS: Windows 10, Windows Server 2016, Debian 10
• Network: Remote over WAN
• Current Core Version (if known): Mar 25 2022, 3863795360

Your config.json file

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "cert": "rm.company.ru",
    "relayDNS": "relay.rm.company.ru",
    "MongoDb": "mongodb://127.0.0.1:27017/meshcentral",
    "_WANonly": true,
    "_LANonly": true,
    "port": 443,
    "aliasPort": 443,
    "redirPort": 80,
    "redirAliasPort": 80,
    "relayPort": 453,
    "relayAliasPort": 443,
    "MpsPort": 44330,
    "MpsAliasPort": 4433,
    "MpsTlsOffload": true,
    "browserPing": 60,
    "browserPong": 30,
    "AgentPong": 30,
    "allowLoginToken": true,
    "AllowFraming": true,
    "manageAllDeviceGroups": [ "user//admin" ],
    "manageCrossDomain": [ "user//admin" ],
    "tlsOffload": "192.168.XX.XX",
    "trustedProxy": "192.168.XX.XX",
    "_plugins": { "enabled": true },
    "ClickOnce": true,
    "maxInvalidLogin": {
      "time": 10,
      "count": 5,
      "coolofftime": 10
    },
    "Compression": true,
    "WsCompression": true,
    "AgentWsCompression": true,
    "desktopMultiplex": true,
    "allowHighQualityDesktop": true
  },
  "domains": {
    "": {
      "title": "Remote Access System",
      "title2": "B2B Cluster",
      "minify": true,
      "ignoreAgentHashCheck": true,
      "newAccounts": false,
      "hide": 5,
      "agentNoProxy": false,
      "novnc": true,
      "mstsc": true,
      "ssh": true,
      "_userNameIsEmail": true,
      "CertUrl": "https://rm.company.ru:443/",
      "AutoRemoveInactiveDevices": 45,
      "myServer": {
        "Backup": true,
        "Restore": true,
        "Upgrade": true,
        "ErrorLog": true,
        "Console": true,
        "Trace": true
      },
      "unknownUserRootRedirect": "https://rm.company.ru/auth-azure",
      "DeviceSearchBarServerAndClientName": true,
      "authStrategies": {
        "azure": {
          "callbackurl": "https://rm.company.ru/auth-azure-callback",
          "newAccountsRights": [ "nonewgroups", "notools" ],
          "newAccounts": true,
          "clientid": "XXXXXXX",
          "clientsecret": "XXXXXXX",
          "tenantid": "XXXXXXXX"
        }
      },
      "deviceMeshRouterLinks": {
        "rdp": true,
        "ssh": true,
        "scp": true,
        "extralinks": [
          {
            "name": "MCRDesk",
            "protocol": "mcrdesktop"
          },
          {
            "name": "MCRFiles",
            "protocol": "mcrfiles"
          },
          {
            "name": "PVE",
            "protocol": "https",
            "port": 8006,
            "filter": [ "tag:PVE" ]
          },
          {
            "name": "PVEBKP",
            "protocol": "https",
            "port": 8007,
            "filter": [ "tag:PVEBKP" ]
          },
          {
            "name": "Winbox",
            "protocol": "custom",
            "port": 8291,
            "filter": [ "tag:MIKROTIK" ]
          }
        ]
      },
      "limits": {
         "_maxDevices": 100,
         "_maxUserAccounts": 100,
         "_maxUserSessions": 100,
         "_maxAgentSessions": 100,
         "maxSingleUserSessions": 5
      },
      "userSessionIdleTimeout": 60,
      "terminal": {
        "launchCommand": {
          "linux": "su -\n",
        }
      },
      "sessionRecording": {
        "_onlyselectedUsers": true,
        "_onlySelectedDeviceGroups": false,
        "filepath": "/root/meshcentral-records",
        "index": true,
        "maxRecordingDays": 30,
        "maxRecordingSizeMegabytes": 30000,
        "__protocols__": "Is an array: 1 = Terminal, 2 = Desktop, 5 = Files, 100 = Intel AMT WSMAN, 101 = Intel AMT Redirection, 200 = Messenger",
        "protocols": [ 1, 2, 5 ]
      },
      "_httpHeaders": {
        "Strict-Transport-Security": "max-age=360000"
      }
    }
  }
}

[si458]

this is a very common issue that is being reported over and over again

the only way of fixing it is

restart the meshcentral server every so often OR disable session recording completely

ill tag other issues into this issue so you can see #4405 #4374 #3868 #3237 #3111 #2896

@Ylianst is away for 6 weeks in september, so if he has chance to fix it before then, brilliant! if not you will have to just hang fire sadly

[Ylianst]

Oh dear. Ok, I am going to try to reproduce this.

[si458]

Oh dear. Ok, I am going to try to reproduce this.

@Ylianst the issue i had was it never stopped working after an exact number of mins or hours, it was always random mins or hours, so it was hard to reproduce

but the only thing in common everyone is having was session recording being enabled, then users cant connect to devices and new devices cant be added, so i hope this helps!

[Ylianst]

When you say "session recording being enabled", i just need to add the "sessionRecording" section to the config.json, reset the server and I don't need to have a remote desktop session actually recording for the problem to occur? I just need to have the recording settings in place and devices don't show?

It would be interesting to see if there is anything in the browser console when this happens. If you hit refresh on the browser to reset the web page, so the device show then? Or is this a server side issue?

[VINISHVV]

@Ylianst

I think this can be reproduced using below method.

Enable session recording in server. Activate record for user. Connect few system simultaneously for a few minute. Install a new agent in any pc, it will not came up in meshcentral, if we run the agent in cmd then we get messsage connecting to server but it will hang there.

[si458]

When you say "session recording being enabled", i just need to add the "sessionRecording" section to the config.json, reset the server and I don't need to have a remote desktop session actually recording for the problem to occur? I just need to have the recording settings in place and devices don't show?

It would be interesting to see if there is anything in the browser console when this happens. If you hit refresh on the browser to reset the web page, so the device show then? Or is this a server side issue?

i believe its a server-side issue, because if you have session recording disabled, you dont have this issue, but with it enabled i had this issue, i could enable the session recording, not use the server to connect to any devices, then after a period of time, i couldnt connect to devices, but then also i couldnt get new devices to connect either, and i hadnt even connected to anything, only had the session recording enabled,

refresh of the browser, made no difference, use a different browser, no new device was shown, even a different computer all together, the new device wasnt shown

with my issue here #4374 i couldnt even get the assistant to connect

[AntonAndreevichMoroz]

When you say "session recording being enabled", i just need to add the "sessionRecording" section to the config.json, reset the server and I don't need to have a remote desktop session actually recording for the problem to occur? I just need to have the recording settings in place and devices don't show?

It would be interesting to see if there is anything in the browser console when this happens. If you hit refresh on the browser to reset the web page, so the device show then? Or is this a server side issue?

I noticed that if an entry for an individual user was just enabled and that user was not using, I.E. Records were not kept, then everything worked. But recently we decided to enable for everyone and then caught the problem again. From what I can conclude that the problem occurs if the records are actually maintained after the server has started.

[Ylianst]

I am going to turn on recording on my test servers and hopefully I will see this happen. Also, does anything show up in meshcentral-data/mesherrors.txt?

[si458]

I am going to turn on recording on my test servers and hopefully I will see this happen. Also, does anything show up in meshcentral-data/mesherrors.txt?

nothing in my case, its blank, empty

[AntonAndreevichMoroz]

I am going to turn on recording on my test servers and hopefully I will see this happen. Also, does anything show up in meshcentral-data/mesherrors.txt?

Ours is also empty. There are only OAuth2 token refresh errors, but they were there even before the entry was enabled. I don't think it's related.

[AntonAndreevichMoroz]

Greetings, @Ylianst! Thanks for the recent improved notification features. But I would like to remind you of the problem in this address. Session recording is a very important security feature. The MeshCentral product is mega-functional, which raises the question of the need for increased security and careful handling of such a tool. To do this, I would very much like to have a working and reliable tool for recording the actions of employees. I would be happy to provide additional information if it helps to resolve this issue as soon as possible. Best wishes.

[AntonAndreevichMoroz]

I would like to add that in another installation with such characteristics, the problem has not yet been observed. But we will continue to observe, since the server is new, there are not so many devices on it yet, and administrators too.

OS: Debian 11 DB: PostgreSQL 15.0 Virtualization: KVM Network: hybrid mode, reverse proxy (NGINX) with ssl offload Mesh Version: 1.0.97 Node: v18.12.0

[zhuravlev-alexey]

Faced this problem. The problem disappeared after changing the database from MongoDB to MariaDB. Looks like it's something specific to MongoDB

[Baltix]

@zhuravlev-alexey this issue when agents are not added isn't specific to MongoDB - I use default NeDB local db and meet this issue in few days after enabled "sessionRecording" in config.json. I've noticed, that SQLite database is supported now, I can try to switch to SQLite3 - there is a nice tutorial which Meshcentral DB choose: https://youtube.com/watch?v=4aMpQMHSjjo

[openncomp]

I recently changed mongodb to mariadb and the problem remained, as written above, when activating automatic recording.

[si458]

i have been able to replicate the issue, if others would like to try and see if they are the same

1. enable sessionrecording
2. connect to 2 different computers in 2 different browser (2 sessions total)
3. play random youtube video full screen for 10mins on both comps (this helps generate file sizes for recordings)
4. disconnect from both computers
5. reconnect to both computers
6. try to add an agent or assistant

[si458]

ok my PR #5490 fixes sessionrecording if you have index set as false (default) i believe im still trying to fix it if you are using index=true for the 10 seconds intervals

[si458]

DEBUG INFO HERE FOR OTHERS IF THEY WANT TO INVESTIGATE WITH ME

obj.getAccelerator in certoperations.js gets called, however accelerator.on('message') never gets called from the fork for some reason?

[si458]

Reopening as only fixed it always indexing, still need to fix the indexing