Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
4.09k stars 549 forks source link

Agents on Linux VM QEMU/KVM load the CPU at 100% #4281

Open openncomp opened 2 years ago

openncomp commented 2 years ago

Server Software (please complete the following information):

Remote Device (please complete the following information):

Additional context Today I noticed on many Proxmox servers a strange CPU load by all Linux virtual machines. Having visited several of them, I found that the MeshCentral agent loads almost all CPU cores allocated to a particular VM. I checked all Proxmox servers, and similar things were observed everywhere. Nothing better than restarting the VM came to mind. After the reboot, everything returned to normal. Interestingly, there was no such problem on the Proxmox hosts themselves. How long ago and after what actions it happened, I do not know, but I can assume that it happened within a few days before I discovered it.

I will observe the behavior of these virtual machines, and if it happens again, I will be able to conduct tests or look at logs.

Perhaps someone has noticed such behavior?

I also found entries in the VM Data Agents console:

uncaughtException1: Error:  => EventEmitter.emit(): Event dispatch for 'end' on 'memoryStream' threw an exception: TypeError: undefined not callable (property 'trim' of [object Function]) in method '()'
uncaughtException1: Error: timers.onElapsed() callback handler on '()'  => waitExit() already in progress

Your config.json file

config.json ``` { "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json", "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.", "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.", "settings": { "tlsOffload": "127.0.0.1,172.22.22.10", "_maintenanceMode": true, "_ignoreAgentHashCheck": true, "cert": "---", "WANonly": true, "_LANonly": true, "sessionKey": "---", "manageCrossDomain": [ "---" ], "orphanAgentUser": "---", "_relayPort": 80, "_relayAliasPort": 443, "relayDNS": "---", "port": 80, "aliasPort": 443, "redirPort": 800, "redirAliasPort": 80, "allowLoginToken": true, "allowFraming": false, "trustedProxy": "127.0.0.1,172.22.22.10", "MongoDb": "mongodb://127.0.0.1:27017/meshcentral", "_noAgentUpdate": 1, "_agentUpdateSystem": 2, "agentsInRam": false, "AgentPing": 15, "_AgentPong": 55, "BrowserPing": 15, "_BrowserPong": 55, "_agentIdleTimeout": 10800, "_webPageLengthRandomization": false, "compression": true, "wsCompression": true, "agentWsCompression": true, "webRTC": false, "_webrtcConfig": { "iceServers": [ { "urls": "stun:stun.l.google.com:19302" }, { "urls": "stun:stun.services.mozilla.com" } ] }, "desktopMultiplex": true, "allowHighQualityDesktop": true, "_userAllowedIP": "127.0.0.1,192.168.1.0/24", "_agentAllowedIP": "192.168.0.100/24", "_userBlockedIP": "file:PSD.txt", "_agentBlockedIP": "file:PSD.txt", "maxInvalidLogin": { "exclude": "172.22.22.0/24", "time": 30, "count": 5, "coolofftime": 360 }, "plugins": { "enabled": true } }, "domaindefaults": { "siteStyle": 2, "minify": true, "newAccounts": false, "_userNameIsEmail": true, "unknownUserRootRedirect": "http://www.epwr.ru/badadvice/txt_1.php", "loginKey": ["---"], "_agentKey": ["---"], "terminal": { "launchCommand": { "linux": "cd ~\n", "_darwin": "clear\necho \"Hello MacOS\"\n", "_freebsd": "clear\necho \"Hello FreeBSD\"\n" } }, "passwordRequirements": { "min": 10, "max": 128, "upper": 1, "lower": 1, "numeric": 1, "_nonalpha": 1, "_reset": 90, "email2factor": true, "sms2factor": false, "push2factor": false, "otp2factor": false, "backupcode2factor": true, "single2factorWarning": true, "lock2factor": false, "_force2factor": true, "_skip2factor": "127.0.0.1,172.22.22.0/24", "oldPasswordBan": 3, "banCommonPasswords": true, "loginTokens": [ "---", "---" ], "twoFactorTimeout": 600, "autofido2fa": false, "_maxfidokeys": null, "allowaccountreset": false }, "agentConfig": [ "ignoreProxyFile=1", "controlChannelIdleTimeout=15" ] }, "domains": { "": { "_siteStyle": 2, "title": "---", "_title2": "---", "_titlePicture": "", "_loginPicture": "", "_welcomeText": "Sample Text on Login Page.", "_welcomePicture": "", "_welcomePictureFullScreen": true, "_meshMessengerTitle": "Text that will be displayed on the top of the messenger window when no username or device name is displayed", "_meshMessengerPicture": "", "_loginfooter": "This is a private server.", "hide": 5, "_minify": true, "_PreconfiguredRemoteInput": [ { "name": "---", "value": "---" }, { "name": "---", "value": "---" } ], "_altMessenging": { "name": "Telegram", "url": "---" }, "mstsc": true, "novnc": true, "ssh": true, "deviceMeshRouterLinks": { "rdp": true, "ssh": true, "scp": true, "extralinks": [ { "name": "HTTP", "protocol": "http", "port": 80, "_ip": "---", "_filter": [ "mesh/(domainid)/(meshid)", "node/(domainid)/(nodeid)" ] }, { "name": "HTTPS", "protocol": "https", "port": 443 } ] }, "_newAccounts": false, "_agentKey": ["---"], "_userAllowedIP": "127.0.0.1,172.22.22.0/24", "_userNameIsEmail": true, "certUrl": "---", "ipkvm": true, "localSessionRecording": false, "sessionRecording": { "_onlySelectedUsers": true, "_onlySelectedUserGroups": true, "_onlySelectedDeviceGroups": true, "filepath": "/home/meshcentral/meshcentral-files/records/---", "index": true, "_maxRecordings": 1, "maxRecordingDays": 90, "_maxRecordingSizeMegabytes": 3, "__protocols__": "Is an array: 1 = Terminal, 2 = Desktop, 5 = Files, 100 = Intel AMT WSMAN, 101 = Intel AMT Redirection, 200 = Messenger", "protocols": [ 1, 2, 5, 100, 101, 200 ] }, "agentCustomization": { "displayName": "---", "description": "---", "companyName": "---", "serviceName": "---", "fileName": "---" }, "assistantCustomization": { "title": "---", "_image": "assistant-logo.png", "fileName": "---" }, "androidCustomization": { "title": "---", "subtitle": "Network Application", "_image": "assistant-logo.png" }, "consentMessages": { "Title": "---", "Desktop": "{0} запрашивает доступ к вашему компьютеру. Предоставить доступ?", "Terminal": "{0} запрашивает доступ к командной строке. Предоставить доступ?", "Files": "{0} запрашивает доступ к файлам. Предоставить доступ?" }, "notificationMessages": { "Title": "---", "Desktop": "{0} подключился к вашему компьютеру.", "Terminal": "{0} подключился к вашему компьютеру.", "Files": "{0} подключился к вашему компьютеру." }, "smtp": { "host": "---", "port": 465, "from": "---", "user": "---", "pass": "---", "__tls__": "When 'tls' is set to true, TLS is used immidiatly when connecting. For SMTP servers that use TLSSTART, set this to 'false' and TLS will still be used.", "tls": true, "___tlscertcheck__": "When set to false, the TLS certificate of the SMTP server is not checked.", "_tlscertcheck": false, "__tlsstrict__": "When set to true, TLS cypher setup is more limited, SSLv2 and SSLv3 are not allowed.", "_tlsstrict": true }, "_GuestDeviceSharing": { "maxSessionTime": 1440 } }, "temp": { "dns": "---", "certUrl": "---", "title": "---", "_title2": "", "_titlePicture": "", "_loginPicture": "", "_welcomeText": "Sample Text on Login Page.", "_welcomePicture": "", "_welcomePictureFullScreen": true, "_meshMessengerTitle": "Text that will be displayed on the top of the messenger window when no username or device name is displayed", "_meshMessengerPicture": "", "_loginfooter": "---.", "hide": 1, "_minify": true, "_altMessenging": { "name": "Telegram", "url": "---" }, "mstsc": true, "novnc": true, "ssh": true, "_newAccounts": false, "_agentKey": ["---"], "_userAllowedIP": "127.0.0.1,172.22.22.0/24", "_userNameIsEmail": true, "ipkvm": true, "localSessionRecording": false, "sessionRecording": { "_onlySelectedUsers": true, "_onlySelectedUserGroups": true, "_onlySelectedDeviceGroups": true, "filepath": "/home/meshcentral/meshcentral-files/records/---", "index": true, "_maxRecordings": 1, "maxRecordingDays": 365, "_maxRecordingSizeMegabytes": 3, "__protocols__": "Is an array: 1 = Terminal, 2 = Desktop, 5 = Files, 100 = Intel AMT WSMAN, 101 = Intel AMT Redirection, 200 = Messenger", "protocols": [ 1, 2, 5, 100, 101, 200 ] }, "agentCustomization": { "displayName": "---", "description": "---", "companyName": "---", "serviceName": "---", "fileName": "---" }, "assistantCustomization": { "title": "Remote Assistant", "_image": "assistant-logo.png", "fileName": "---" }, "androidCustomization": { "title": "Network Manager App", "subtitle": "Network Application", "_image": "assistant-logo.png" }, "_GuestDeviceSharing": { "maxSessionTime": 1440 } } } } ```
krayon007 commented 2 years ago

I'm going to try to setup Proxmox VE and set up a test environment, to add it to my VMs for testing.

openncomp commented 2 years ago

I could roll back my version of MeshCentral to update to 1.0.53 again if it would do any good.

I also observe the behavior of the MC on virtual machines, but after that reboot everything is calm.

krayon007 commented 2 years ago

Your linux instance running inside proxmox ve that is seeing the 100% cpu usage... Is the host running kernel based virtualization, or container based virtualization?

si458 commented 7 months ago

I see this happen every so often but I've narrowed it down to, me restarting my meshcentral test server very often and very quickly when I'm changing code, and I'm talking like 5 restarts within 3mins