search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
4.25k stars 569 forks source link

Agent Memory Issue #754

Open ryanblenis opened 4 years ago

ryanblenis commented 4 years ago

Hi Guys,

It looks like when you start a remote terminal (powershell or cmd) then disconnect, the conhost.exe process that is spawned is left and never cleaned up. Hence if you connect then disconnect 10 times from powershell or cmd the 10 processes sit there and take up memory until MeshAgent.exe crashes or is otherwise restarted.

This was found on Win 10 1903 with the new virtual terminal. Not sure if it affects the legacy code as well.

PathfinderNetworks commented 4 years ago

I am seeing the same thing. I just tried it on a Server 2019 box. Exactly as ryanblenis describes. I also tried it on Server 2012 but it only seems to spawn one additional conhost.exe process. If I disconnect it doesn't end that process. But it also doesn't seem to spawn any extra for future connections.

krayon007 commented 4 years ago

Ok, I figured it out... This only affects systems that are using the new ConPTY API... I created a new PseudoConsole, for each session, and cleaned up the child process on exit, but forgot to close the PseudoConsole. I'll fix fix the cleanup code, and give Ylian the headsup.

krayon007 commented 4 years ago

This might take a bit of time to resolve, as I ended up having to file a bug with Microsoft, becuase the ClosePseudoConsole API doesn't work correctly. MSFT has apparently acknowledged the bug, and linked it with another bug entry with similar behavior...

In the meantime, I'll work on a work-around, that involved spawning a child process when invoking a root terminal, this way when the process exits, it will clean up correctly, as that is the current behavior when you spawn a user terminal. Not optimal in terms of efficiency, but at least it wont leak.

ryanblenis commented 4 years ago

It looks like you've got a workaround or fix in place (for a while now). If it is a workaround, should I leave this open until a fix is implemented?

Otherwise just feel free to close. Thank you as always!

vedran-dobos commented 4 years ago

I am seeing this very often in the past days, on various machines (Win 10, Win7). But I'm not even sure I have to connect, as I receive complaints from users about very slow PCs, and I have not connected to them in days or weeks. It's getting to the point that I have to uninstall the agent from the users' PCs for them to have a normal PC again.

Isn't there anything that can be done? Is it not affecting everybody?

krayon007 commented 4 years ago

I am seeing this very often in the past days, on various machines (Win 10, Win7). But I'm not even sure I have to connect, as I receive complaints from users about very slow PCs, and I have not connected to them in days or weeks. It's getting to the point that I have to uninstall the agent from the users' PCs for them to have a normal PC again.

Isn't there anything that can be done? Is it not affecting everybody?

This looks to be unrelated to the previous issue. Can you post more details with regards to the server version you are running, and other symptoms. Is the agent appear to be consuming 100%cpu, excessive memory, etc? Normally when you don't connect to the agent, it should be completely idle, except for an occasional ping/pong with the server.

Bryan

vedran-dobos commented 4 years ago

This looks to be unrelated to the previous issue. Can you post more details with regards to the server version you are running, and other symptoms. Is the agent appear to be consuming 100%cpu, excessive memory, etc? Normally when you don't connect to the agent, it should be completely idle, except for an occasional ping/pong with the server.

I do IT support for a few users around me and saw the issue randomly appear. Users complaining about slow PCs (mostly Win 10). Was blaming it on Windows or an update in the beginning, until I narrowed it down to Meshcentral. And I don't remember connecting to their PCs for the last couple of days before their complaint. Cannot reproduce it on demand unfortunately.

Last user that reported this was this week. Don't remember the exact date (Monday-Wednesday), and therefore the server version, as I update almost diligently to the latest version, should have been 0.6.65, or the one before it. Agent was installed 2-3 months ago. Windows 10 Home x64 1909 (18363), i5 4th Gen, 4GB RAM, 360GB SSD. Luckily I was on site to witness it firsthand. And I'm sure I did not connect to it in the past days. The PC was slow as an old 486. In task manager the Mesh Agent Service showed to be using around 50-100MB of ram and a few CPU %, but as soon as I killed it, the total CPU and RAM usage plummeted and the PC became usable again. I uninstalled the agent afterwards.

The latest issue I saw was yesterday, using the latest 0.6.65 server version. Agent was downloaded from the same 0.6.65 server version on a Windows 7 x64 PC and after a few hours CPU and RAM usage were at 99-100% constantly. I was however connecting to the Desktop a bunch of times (10-15) before it happened. I removed the agent immediately afterwards.

I'd gladly help in any way I can.

V