Zombie child processes eating large amounts of RAM

[niklasholm]

I've discovered that when building large amounts of packages, after a while the physical memory usage has increased drastically for no apparent reason; the physical memory usage by processes in Task Manager doesn't add up to the actual memory usage. Using RAMMap, I can see that there are many tens of thousands of terminated msys processes (bash, tee, sed, xargs etc) each taking up 24 K of memory, adding up to more than 3 GB of "lost" memory.

A similar issue is described by a Google engineer while building Chrome, although, the cause in that specific case does not apply here (I do not have CcmExec.exe running). From what I can understand, it's either caused by not closing the process handles correctly, in which case it might be a bug in msys, or it's caused by a bug in Windows. Unfortunately, the tool provided in the link does not reveal what process(es) keeps the zombies alive. Further investigation is required.

[niklasholm]

No comment on this? It's quite a huge problem and very detrimental to MSYS2's use as a build environment.

[mati865]

I haven't seen issue like that but there were many other reports, maybe are you using software which injects dlls to running processes? You might want to check https://cygwin.com/faq/faq.html#faq.using.bloda

[niklasholm]

Well, pretty much every anti-virus solution is on that list, so yeah. Currently, I'm using Avast. Is going completely without AV the only solution? What is Cygwin/msys doing that other processes without this problem (i.e. most) doing differently?

[mingwandroid]

Just use Windows defender. It's the only av you need. The others aren't very good IMHO.

[niklasholm]

Windows Defender is also on that list.

[mingwandroid]

Heh, well, it's never caused me much trouble (unlike all the others).

[niklasholm]

Also I'm on Windows 7, but I guess Security Essentials is basically the same minus the rootkit/boot protection. If someone could confirm that MSE doesn't interfere with msys, I'd consider giving it a go, although reluctantly because Avast has served me well and MSE lags behind in protection scores.

[mati865]

To be honest number of false positives from MSE/Windows 10 Defender is overwhelming. Both of them would randomly detect threat when updating MSYS2 and virustotal just proves it was false positive.

[niklasholm]

So... Either way, you're screwed, nothing can be done except uninstalling all AV?

[mati865]

@niklasholm you can exclude MSYS2 directory in your AV.

[niklasholm]

I've tried, doesn't make a damn difference.

[mati865]

I'm not using Avast but usually there are 2 separate exclude directories, one for the scanner and one for the real-time protection.

[mingwandroid]

I have never suffered a virus in my life. I take just the normal precautions but have never run a (continuously scanning) virus scanner.

The industry is largely based on snake oil (I've looked at real virus detection 'logic' it was beyond awful). They make computers far slower than they otherwise be and basically generate a load of heat based mostly on fear. Don't bother with it, John McAfee is right.

[niklasholm]

@mati865 Yes, I've excluded it everywhere still doesn't make a difference. Disabling it completely also doesn't make a difference.

@mingwandroid Most IT security departments would disagree with you. And how do you know that if you don't have AV? Most malware does their best to hide from the user. AV stops loads of attacks, that's a fact. While it is true that very security conscious users can survive in the wild without AV, most people using a computer wouldn't.

[mingwandroid]

https://www.howtogeek.com/225385/what%E2%80%99s-the-best-antivirus-for-windows-10-is-windows-defender-good-enough/

This in particular gives me cause for concern:

Furthermore, security is about more than raw protection scores. Other antivirus programs may occasionally do a bit better in monthly tests, but they also come with a lot of bloat, like browser extensions that actually make you less safe, registry cleaners that are terrible and unnecesary, loads of unsafe junkware, and even the ability to track your browsing habits so they can make money. Furthermore, the way they hook themselves into your browser and operating system often causes more problems than it solves. Something that protects you against viruses but opens you up to other vectors of attack is not good security.

[mingwandroid]

@niklasholm to debug this you probably need to use ProcMon and see what DLLs you AV software is hooking.

[niklasholm]

All that "bloat" is completely optional. I only have File Shield and Behaviour Shield installed. Besides, How-To Geek does not offer any evidence to support their claims that it makes the system less secure, so I wouldn't be too concerned. Throwing out the baby with the bathwater by getting rid of AV completely certainly does not improve security.

I'll try to find something with ProcMon.

[revelator]

yep also running into this, the problem is that win10 changed the way of handling the standby list to be more in line with linux. the problem with that is that it does a piss poor job of it ;S sadly in msys2 case 16 gb ram is not enough my pc will grind to a halt after building two packages and even installing a memory cleaner specifically setup to free the standbylist wont help as win10 refuses to release the memory even in admin mode.

[woot000]

Did either of you two @niklasholm @revelator ever find a solution to this? I'm currently experiencing the exact same problem right now. I don't have any antivirus interfering with MSYS2 and I have no idea what could possibly be causing this

[revelator]

sadly no. for my part it turned out that certain software triggers a memory leak in the cygwin/msys2 runtime, ntvdmx64 for instance. but it seems there are more examples. also running msys2 for a long time even without these seems to trigger the memory leak just happens a lot slower.

[revelator]

basically anything that hooks system dll's trigger it msi afterburner is also on that list though far from the worst offender. the ntvdmx64 dev tracked it to the cygwin heap manager but other than zeroing the memory map of every process after exit (might cause unforeseen problems maybe) im blank as for how to fix it.

[revelator]

https://github.com/leecher1337/ntvdmx64/issues/100

this was the bug report i filed with the ntvdmx64 dev, the function that seems to cause this is highlighted. he provided a fix but as it turned out it does not work 100% on win10 and later, win7 is not affected anymore atleast in regards to ntvdmx64.

[woot000]

basically anything that hooks system dll's trigger it msi afterburner is also on that list though far from the worst offender. the ntvdmx64 dev tracked it to the cygwin heap manager but other than zeroing the memory map of every process after exit (might cause unforeseen problems maybe) im blank as for how to fix it.

Is there any way to check what program could be causing this? I've tried replicating this issue on a virtual machine by installing various programs already installed on my host machine to no avail

[Biswa96]

Is there any way to check what program could be causing this?

The issue is not caused by msys2 directly. See this list of programs that can interfere with msys2 processes https://cygwin.com/faq/faq.html#faq.using.bloda (4.43)

[revelator]

not sure if there is a way to reliably check for interference with the msys2 runtime no, but all programs who hook system dll's might interfere in some way (not nessesarily breaking it), so if one notices the standby list growing out of proportions when msys2 is running its a safe bet it will lead to breakage at some point. these programs could then be added to the list from cygwin warning others to be on the lookout for those programs. microsoft is slowly mowing away from allowing dll hooking from third parties anyway as it causes to many stability problems in windows.

[woot000]

In case anyone else comes across this issue, the cause of the zombie processes for me was the latest AMD Software: Adrenalin Edition. For some reason, the drivers released in 2024 (24.1.1 and 24.2.1) are prone to creating lots of zombie child processes. Version 23.12.1 of the AMD drivers is the latest version that doesn't create zombies.