Windows Defender/Anti-malware Causing Performance Issues after CU update

[rdodev]

So, after uninstalling my old Ubuntu Bash lxrun /uninstall /full and after the update lxrun /install I noticed, as promised, that processes running in WSL could be seen in the Windows task manager (yay!); however, there is an issue that consistently causes Windows Defender/Anti-Malware service to consume over 50% of CPU utilization whenever code compilation or builds are happening inside bash. Back last version (before the processes were visible to Windows -- or more sandboxed, not sure) this was not an issue at all. So I'm wondering if this is a worthy trade off? I, for one, as a dev, would prefer to maximize build/compile performance even if that means reverting to a more sandboxed environ (or at least the option to do so).

[aseering]

For what it's worth, I always add an exclusion for my big source-code build directories to Windows Defender, for exactly this reason. I trust my own code, and the performance hit is too big otherwise.

[kumarharsh]

Adding to @aseering's comment, this is a problem even with yarn: https://github.com/yarnpkg/yarn/issues/990

[kayakyakr]

Just tried it out @aseering and noticed a pretty big speedup of a rails app load time. Nice!

I added an exclusion for both my networked drive and my lxss folder.

[tanseydavid]

Adding an exclusion does NOT fix the performance problem. Adding an exclusion for VS does NOT fix the performance problem. Adding an exclusion for IIS does NOT fix the performance problem.

The only solution that works for us is to disable REALTIME protection in Windows Defender.

I also want to note that the performance impact is DRAMATIC.

For instance, a rebuild of a VS2017 solution that takes ~20seconds to complete with REALTIME protection turned off, takes more than 3 minutes to complete with Windows Defender REALTIME protection enabled.

Another example: a GIT local check-in with one file changed, completes almost instantly with Windows Defender REALTIME protection DISABLED. This same checkin takes more than 2 minutes with Windows Defender REALTIME protextion enabled.

The exclusions DO NOTHING to improve the situation.

[rdodev]

@tanseydavid yes exclusions do not seem to have helped at all for the linux side. :(

[kumarharsh]

Yeah, it feels like adding exclusions is just a placebo - the performance is still incredibly slow for a lot of stuff related to development. But how do we even raise an issue with the windows defender team about this?

[sunilmut]

@tanseydavid - The issue that you have described, is that under WSL? Or, just in Windows general outside WSL?

@rdodev & @kumarharsh - In WSL, are you guys seeing any differences when defender realtime protection is turned off for the build directory? If you have any data to quantify, that will be useful.

I would like to get some clarification here before reaching out to the defender team internally.

[sunilmut]

Also, can you share your Windows build numbers, using the ver command from CMD?

[kumarharsh]

10.0.15063

I stopped using WSL due to performance issues, and don't usually run it now... but even with normal powershell, the performance of, for example yarn install is not anywhere near linux when defender is running. You can see some benchmarks in the yarn issue listed above. I realise I'm not adding much to the WSL part of the discussion, but just wanted to add some details for the defender team's benefit.

[rdodev]

@sunilmut this should be very easy for you folks to replicate and grab as many benchmarks as you want.

1. Install WSL
2. Install git/gnumake, etc.
3. Clone kubernetes/kubernetes (for example, or any large repo)
4. See your CPUs pegged at 100%
5. Profit

[fedu]

Aww, just ran into this, while I was trying the Bash for Windows 10 for the first time. It actually freezes the whole Webpack process and it will never finnish, no matter how long you wait.

Guess the Bash is still unusable for web developers.

[ow]

Blaaaah this is killing me as projects get bigger. Has nobody at Microsoft looked at all? I've seen a ton of closed threads but no action. I have to disable Defender all the time to get Git to finish in reasonable amounts of time (2s for a checkout with it disabled vs over a minute). I've tried the workaround - it does nothing.

[justinmchase]

Same here. I also encounter a lot of disk errors while doing large npm installs. I can see windows defender killing my CPU even though I have a total exclusion on my E drive. It seems like npm may be caching things on an un-excluded directory before unpacking which is triggering windows defender and causing disk issues.

[tara-raj]

The team is looking into this and will continue investigating. FWIW this rolls up into a broader filesystem perf improvement endeavor.

In the meantime if you can provide us with more specifics so we can generate a repro that would be very helpful. @ow which repo are you using? @justinmchase which errors do you hit?

[Sparkx120]

@tara-raj thanks for the update. Just to toss my own experience out there, I am running 16299 Enterprise and my Windows Defender is locked on due to company policy. As soon as I start WSL I see constant CPU usage from the Defender service hovering around 25%. This goes away the moment I shut down WSL. Inside I am just running tmux and bash, running ssh does not seem to affect it much. Any filesystem actions can take a while especially on /mnt/c. The high CPU usage reduces my computers overall performance and especially the battery life. Adding an exception on the Linux rootfs folder helped, it was 50%+ without it, but it does not solve the issue entirely.

Thanks for looking into this.

[justinmchase]

Here is a pretty simple repro for the bug I keep seeing:

1. In WSL install node v8.7.0
2. Unzip attached file to /mnt/c/bug , cd into dir
3. npm install

bug.zip

This could be a node/npm bug... I'm not sure yet but I use the same directory linking feature on osx and it doesn't have this problem. The problem appears to be a combination of being in a linked directory and having the same dependency in both modules. Also i am doing everything under /mnt/e, which I have an exclusion for and ends up being reasonably fast but if I were to copy these files to mv bug ~/bug it will trigger windows defender scans which massively slow it down. Also global npm installs or installing a module for the first time which I think is globally caching off of /mnt/e. If I knew where the unmounted files for wsl were located I'd add that to windows defender exclusion. Might be nice if that was done during wsl install by you guys too.

Error code:

22:04:52:justin:/mnt/e/code/bug$ npm install
npm WARN bug@1.0.0 No description
npm WARN bug@1.0.0 No repository field.

npm ERR! path /mnt/e/code/bug/test/node_modules/mongoose/node_modules/async
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename
npm ERR! enoent ENOENT: no such file or directory, rename '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/async' -> '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/.async.DELETE'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/justin/.npm/_logs/2017-12-20T04_04_57_053Z-debug.log

[robertquitt]

I'm having heavy performance issues related to Windows Defender and WSL as well. For me, it's when I'm switching tmux panes and opening files in vim. Disabling realtime protection in the Windows Defender Security Center causes a massive (>100x) speedup in performance for these tasks.

I'd really appreciate it if anyone had a workaround besides just disabling realtime protection.

Microsoft Windows [Version 10.0.16299.125] 
Linux goose 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux

[kumarharsh]

FWIW this rolls up into a broader filesystem perf improvement endeavor.

@tara-raj Would this endeavor be limited to WSL, or would also transcend to Windows' NTFS filesystem too?

[ow]

@tara-raj I can get the poor performance to occur on almost any repo.

Do a git clone on something like this - the performance will be slow in general, but can push it further by committing new changes or just installing with npm - in all seriousness I'm turning Defender off 6-8 times a day to get the performance bottleneck out of the way. I've not timed it, but I'd hazard a guess it's more than 4x as bad with Defender enabled.

There's a ton of documentation about this across repositories — Yarn has a bunch of ongoing threads about this and has been writing workarounds to handle it. Generally speaking, WSL works great, but I will say these performance issues have been crippling and frustrating.

[kayakyakr]

I'm working on migrating from the old version of bash to the windows store version of bash and cannot find the new location of the lxss folder. Anyone know where volfs is being stored these days?

[ow]

For those here having the same issue, I've just completely given up on Defender pinning my CPU all the time. Would prefer to have it on, but currently nigh unusable on my setup.

If you want to force Realtime protection off, you can use this registry key, since Microsoft insists that it will turn itself back on within a few hours on Fall Creator's and up.

Paste below into a file, add .reg extension, double click. Reboot and boom, hey, it's actually usable.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender]
"DisableAntiSpyware"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender\Real-Time Protection]
"DisableBehaviorMonitoring"=dword:00000001
"DisableOnAccessProtection"=dword:00000001
"DisableScanOnRealtimeEnable"=dword:00000001

[rdodev]

The sad part is that this issue was reported over 8 months ago and we've yet to see any progress or patch to remedy the issue. Instead of having to disable Windows Defender, they should revert to the way WSL worked before and let it be its own isolated subsystem. Integration with Windows is what caused all these perf problems.

[bitcrazed]

Please bear with us. We are investigating. If it was a simple issue to fix, it'd have been fixed by now 😉

[marcfor]

@ow Weird, I tried cloning crafty-vagrant, both within the WSL file system and in /mnt/c and neither seemed to take super long, nor did my CPU go above 15%. I only heard about WSL a week ago and installed it very recently. Have you been using it for long? Maybe you've got an older version installed? Or some cruft left over from an old version?

[ow]

@marcfor I have been using WSL since the day it launched, and have written much about my experience with it so far. Initial repositories appear fine, but once you begin to get complex directories in them -- run npm install, composer install, vagrant up on Crafty-vagrant -- then start trying to commit on a regular basis and it's agony. I can reproduce the behavior across three machines, one is just a few weeks old.

[bohrshaw]

I once ran git status in a big repository under /mnt/c from WSL, it's unresponsive for a minute. While the same operation using the Windows native git is completed within one second. I suppose this is also related to Windows Defender.

[2ps]

As an FYI. We use the WSL Switcher to use centos under WSL. Adding exclusions helped improve git performance by 50%.

$lxss = $env:LOCALAPPDATA
Add-MpPreference -ExclusionPath "${lxss}\lxss"

[2ps]

Adding: @bitcrazed , please feel free to reach out to me. Our team uses WSL extensively, and we're happy to get you use cases / situations to reproduce.

[naefl]

Cannot reproduce effects of disabling real-time protection or adding exclusion to project folder. Most noticeable is when running yarn start. On MacOs this takes roughly 2-10s, on Ubuntu 16.04 around 2s and on WSL I get between 40-60s which is unbearable. rebuilding takes 200-800ms on Ubuntu, 1-2s on MacOs and 9-20s on WSL. No effect on this by changing Defender/Firewall settings

[therealkenc]

Cannot reproduce effects of disabling real-time protection or adding exclusion to project folder.

Note there are a few unrelated patterns that show poor performance on WSL. Turning off Defender will probably (possibly?) help for the many itty-bitty-files case. It won't save you when doing git status on a ginormous repo on /mnt/c though, for example. It won't help at all for other cases, notably when thrashing large blocks of virtual memory. I am not saying that's what is happening with yarn. Maybe there are other patterns, who knows. It is difficult to get good data. Worse, #1577 gperf doesn't work. The most constructive thing to do, for now, is generate a strace -tt and start looking for red flags. Often where the program is spending the time is obvious. Understandably this is not always a very satisfying exercise, because you find out "it's that" and then the follow-up is "yeah it's that". But it does help.

[ChrisJefferson]

I can reproduce this very easily, doing a large cp with real-time protection turned on and off. Open and close calls in particular seem to take a long time, for example with protection then without protection:

10:15:49.752801 lstat("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000075>
10:15:49.752940 stat("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000063>
10:15:49.753057 open("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_RDONLY|O_NOFOLLOW) = 3 <0.003043>
10:15:49.756173 fstat(3, {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000042>
10:15:49.756358 open("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_WRONLY|O_TRUNC) = 4 <0.001099>
10:15:49.757559 fstat(4, {st_mode=S_IFREG|0777, st_size=0, ...}) = 0 <0.000071>
10:15:49.757727 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000059>
10:15:49.757878 read(3, "$ Grid indexing is 1-based\n\n$ 15"..., 131072) = 889 <0.000099>
10:15:49.758068 write(4, "$ Grid indexing is 1-based\n\n$ 15"..., 889) = 889 <0.000110>
10:15:49.758231 read(3, "", 131072)     = 0 <0.000071>
10:15:49.758356 close(4)                = 0 <0.000809>
10:15:49.759221 close(3)                = 0 <0.000045>

10:15:17.106272 lstat("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000061>
10:15:17.106379 stat("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000056>
10:15:17.106483 open("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_RDONLY|O_NOFOLLOW) = 3 <0.000093>
10:15:17.106655 fstat(3, {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000066>
10:15:17.106800 open("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_WRONLY|O_TRUNC) = 4 <0.000260>
10:15:17.107106 fstat(4, {st_mode=S_IFREG|0777, st_size=0, ...}) = 0 <0.000024>
10:15:17.107179 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000013>
10:15:17.107229 read(3, "$ Grid indexing is 1-based\n\n$ 15"..., 131072) = 889 <0.000034>
10:15:17.107301 write(4, "$ Grid indexing is 1-based\n\n$ 15"..., 889) = 889 <0.000041>
10:15:17.107381 read(3, "", 131072)     = 0 <0.000025>
10:15:17.107442 close(4)                = 0 <0.000077>
10:15:17.107554 close(3)                = 0 <0.000027>

[ow]

Note there are a few unrelated patterns that show poor performance on WSL. Turning off Defender will probably (possibly?) help for the many itty-bitty-files case

Erm, I don't know if I buy this. Git status, commit and anything NPM related improve dramatically with Windows Defender disabled. I had to resort to force disabling via Group Policy to make any of these tools usable.

[firewave]

Well, that the performance of the filesystem of WSL is slower than an actual system is a known issue since day 1 and well documented in benchmarks (e.g. Phoronix). The team even acknowledged that improving the performance is a high priority goal at the moment. The virus scanner performance impact also has always been there for me and if anything filesystem-related is happening it gets involved. McAfee is even worse than Defender. Disabling the Defender real-time protection helps but it's still held back by basic performance limitations. I thought somebody once mentioned that it is a known issue caused by the way WSL is stored and it it would require reworking of it to get this fixed. So far it could live with it and had to move some more complex things I did on WSL to an real VM eventually because of other reasons so I hit me less. But since CLion is now having support for WSL and having no control over the virus scanner at all in an corporate environment makes this issue more severe for me now.

[ChrisJefferson]

Just as a benchmark, I find turning off Windows Defender speeds up cp of a large directory by about a factor of 8.

[galvesribeiro]

Just to reinforce this (old) issue that isn't solved yet, I just did a sudo apt upgrade. Look at this picture and see who is always fighting the Dpkg process for resources:

If I'm compiling something, it is even worse... I think the exclusion of the source directory is being totally ignored.

I understand that disable Windows Defender on registry is not the proper fix as we would have other security issues later on, but have this issue that long open is frustrating. We see more and more fixes/enhancements getting added to WSL but this, which is critical to any use case, is not sorted yet. :(

[fanoush]

just noticed this too, windows 10 pro version 1709, it looks like even reading simple file repeatedly triggers this. This one just flies with zero cpu taken by antimalware service

while :; do echo -n Hello ; done

And this one takes >10% cpu by antimalware service and the output is slow as typewriter

echo -n "Hello" >file
while :; do cat file ; done

And btw I have excluded %LOCALAPPDATA%\lxss folder but don't see any difference

[Archenoth]

Doing an apt-get upgrade can also cause some totally gnarly disk access

[therealkenc]

Doing an apt-get upgrade can also cause some totally gnarly disk access

To the point people think it is stuck.

[gmcclins]

A few people in here have said that adding an exclusion in Windows Defender does nothing but I wanted to go ahead and give it a try anyway. A quick google search for a directory that I should exclude was C:\Users\{user}\AppData\Local\lxss, but pretty sure that was legacy.

Most instructions I could find, granted was a quick search all pointed to excluding a directory, but I noticed that Windows Defender Security Center also allows excluding process. So I excluded

Apt
Bash
Dash
Dpkg
Init
Mandb
Sudo
Ubuntu

And noticed an immediate improvement. Although it is not blazing fast, it has sped up a little bit during this huge install it was already working on. Also for the first time during this install, I am not seeing my disk constantly at 100%. I'm at 58% right now, yes - I have seen it go back up into the high 90s but it goes back down. This is much preferred to constant use.

As for the speed increase. I cant say for sure. I was mid install but it definitely seems like its going faster. Even if not, I will take the disk saving. One thing I also noticed is there is a subprocess of Ubuntu, ubuntu - but Windows Defender will not allow adding Ubuntu and ubuntu. Not sure if that's an issue.

[WSLUser]

@gmcclins Is this on Skip-Ahead Insiders? They're supposed to allow per process control now. If you are on Skip Ahead (and the reason you were able to able to add those processes as I suspect it is), I would create an issue for not allowing whatever sub-processes can't be added.

[gmcclins]

@DarthSpock I am on Windows 10 Pro Version 1709 (OS Build 16299.309). I am in Release Preview. Also I edited the post now, but it seems my grammar check made things unclear. What I was trying to say was there is a process Ubuntu and then also ubuntu. I'm not sure how Defender handles a difference in capitalization and it won't allow adding both.

I reset Ubuntu back to default today and am currently updating. I'm about 15 minutes in and at 37% so it is still pretty slow and no surprise Antimalware is chugging away

There is still the improvement in disk use, it gets high at times, but it's not a constant 100% anymore.

There are probably other processes that should be excluded but what I have so far I got from performing an update and install within Ubuntu and watching Windows Task Manager. As far as the blocking itself, Windows Defender Security Center, Virus & threat protection settings, Add or remove exclusions.

[WSLUser]

Ah ok, that makes a difference. Yeah, you have to exclude the entire rootfs for all pico processes to run without errors or latency due to Defender (or any other A/V/FW vendor for that matter). It isn't a satisfactory solution but a workable one until the next release of Windows or upon your A/V/FW vendor supporting pico processes.

As far as case sensitivity goes, this is fixed in the Insiders and will be available in 1803.

[glennmartinez]

Can confirm that I just installed Ubuntu yesterday and after installing all the goodies whenever I launched Ubuntu bash it would slow down to like 20 seconds before I get the terminal prompt.

After adding the exclusion it only takes 3 seconds now. This is only on startup, not sure that i'll experience further lag doing other work but i'll watch out for it.

[vheathen]

After the series of updates (17120 -> 17128) I'm also facing both #3034 and this (Windows Defender high CPU load) issues. Even after full stack of exclusions I can't run dev tools (tests for example, check attached pics please).

[asolopovas]

Same here, I don't understand why microsoft defender won't wsl by default?

[WSLUser]

upgrade to 17133 for the other issue. As far as Defender goes, you'll have to be on Skip-Ahead to see the change in Defender's behavior. SCU won't have it unfortunately.

[ow]

Be wary of skip ahead, last I chefked has a breaking change where Hyper V is enabled by default on all Windows 10 versions. You won't be able to boot VMs with anything else.

I just disable real time protection all the time now and performance is amazing - but this really isn't a solution. It's a real shame.

[Thell]

After going 13+ hours for a large install I refused to make any major changes for the last couple of months, but then got a dreaded Error while moving old database out of the way. AppStream cache update failed and decided to clean which also failed, so a fresh restart was in order and I figured I'd add a little timing info for those deciding if they want real-time on/off...

Fresh ubuntu install; first upgrade on a 5400rpm platter drive without Realtime Protection and with.

Personally, I'll be disabling when doing large installs (like Tex Live) and living with the pain until some form of relief comes along...

PS - Windows 10.0.16299

[bluddy]

At this point I have the windows defender config panel open in the background, and I just repeatedly go to it and turn real time protection off while I'm working. It's a real pain, and it needs to be fixed. Not to mention that it leaves my PC vulnerable.

[bitcrazed]

@bluddy We hear you and share your pain. @tara-raj & team are looking into this issue, but any fix in this area will require great care, so will take a little while. But we ARE keen to improve this scenario.

Thanks for your patience.