Major performance (I/O?) issue in /mnt/* and in ~ (home)

[Mika56]

A brief description

As a Symfony developer, it's always been hard to get a stable/fast development environment. My current setup is a Ubuntu running under VirtualBox (using vagrant). While page generation is fast, my IDE accesses my PHP files through SMB, which is really (sometimes horribly) slow. I'm now trying to use WSL to improve all of this. However, I'm having a major performance issue when using /mnt/* folders. If I set up a Symfony project under /mnt/c, it is really slow. If I set it up under /home/mikael, it is very fast.

Expected results

Drives mounted under /mnt should be as fast a any other folder.

Actual results

With a new Symfony 3.1.3 project, under /home/mikael takes between 100ms and 130ms to generate the home page. The same project under /mnt/c/ takes between 1200ms and 1500ms.

Your Windows build number

10.0.14393.51

Steps / commands required to reproduce the error

# Install PHP5
$ sudo apt-get install -y php5 php5-json

# Download Symfony installer
$ sudo curl -LsS https://symfony.com/installer -o /usr/local/bin/symfony
$ sudo chmod a+x /usr/local/bin/symfony

# Download Symfony
cd
symfony new symfony_test

# Start Symfony
cd symfony_test
php bin/console server:run

Open your browser and go to http://127.0.0.1:8000/. Once the page is loaded, refresh it (on first request, Symfony had to generate its cache). Generation time is displayed on the bottom left

You can then do the same under /mnt/c/

cd /mnt/c/
symfony new symfony_test
cd symfony_test
php bin/console server:run

Additional information

I've added my dev folders as excluded folders in Windows Defender, as well as %LOCALAPPDATA%\lxss. I've tried having my project in ~ and pointing my IDE to %LOCALAPPDATA%\lxss\home\mikael\ but as I've later read, there is no supported way of editing WSL files. WSL is installed in its default location under C (no strange junction or symlink), which is a healthy SSD. My computer is attached to a domain, if this might have any influence.

[ztolley]

Any update on this?

[fpqc]

@ztolley https://github.com/Microsoft/BashOnWindows/issues/873#issuecomment-313472537

[aurelien-reeves]

Hi,

May we expect major improvements regarding that issue with the incoming Fall Creators Update?

[r-k-b]

@aurelien-reeves yes; subjectively, it is much faster. I've been using FCU for a week or so now.

[nickjj]

@aurelien-reeves I am running the fall update and was skeptical of WSL for a long time but I'm actually using it now.

There's still a ton of room for improvement for performance if you want to gauge it against a VM driven Linux box without using mounts or shared folders but for day to day development it's usable.

That's not just "usable but hating life" either. I've ran into a couple of edge cases but overall I can cope with 5s reloads instead of 2.5s reloads for every day web development.

[jbgomond]

@aurelien-reeves much faster for me too. My Symfony App loads in 200ms now :)

[aurelien-reeves]

Indeed I've updated Windows, and yes, this is far better!

Great, I'll finally be able to work without VM :)

[vnenkpet]

I'm not really seeing any major improvements. Still feels multitudes slower than native Linux. Running npm install is a nightmare.

[bmayen]

Likewise, I don't see any major performance improvements in I/O specifically.

[bitcrazed]

You should see a few moderate IO perf improvement in Fall Creators Update, and we're working closely with the filesystem team to make more substantial improvements will in future Windows releases.

[evelant]

WSL is so amazing except for the IO performance problem. Like others have said developing with npm is a nightmare. If WSL was as fast (or nearly as fast) as native linux I would probably use windows on all of my machines, but I am forced to dual boot now until the performance improves.

[nickjj]

@AndrewMorsillo You can get around dual booting by using something like vmware with Unity mode. It's what I did for about 5 years until I switched to WSL last month. I have no problem running npm driven apps, but in my case the Node app is ran through Docker, but I'm using WSL to interact with Docker which happens to be ran through Hyper-V using Docker for Windows.

[bitcrazed]

Bear with us - we're working closely with the Windows file storage team on a number of improvements that'll result in noticeable perf improvements, plus improved interop etc. in future OS releases.

FWIW, I am, LITERALLY as I type this building a web project using Gulp & node, Jekyll/Ruby, image & CSS minification, etc. While the build is not as zippy as on a real Linux VM/box, I've found it to be quite acceptable for now. YMMV

[fpqc]

@bitcrazed I'm dying to know how you guys are approaching this. I have all kinds of hypotheses, but I'm just curious as hell about which route y'all are taking. As it gets closer, do you think we could get a new blog post explaining the change?

[vnenkpet]

Well I am glad you're really working on this. This is the one killer feature for most node/react developers and probably the last major blocker for converting many to Windows. Right now I still have to dual boot if not use Mac instead...

[Mika56]

Anyone been able to create files from Windows? Release notes for 16237 stated "Use default attributes for files without EAs in lxfs (root, root, 0000)". I'm running FCU (16299) and files created in C:\Users\Mikael\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\rootfs\ are still not visible on the Ubuntu side

Edit: never mind, looks like you need to restart the Ubuntu part for files to be visible, so this can't be used either :(

[rococode]

As I understand it, this issue is directly related to #981? I wanted to mention for others struggling with slow git commands - you can call cmd.exe directly from WSL and it runs much faster! It's a simple solution but I hadn't seen it anywhere else and I just thought of it now and figured I'd share.

I personally was having the issue where stuff like git add or git status would sometimes take 10+ seconds.

I added these aliases to my .zshrc (I think bash has aliases too, right?)

alias gs="cmd.exe /C git status -s -u"

alias gaa="cmd.exe /C git add --all"

alias -g gc="cmd.exe /C git commit -m "

These commands are working great for me!

All of that runs very quickly (<1s per op), whereas the same sequence of commands with calls to the built-in git takes easily >10 seconds.

I'm sure similar approaches can be adopted for other programs with poor performance as well.

[Mika56]

Hi @rococode, While this works for tools like Git, it just cannot work for web servers like Apache. If I wanted it to be running on Windows, I simply wouldn't use WSL

[bitcrazed]

@Mika56 - you have read this, right? The guidance still stands:

DO NOT, under ANY circumstances, create and/or modify Linux files using Windows apps, tools, scripts, consoles, etc.

[bitcrazed]

@fpqc - re. documenting how we fix this - @tara-raj is driving the authoring of several blog posts, including one to document this issue & solution in some detail. Stay tuned ;)

[Mika56]

@bitcrazed Of course I have! But part of my job is to read and ignore warnings 😉 In all seriousness, I'm still not doing anything serious with WSL because of this performance issue, so I couldn't care less if I break it. It thought it was worth a try, seeing that files without extended attributes should be seen as root/root/0000

[SvenGroot]

@Mika56 As you discovered, LxFs does a fair bit of caching which causes it to run into this issue. Adding the ability to recognize files without EAs was done as a mitigation that gives you an easier path to recover a file if it accidentally got stripped of its EAs. Previously, that file would be entirely inaccessible in WSL, so recovering it was difficult. Now, it can be recovered with chown/chmod, which is much easier.

This change wasn't meant to allow external creation of files. It does work that way, but only if WSL isn't running when you do it.

We are working on improving the interop features of both DrvFs and LxFs, as well as to improve the performance of both file systems.

[therealkenc]

I have been meaning to thank you for getting that one Sven. So thanks 💯.

[oblitum]

Recently I had to remove my zsh dotfiles and clone it again clean, they are based on Prezto. I simply don't know why, and I'd be interested in any tip on how to debug this, but since then loading up my zsh/prezto/tmux setup is utterly slow, and any directory access or tmux pane creation takes really a lot of time. I don't understand how this is happening because it was fine before, I simply redownloaded the dotfiles and now it's just unusable. They work fine on real Linux and macOS.

[oblitum]

Actually, I simply don't know what happened, even on bare Bash (not Prezto based Zsh setup), git operations and file access are too slow.

[oblitum]

OK, by using SysInternals Procmon I identified that "Windows Defender" is responsible for 95% of the issue, making file access like 100x slower, and disabling real-time protection fixes most of it. Sadly, it solely offers to disable this protection temporarily and adding "C:\Users\francisco\AppData\Local\lxss" as an exclusion doesn't help, I had to disable real-time protection and a bunch of other scanning features through Group Policy Settings. I really hate Windows Defender, I wish I could uninstall it completely just leaving the firewall, but as far as I know it's impossible, I've tried to disable it before on another machine but noticed that it just muted the program, the services were still present and I stopped trying to turn them off, IIRC I had to start in safe-mode or something worse to be able to change the registry key for making it a manual start service.

As an IT professional I know well that 99,9999999% of the time Windows "Defender" will only serve to make my machine slow, and bypassing these kind of protections is easy, it only serves to protect, maybe, the layperson that don't know how to take care of himself on the web and is target to the most common threats that Windows "Defender" is able to catch. In sum, it's useless. At least I'm happy that Windows is not my platform, it just infuriates me when I have to touch it and have to deal with all this nonsense.

[lightrow]

a single group policy setting ( turn off ) should be enough to tame windows defender..

it is however not quite enough for the IO problem... Still subpar compared to docker

[oblitum]

@lightrow Right but, I preferred to just turn auto-scan features off, instead of globally turning Windows Defender off, so to let firewall or whatever other feature on. For me this is the kind of thing that can still be left on always, because it doesn't reduce computer speed abruptly. Agreed that even after that it's not bare metal speed, but for me it has improved orders of magnitude. When Windows Defender's auto-scan was on, it was almost unusable.

[lightrow]

this is not really related to the topic, but i have windows defender off and im still getting the firewall "pop-ups", i believe these things are kinda separated, considering firewall in its current form has been part of windows since at least windows XP, while windows defender ( aka microsoft security essentials ) is a new addition, unrelated to it.

in any case, turning it off does speed things up considerably, not just WSL but every single operation, involving managing files.

[oblitum]

@lightrow right, but I still preferred to leave protections related to downloads and online access on.

[oblitum]

@lightrow another reason I preferred it that way instead of globally turning it off through group policy is that just disabling the real-time scanning will leave Windows Defender as if it was still active (except for real-time scanning), which leaves its icon in the taskbar untouched and you can click on it and see its security panes (with one showing real-time scanning is off, etc). When you disable it globally it gets into a weird state, the Windows Defender is still running in your system, but you can't access it, the icon is still there in the taskbar, but it gets an "X" on top of it, and clicking it doesn't bring any application, so it's just weird to have an application running in that state. As I said earlier, if I could actually uninstall it, I'd do that, because then I'd not be left with this zombie app. As I don't like a zombie application state, I just disable the run-time thing.

[lightrow]

eh, no, the defender process ( MsMpEngine.exe ) is not actually running when it's disabled. The tray icon is a separate process, which is just that - a harmless tray icon, acting as a shortcut to access windows defender panel. It simply autostarts with windows and can be disabled in task manager's "Startup" tab. And the defender's settings panel itself is hardcoded into windows control panel. So it's not a "zombie", but rather a "Sleeping Beauty"...

[oblitum]

@lightrow no, it's just a zombie with many active parts left scattered: services you can't turn off, tray icon, control panel, very frequent updates, etc, no beauty in that :)

And besides (can't confirm), but I've heard that after some upgrades, if you remove, it just gets back, tray icon process, etc. When I say it's left ON even if you disable it globally, I mean that, it's still updating and restoring settings, services are still auto starting, ...

Manual scan is still convenient to have too.

[ziruizhuang]

It seems that the IO performance is even worse after the Meltdown Vulnerability Patch.

[bitcrazed]

@ziruizhuang Performance on all platforms will be affected by mitigations to Meltdown, though for most users, the impact will be largely negligible. Of course, YMMV.

https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)#Mitigation

[WSLUser]

exposing the hardware to WSL via API would alleviate the further reduction in performance (as well as the original performance issue). Is there any news on that front? Perhaps using Windows AF_Unix - WSL interop could be another method as well (Could simply be a matter of passing the driver files from Windows to WSL with some magic in between).

[raupie]

Is this going to be addressed in the next release (1803)? At a crossroad and need to know.

[aseering]

@raupie what, precisely, are you referring to as "this"? The OP? @DarthSpock's comment? Some other comment?, there's a lot of discussion on this thread...

I don't work at Microsoft, but my current understanding is that I/O performance has been improving and will continue to improve, but it still won't match native Linux performance (native kernel as well as filesystems whose on-disk structures are performance-tuned for the types of permissions and metadata operations that Linux apps typically perform) in all cases.

[raupie]

Sorry, I should have been more specific. I was speaking about the OP. I'm curious to know if anyone has tried the insider builds with Symfony (or any large framework) to see if the I/O performance is any better. I'm about to embark on a year long project and I have to get LAMP development environments working immediately. WSL + Docker for Windows has been great for WordPress and the workflow is exactly what I want for Symfony. It provides immutable architecture and native Linux tools all on Windows machine. So WSL is really exciting and almost the holy grail of development environments for Windows users. The options I have is to reformat and install Linux, set up virtual machines on Windows or hopefully WSL + Symfony becomes usable in 1803 (March?).

Thank you for your feedback, I was afraid that would be the case. This is no small feat so props to the WSL team for everything so far!

[rfunduk]

The current state of this issue is that the WSL team is working on it. Right now "doesn't quite match native Linux performance" would be a great place to be, I would be happy with a 15-30% perf hit, but it's many many times slower than that. We've been promised they are working on a fix. See comments from @bitcrazed in this thread:

https://github.com/Microsoft/WSL/issues/873#issuecomment-345107885 https://github.com/Microsoft/WSL/issues/873#issuecomment-347297106

[WSLUser]

@aseering I'm suggesting that we take advantage of AF_Unix interop to pass the driver files that you can browse to in File Explorer into WSL. The issue however is they are Windows drivers, not Linux, hence the "magic" where several things could occur to make it work with WSL. One method could be commenting out the windows code and uncommenting the linux (if they're both in the files and getting it put into the files by the vendors if not. I created a trouble ticket with AMD to see if something like this is feasible as I use both their CPU and GPU (Threadripper is awesome!))

This would boost performance in WSL because it would match Windows exactly (probably slightly better, Linux always was better at harnessing the full potential of hardware but of course that depends on a kernel we don't have but we could always implement the features of the linux kernel to emulate what each kernel version is capable of)

[Mika56]

I've tried a few things since. Samba is a slow in WSL as a VM. I could not find any viable NFS server for Windows, so neither WSL nor VM can benefit from this. Docker for Windows using MobyLinuxVM uses some sort of SMB, so here we are again Last couple Docker for Windows brought "native" Linux containers, but yet again it is slow. Meltdown fix is probably gonna slow everything down too...

I'm now considering developing on Windows, PHP behaves very similarly on Windows and Linux, so dev on Windows, deploy on Linux 🤷‍♀️

[aseering]

@Mika56 I'm curious, how slow is Samba in WSL as compared to DrvFS in WSL? (Do you know offhand?)

[aseering]

@DarthSpock are you seeing slow file data operations, or slow file metadata operations?

For example, many build tools use lstat() to get file permissions, file modification times, etc. In Linux, that data is all stored in one place on disk, and it's usually cached in kernel memory. In Windows, even a AF_UNIX-based solution will have to pull that data in from a bunch of different places in NTFS every time your program requests it.

Native Windows programs usually use batch or cached operations to work around this. For example, Windows git has a metadata cache. If you disable the cache in your git config, Windows git gets dramatically slower. Linux git just hammers lstat() all the time, because it's a fast operation in real Linux.

[WSLUser]

Thanks for the quick lesson. Hmmm, I'm not exactly sure which is worse, but only really notice when doing a particularly large projects. Usually I try to break it into smaller executable pieces and in cases where that isn't possible, I end up using ssh into a CentOS VM. I've just noticed that WSL doesn't take full advantage of the hardware we have and some applications do on Windows (even Windows isn't tapping the hardware for all it's got though does a better job at it than WSL at this point in time. Side note: hope that new power feature released comes to more than just Pro for Workstation and Enterprise so I can use at home too).

[lightrow]

@Mika56 , for Docker you could try using docker-sync with unison. A bit of a pain to setup though

[naefl]

Is it possible to redirect e.g. npm run to start in a Linux native chrome window running on XServer? Since I read the I/O problems are particularly strong when being on the windows side of things, i.e. /mnt/c/, I thought this might be a solution since it wouldn't have to communciate with any windows program if you can run it on a native browser

[firewave]

A current benchmark by Phoronix: https://www.phoronix.com/scan.php?page=article&item=wsl-february-2018&num=2

Although the article mentions it got slower it's actually on par with an benchmark from 2016 which is already linked in here.

[WSLUser]

@naefl Try setting the default browser to the Linux chrome (or firefox) export BROWSER= "path to chrome." If you have a pointer to BROWSER, it should try executing to your intended browser and you can change the variable to Windows or Linux browsers to test it for each. Post here how it goes. EIther way it goes, that will allow others to know the result of trying to run through native Linux browsers.

[gnodiah]

what's going on?