Major performance (I/O?) issue in /mnt/* and in ~ (home)

[Mika56]

A brief description

As a Symfony developer, it's always been hard to get a stable/fast development environment. My current setup is a Ubuntu running under VirtualBox (using vagrant). While page generation is fast, my IDE accesses my PHP files through SMB, which is really (sometimes horribly) slow. I'm now trying to use WSL to improve all of this. However, I'm having a major performance issue when using /mnt/* folders. If I set up a Symfony project under /mnt/c, it is really slow. If I set it up under /home/mikael, it is very fast.

Expected results

Drives mounted under /mnt should be as fast a any other folder.

Actual results

With a new Symfony 3.1.3 project, under /home/mikael takes between 100ms and 130ms to generate the home page. The same project under /mnt/c/ takes between 1200ms and 1500ms.

Your Windows build number

10.0.14393.51

Steps / commands required to reproduce the error

# Install PHP5
$ sudo apt-get install -y php5 php5-json

# Download Symfony installer
$ sudo curl -LsS https://symfony.com/installer -o /usr/local/bin/symfony
$ sudo chmod a+x /usr/local/bin/symfony

# Download Symfony
cd
symfony new symfony_test

# Start Symfony
cd symfony_test
php bin/console server:run

Open your browser and go to http://127.0.0.1:8000/. Once the page is loaded, refresh it (on first request, Symfony had to generate its cache). Generation time is displayed on the bottom left

You can then do the same under /mnt/c/

cd /mnt/c/
symfony new symfony_test
cd symfony_test
php bin/console server:run

Additional information

I've added my dev folders as excluded folders in Windows Defender, as well as %LOCALAPPDATA%\lxss. I've tried having my project in ~ and pointing my IDE to %LOCALAPPDATA%\lxss\home\mikael\ but as I've later read, there is no supported way of editing WSL files. WSL is installed in its default location under C (no strange junction or symlink), which is a healthy SSD. My computer is attached to a domain, if this might have any influence.

[fcicq]

with the current design of DrvFs, I guess this problem is hard to resolve. I recommend you to run a ssh server inside WSL and do the edit via ssh to bypass some other issues.

from https://blogs.msdn.microsoft.com/wsl/2016/06/15/wsl-file-system-support/ "DrvFs also disables directory entry caching to ensure it always presents the correct, up-to-date information even if a Windows process has modified the contents of a directory."

[fpqc]

Also @Mika56 the devs have said they are working on improving performance by redesigning parts of how file I/O works. That's probably why they haven't documented exactly how it works yet, since it is very much in flux at the moment.

[ajaykagrawal]

there is no supported way of editing WSL files

I wonder how difficult it would be for popular IDE and text editors to become WSL aware and preserve the attribute of WSL files (the security model may have some inconsistencies but I might be ok with it)

[fpqc]

@ajaykagrawal Some text-editors have an ability to save in-place rather than overwrite. I heard someone say that they enabled that setting in one or another text editor and it worked.

[ajaykagrawal]

Thanks @fpqc. I don't quite know what save in-place means but I hope more editors are updated to have this ability

[Mika56]

@fcicq I'm using an IDE for a reason :) I need my IDE, and mounting a folder through SSH, FTP, SMB or whatever would be no improvement on my current environment

[fpqc]

@ajaykagrawal Basically instead of writing a new file and saving over it, the text editor has a delta and applies it to the file, at least that's my guess.

[aseering]

In the old days, whenever you hit "save", text editors saved into your existing file. This was simple and efficient, but if your editor crashes or was killed mid-save, or if you run out of disk space, or if anything eelse weird happens, you might end up with a corrupt garbled mess that's half the old file and half the new file. Or it might just be completely empty / wipe out all of your data entirely, if your app implemented "save" with an initial truncate, which is easy to do by default on Linux at least.

These days, text editors almost always "save" by creating a temporary file in the same directory, writing the entire new file contents to the temporary file, and then moving the temporary file on top of the old file, overwriting the old file in the process. "Move" within a directory is guaranteed in most cases to be strictly atomic -- even if your computer loses power mid-Move, you'll always be left with either the new file or the old file; not a corrupted intermediate.

Clever, right? Saves lots of "I lost my data!" headaches. Thing is, if an application creates an arbitrary new file, there's no way for the OS to know that it's supposed to have the same magic permissions as some other existing file. That's what's breaking WSL.

Some editors let you go back to the old-style behavior. Has more risk, but avoids issues in some situations.

[fpqc]

@aseering it looks like Windows move preserves those attribs, or at least robocopy does?

[aseering]

Read more carefully: It does an excellent job of preserving the attributes of the wrong file :-)

[EDIT] I can't speak for robocopy, just regular move, but what you want in this scenario is the opposite of what you want in most scenarios.

[fpqc]

@aseering yep I am aware. I just thought that Windows commandline tools preserve even hidden ntfs attributes (moving is different from editing a file, since the atomic operation is done at the filesystem level, I think) like robocopy does.

[aseering]

@fpqc -- ah, I see what you're getting at. Yeah, applications can copy extended attributes, if they're programmed to do so correctly.

[Adraesh]

Exactly the same "issue" detailed above, as I am as well a Symfony dev.

My SF projects are all located into /mnt/d/.... my apache2 server (running on ubuntu bash) root folder is linked using a symlink and the perfs are aweful ...

This also lead to some issue with Symfony cache system btw.

[baroso]

I've not noticed any performance issues with my IDE IntelliJ/PHPStorm, but the execution of PHP/Symfony code (running LAMP/Symfony with SuluCMS installed) ist between 4-10 times slower (total execution time) than on a normal Ubuntu system ...

WSL 14915 second run

Ubuntu 14.04 second run

Why is it so slow? Are there any workarounds, maybe moving the vendors folder of Symfony?

If you can fix this issue, it will be the best environment for developing true DOT.NET and true PHP on Windows!!! Thanks, keep up the great work!!!

[baroso]

Is this issue beeing worked on?

[fpqc]

@baroso Yep. It's one of their top priorities after getting all the web-programming stuff working.

[nickjj]

@fpqc Is there another issue to watch to see progress?

Slow mount performance is the only thing holding me back from upgrading to Windows 10 and using WSL.

[fpqc]

@nickjj All I know is that the devs have said that it is a top priority, but they haven't released details on how they are going to pull it off. It did sound like they have an idea of how to do it though.

[pachkovsky]

Any updates on this?

[fpqc]

nope. Mum's the word on this'n. Major performance improvements I think are targeted for RS3 not RS2.

[nickjj]

What does RS3 mean in the grand scheme of things? Will it still make it into the next official big update?

[fpqc]

RS3 insiders builds will be rolling out probably in May. Release in December?

[nickjj]

Thanks. Guess I'll come back next year. Hopefully things are ironed out.

Using the insider's build is out of the question for me due to the severe breach of privacy that it entails.

[bitcrazed]

@ajaykagrawal As a general rule of thumb, whenever the question "I wonder how difficult it would be" is asked, assume that the answer is "very, very difficult", and/or "takes an enormous amount of time with no guarantee of success" ;)

In this case, imagine trying to get the owner of every application that opens and saves text files to modify their apps to support a different way of opening the files without read locks, and writing changes to those files without destroying extended properties managed by an external process.

If the underlying issue was simple to solve, we'd likely have already solved it. However, yes, we are very aware of the issues and we do aim to work on improving Windows <--> Linux filesystem interop in the future.

[aseering]

I realize this is a hard problem, but, just reporting that I'm still (build 15042) finding DrvFs to be much slower than Linux for my common use cases, to a degree that impacts my productivity somewhat.

A good specific representative-example workload that I would like to see optimized is the Boost build process:

http://www.boost.org/doc/libs/1_61_0/more/getting_started/unix-variants.html#easy-build-and-install

The build steps should be heavily CPU-bound in the compiler; that's fine. But I would love to see the final ./b2 install command run faster. In my experience, it's many times slower on Windows than on Linux.

Note that Boost's build system is cross-platform. And the final install command is just as slow under regular Windows :-) It would be wonderful if the Windows version got faster too.

[fpqc]

@aseering fwiw, the major performance improvements were not planned for RS2. Even 2 months ago the devs were saying it was planned for RS3.

[aseering]

@fpqc -- yep, I'm just keeping the ticket alive :-)

[matthewrk]

Is there any workaround to this for the time being? I'm considering switching to Windows for my main OS (currently macOS) but this presents a roadblock for me, I'm seeing around 1s added to RoR response times which makes development using WSL a bit too cumbersome.

[xob]

@matthewrk Not really a viable workaround for everyone, but what I'm doing in the mean time is running unison to sync files from /mnt/c/pick/a/folder to ~/pick/a/folder, and then I'm running ruby (in my case) from ~/pick/a/folder instead of straight from /mnt/.

It's still slower than running a linux VM, but it's a lot better and good enough for my development needs.

[matthewrk]

@xob thanks for the suggestion, that works pretty well! how did you get around the fact that everything in /mnt/c/ is owned by root? once its synced over to my ~/destination folder its still root:root so my app can't write its logs etc.

[xob]

@matthewrk I have never had problems with root, but I believe that the reason I have never had problems with this is that I use unison with the -perms 0 parameter. If I recall correctly, that parameter means that permissions are not copied when files are synced, and gives the permissions to the user running unison instead.

If that doesn't do it, here's the full unison command that I run: unison -perms 0 -times -auto -batch ~/destination /mnt/c/source -repeat 5

Hope it helps!

[matthewrk]

@xob Thanks! it turns out that although the files are root:root they're also 0777, but my sync was only persisting the owner, not the permissions, so a little fix and it should be all working, I appreciate your help :)

[nickjj]

@xob Do you have any benchmarks on comparing unison to a Linux VM? What type of apps are you developing?

I'm using a Linux VM currently and it's all great, but since inotify works with WSL now I'm willing to try it again once the performance issues are cleared up (with unison for now).

[xob]

@nickjj I work on a SAAS application developed in ruby on rails (ruby 2.2.5, rails 4.1.16).

As for benchmarks, I can easily compare the app running in development environment (no compressed assets or anything), on both Bash on Windows and in a Docker container (which is native Linux, granted it's Debian, not Ubuntu, but that shouldn't make much of a difference).

The benchmark I am doing is a measure of the time it takes to do a full-page refresh (ctrl+F5) on one page of our app, once logged in. The measure is taken 5 times, and here are the averages: Bash on Windows, in /mnt/c: 32.322 seconds Bash on Windows, in /home/user (unison): 16.552 seconds Docker container (Debian): 5.208 seconds

According to the benchmark, at least in my case, using unison to run my server from /home/user makes my app twice as fast, compared to running it from /mnt/c. However, compared to native linux (in my case Docker, but a VM would have similar results) is 3 times as fast as running from /mnt/user.

I hope that answers the question!

[nickjj]

Thanks @xob, that clears it up but what hardware specs are you using to get those figures?

I'm using Docker too ("natively" on Linux (xubuntu GUI) with unity mode on VMware) and I'm very happy with the dev speed on my Rails apps. It's basically the same speed as a real native Linux box without VMWare.

Honestly, I don't think I could deal with a 15s wait on each page refresh. Now, of course that's tied into your app's size and computer specs but a 3x difference is pretty big. You have more patience than me!

I think I'll continue holding off on WSL and just see what MS comes up with later this year. I only want WSL because unity mode is no longer officially supported by VMWare for Linux guests, so I'm using an aged distro of xubuntu (14 LTS) since 16 does not work.

[xob]

@nickjj The benchmarks were all run from the same machine. All of my app dependencies (database, etc) run in Docker containers, in all cases. The only difference between the tests is the rails server.

The hardware specs of my machine: Intel Core i7 4770 CPU @ 3.40GHz 16GB RAM Kingston SSDNow V300 SSD (240GB)

Not sure that the rest of the specs (which I am too lazy to look up) are relevant.

[nickjj]

Thanks. Your specs are pretty similar to mine (i5 4460 @ 3.2GHz / 16GB RAM / SSD).

When you say a Docker container, are you using Docker for Windows (the Hyper-V solution) or are you dual booting into a native Debian box?

[xob]

Docker for Windows with Hyper-V, yeah.

[nickjj]

Cool, but I guess one limitation of that is your code editor lives on Windows, which means you lose out on things like language specific features, since you surely wouldn't use Ruby on Windows. If you don't mind continuing this discussion, I'd love to ask you 1 or 2 more questions (without spamming this issue): feel free to shoot me an e-mail if you want at nick.janetakis@gmail.com.

One relevant thing that did come from this discussion is that Docker managed to get reasonable mount performance from Linux OS <-> Windows. I wonder how much of that can be extracted from Docker and placed into WSL.

[DJWassink]

Did anything happen performance-wise since february? This performance issue on /mnt/* is really a showstopper for me, everything works nice but it so slow that I went back to a dual boot again.

Is it any beter nowadays or is there still a performance hit of 5~10x?

[fpqc]

No release notes up; not sure if we've even gotten a WSL update since late May.

[bmayen]

A brief, periodic update on any internal progress being made on this (or even lack thereof) would be great. This issue is the elephant in the room for many people preaching the "Windows can now support devs better than OSX in many cases" narrative to their OSX compatriots. After initially showing off this tech to any coworker who would listen, I mostly keep quiet about it after repeated responses of, "it's undeniably cool, but you don't actually use it when it's that slow, do you?" It's just a complete dealbreaker for most people, and first impressions are the greatest.

Given how big of a blocker this is for a large demographic of users, any periodic updates we could get on its progress would be hugely appreciated!

[fpqc]

@bmayen this one is top of their priorities. It looks like whatever solution they came up with requires the involvement of teams outside the WSL team itself, which means it is not just a matter of the team doing what they want but also them waiting on another team that may not prioritize this as highly.

Also, changes to the core primary harddrive filesystem drivers probably have to go through more stringent code review even for insiders builds for obvious reasons.

But yeah, it would be nice if the team would give us the gist of how they're attacking this problem, but it seems like company policy at Redmond is to avoid giving this kind of info until they have it all finalized

[sunilmut]

Adding @SvenGroot to see if he can provide some insight on this.

[sunilmut]

I ran some benchmark tests from the phoronix benchmark tool, especially the ones targeted at Disk I/O. @michaellarabel has published phoronix results on WSL vs Native Ubuntu, both on Anniversary Update and Creators Update. I looked at two tests, specifically the PHP compilation and SQL test. The tests were run on the same system on a recent insider build 16215 and Creators Update, to see if there has been perf improvements. All the default settings were used and no special perf tuning was done. I am not including the specifications of the test system, to specifically keep the focus on the delta, rather than the absolute.

Some additional notes:

• All the tests were run on LxFS (~ or $home dir)
• Lower is better in the below tables.

Scenario: build-php-1.3.1

Build: 16215

Time in Sec ->	Min (s)	Max (s)	Avg (s)
Run 1	228	259	238
Run 2	239	348	261

Build: 15063 - Creators Update

Time in Sec ->	Min (s)	Max (s)	Avg (s)
Run 1	289	307	296
Run 2	284	358	313

Scenario: SQLite 3.8.10.2

Build: 16215

Time in Sec ->	Min (s)	Max (s)	Avg (s)
Run 1	1339	1412	1367
Run 2	1272	1293	1283

Build: 15063 - Creators Update

Time in Sec ->	Min (s)	Max (s)	Avg (s)
Run 1	1426	1459	1438
Run 2	1330	1693	1470

Observations: The perf (during the above runs and for the above scenarios) on 16215 build was consistently better than those in 15063. And, there seems to be moderate gains as well, though, it's not clear if that is consistent.

We encourage everyone to run the tests locally to verify and assess. And, for those who hasn't voted to get perf improvements on our User Voice page, please do so.

Note: These are just results from my test system and should not be taken as established results.

[sunilmut]

@michaellarabel recently published the benchmark numbers with WSL build 16232 and also included numbers from VirtualBox. Interesting read. Thanks, @michaellarabel. It continues to highlight the WSL's weakness (and it's strengths) in the I/O handling and something that we are well aware of. And, we will continue to invest in this space. For transparency, many portions of the solution lie outside of WSL itself.

[jamesmugford]

I wonder if there's any solution to mirror a directory under /mnt/c/ and ~/

So for example: /mnt/c/exampleproject mirrors to ~/exampleproject

Ideally it would use inotify events to trigger say a simple cp or rsync command to keep the 2 directories in sync.

It would be a hack but it would at least make WSL usable.

Without this, WSL is just not an option for many/most. (Why can't we just have nice things???)

[fpqc]

The latest build applies default permissions (root:root, 0000) to files missing extended attributes. I think they're going to work on a better solution than you're describing (like mounting WSL's ~ as a FUSE type thing using GVFlt maybe).

[jamesmugford]

Ooohhh! Thanks for sharing that!

I wonder if this default permission could be manually configured (to something like root:root 0755)?

This would make WSL fairly usable, for now.

[fpqc]

@JamesMugford No idea, I'm just going from the 16237 patch notes: https://msdn.microsoft.com/en-us/commandline/wsl/release_notes