Open SteveSandersonMS opened 4 years ago
I stumbled on this problem when working with a Ruby project in WSL2 (with WSL extension in VS code). I first noticed that the source control panel did not update on saving files. Then, that debug configurations (edits to launch.json) only were refreshed after restarting VS code. Finally the same things with tasks... so I had to investigate.
Moving my project files to the Linux side was not an option (I run Sketchup on the Windows side, consuming the Ruby files). But I still need a snappy development environment, with Linux/WSL capabilities.
Sadly I realized there are (currently) no simple fixes to the broken inotify functionality of the mounted 9P windows file system (I tried a number of things, including watching the directory inside of Linux and injecting change events there). No easy way forward... back in cross platform murky land...
My solution in the end looks like syncthing - a versatile directory sync tool - over TCP. So I have two copies of the project, one on Windows side and one on Linux side. And they are kept in sync, and both behave as local file systems (because they are).
Took me a day to stumble, research, learn about the issue and experiment with this solution...
Would be nice to have this working out of the box with WSL2.
@asteinarson I had a similar setup for a while with Unison, but had to give it up because it would more often than not crash when changes occurred on both sides (IDE + git on Windows, web server + CLI tools w/ file watchers on Linux VM).
I hope you'll be luckier!
npm run serve
from git bash. It will hot reload.
bump
The issue can be avoided by not mixing file systems, e.g. NTFS with Linux file system until a permanent fix is available by Microsoft for WSL2.
To help others saving time and pain, I wrote up my findings in an article. In it I also describe how I resolved the issue for the combination of VSCode, WSL2 and a Linux-based dev container for .NET. Here is the link: https://levelup.gitconnected.com/docker-desktop-on-wsl2-the-problem-with-mixing-file-systems-a8b5dcd79b22?sk=53d24e33a9f247fd626e3aa6959de7d4
My solution to have everything work seamlessly is to downgrade to WSL1.
Do Not Upgrade to WSL2.
Going back to WSL1 ? Open a PowerShell :
wsl -l -v
wsl --set-version <distro-name> 1
Even if your favorite program advocate for it (like Vscode who tell you WSL2 have better performance), there's so many pitfall that it's not worth doing it.
File system is broken, you can't work in /mnt/c or /c/, you have to put all of your workflow into the Linux File System (that mean you can't access it with Windows app like GitKraken). And nothing seem to evolve.
Edit : I don't want to be to critic because I really like WSL, it's a game changer project. It's not bad to wait things stabilized before upgrading to WSL2. Homewever, I revert what I said about the /home/ directory, it seem it's a best practice to put your files here for better performance and file system seem to work fine. You can still access your files on Windows app with "\wsl$" path, it's a network file system. But anyway, mixing file systems is always a complicated issue. I hope this issue will get more traction in 2021 👍 .
https://devblogs.microsoft.com/commandline/wsl-2-is-now-available-in-windows-insiders/
File system is broken, you can't work in /mnt/c or /c/, you have to put all of your workflow into the Linux File System (that mean you can't access it with Windows app like GitKraken). And nothing seem to evolve.
This. I don't understand why we have to pick between decent file-system performance under Linux, or a normal, practical, functioning everyday life with Windows. These shouldn't be mutually exclusive options.
If the file-system performance in Windows isn't up to the task, really what they should be doing is improve the native Windows file-system performance - which would surely be a win, not just for Linux usage.
If the file-system facilities in Windows can't handle the Linux permission data, maybe it's time to enhance the file-system with better meta-data capabilities.
If the file-systems are too different to integrate, maybe it's time to add support for native Linux file-systems in Windows.
And so on.
Having to use a virtual disk seems like a tedious workaround - I'm really hoping we'll someday soon see real improvements to the underlying technology, instead of these half measures. Four years on and WSL remains this "would-be awesome" feature that seems to always be "almost there", but never quite gets there.
Damnit, Microsoft, we're ready - now take us there, please! 🙏
@kMeillet and @mindplay-dk that's some overall great feedback! I agree completely, one of the biggest challenges for any user upgrading to WSL 2 is the fact that they will have to put their projects into the Linux file system to gain the performance advantages. On the team we're investigating ways to improve this experience (and always open for different ideas!). One of the major factors here is that currently the maximum speed we could theoretically make WSL 2 run on the /mnt/c/ file system is the same speeds as WSL 1, which is still too slow for many users. Overall I believe this speaks to the points in this thread, and we are looking into plans here that are going to a be a mixture of long term and short term improvements.
Honestly, the suggestion to use linux filesystem is killing me. Why? Because the only filesystem you can trust is your real machine filesystem. I manage my folders and files and I know what I'm doing with them. But I don't manage the volume in WSL2 (I even didn't create it - it was "magically" created with the system upgrade and activation of WSL2). So, personally, I'm afraid that due to some kind of my ignorance I risk to delete it occasionally. Maybe to some of you it will sound silly/not enough persuasive, but for me switching to this way of workflow is almost not anyhow different from running just a virtual machine with a mounted folder. With the slight bonus for a VM due to the fact that I do there everything explicitly.
One of the major factors here is that currently the maximum speed we could theoretically make WSL 2 run on the /mnt/c/ file system is the same speeds as WSL 1, which is still too slow for many users.
Personally, I would love to have WSL 2 with the same performance of WSL 1, but usable. As it is, it's just not usable to me. With WSL 2 I would be able to tell my IDE (JetBrains) to use Git from the Linux VM, making Husky hooks work (Linux binaries don't work when using Git for Windows). Unfortunately, this feature only supports WSL 2.
One of the major factors here is that currently the maximum speed we could theoretically make WSL 2 run on the /mnt/c/ file system is the same speeds as WSL 1, which is still too slow for many users.
How about adding a write-behind cache buffer of some sort? Even a few megabytes should help tremendously, as the performance problems are mostly felt with things like npm install
, compilers etc., which creates thousands of tiny files.
I understand this would make the guest OS less resilient to accidental shutdown or a hard crash - but I think, mostly, Linux systems on WSL are used for development, where someone would likely prefer performance over some theoretical risk of data loss under some unusual circumstances? (I've never heard of anybody running production services on WSL? You could of course provide a way to disable a cache feature.)
One of the major factors here is that currently the maximum speed we could theoretically make WSL 2 run on the /mnt/c/ file system is the same speeds as WSL 1, which is still too slow for many users.
How about adding a write-behind cache buffer of some sort? Even a few megabytes should help tremendously, as the performance problems are mostly felt with things like
npm install
, compilers etc., which creates thousands of tiny files.I understand this would make the guest OS less resilient to accidental shutdown or a hard crash - but I think, mostly, Linux systems on WSL are used for development, where someone would likely prefer performance over some theoretical risk of data loss under some unusual circumstances? (I've never heard of anybody running production services on WSL? You could of course provide a way to disable a cache feature.)
That's a very good point, at least, I don't really care about data loss unless it break my system
So any news about this bug fixing? It's very annoying moving the code to linux filesystem to properly develop...
I just want VS Code Live Server extension to work correctly on mnt\c with wsl2. Downgrading to version 1 does not feel like a good workaround.
Try prefixing your WSL2 command invocations with the following environment variable, if the commands need to subscribe to filesystem I/O notifications:
CHOKIDAR_USEPOLLING=true
@dejayc this will work but it comes with a huge performance cost. CPU usage will increase significantly (30% on my system) as it needs to do a lot of active polling of the file system for changes. Additionally, the change detection is not nearly as fast as on the pure linux file system.
This should be a huge warning before allowing any upgrade. A lot of effort to migrate to WSL2 and now I have to make my way back. So much time was lost trying to understand what was happening for no reason.
I am developing code which needs to run on both Windows and Linux. My primary development environment is Windows, but sometimes it is useful to be able to build and test the same code in a Linux environment during development.
I have a symlink in my WSL environment to my main dev code area in Windows. I have just built in WSL2 and found a GCC warning I don't care about in a boost header, so I updated my source to disable that warning while including the header. The build still failed.
After a bit of head-scratching, I thought to do an ls -l on the file under WSL, and noticed the last-modified time was still yesterday afternoon, but the symlink is definitely pointing to the file I just modified.
Putting my development tree in the WSL filesystem and linking from Windows makes NO sense to me.
I consider this a serious bug in WSL2. Fortunately I found this thread and downgrading to WSL1 fixes the problem, but this is a deal-breaker for me with WSL2.
With the recently-announced official support for GUI apps, this seems now even more important?
Since this opens WSL to non-developers, who just want to run Linux desktop software on their PC, aren't they going to run into weird situations when they try to work with files and documents in their PC home folders? 🤔
If you use VSCode, this workaround may be useful: I have switched to container based development. So, all your work files are located on the container, not on the windows "native" partition. The key which made this workaround useful for me is that you can still access the container source files via windows explorer as a psudo-network share. I can still use SourceTree (a windows, GUI based git client) by passing it the git repo location. Having a gui based git-client was the main reason I wanted my source files on the windows file system, so this workaround effectively solves everything for me.
If you want to access your docker containers files (mounted partitions) from windows, it is located at something like: \\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\vsc-remote-containers\_data
This location is very poorly documented and a lot of false information floating around at places like stack overflow. only by luck did I stumble on it.
I hope that helps you out too! Again, This is useful if you need to access the 'source' filesystem from windows while needing fast+fs notify support. You do need to watch out for filesystem permissions though, as in my case, my first attempt resulted in a slightly corrupt git folder because permissions defaulted to ROOT/ROOT. After I ensured the folder permissions include the linux user's user/group and re-downloaded the repository everything worked fine (and no git corruption has occured)
Edit: a big bonus side effect is that with vscode, the linux container's definition is now included in the git repository under a .devcontainer
folder, ensuring that the entire dev stack is now reproducable using the git repo as the single source.
If you use VSCode, this workaround may be useful: I have switched to container based development. So, all your work files are located on the container, not on the windows "native" partition. The key which made this workaround useful for me is that you can still access the container source files via windows explorer as a psudo-network share. I can still use SourceTree (a windows, GUI based git client) by passing it the git repo location. Having a gui based git-client was the main reason I wanted my source files on the windows file system, so this workaround effectively solves everything for me.
If you want to access your docker containers files (mounted partitions) from windows, it is located at something like:
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\vsc-remote-containers\_data
This location is very poorly documented and a lot of false information floating around at places like stack overflow. only by luck did I stumble on it.I hope that helps you out too! Again, This is useful if you need to access the 'source' filesystem from windows while needing fast+fs notify support. You do need to watch out for filesystem permissions though, as in my case, my first attempt resulted in a slightly corrupt git folder because permissions defaulted to ROOT/ROOT. After I ensured the folder permissions include the linux user's user/group and re-downloaded the repository everything worked fine (and no git corruption has occured)
Edit: a big bonus side effect is that with vscode, the linux container's definition is now included in the git repository under a
.devcontainer
folder, ensuring that the entire dev stack is now reproducable using the git repo as the single source.
Thanks for sharing this, @jasonswearingen
@mindplay-dk WSL overall is targeted towards developer audiences, GUI app support in WSL is targeted towards allowing developers to use Linux GUI apps on Windows.
The feedback in this thread is super valuable to us. I understand that many folks can't move their files over to the Linux file system, and it's helpful for us to see the exact whys listed above. As @jasonswearingen detailed there are some great workflows now for projects living in the Linux file system (VS Code Remote, and now GUI apps allows you to access these projects more easily). I'll make sure to keep this thread updated with any news here, thank you!
For me, the use use case I look for is to be able to have a file open in a Windows app (usually an editor) and as that file is updated (often by tooling) on the Linux side, I 'd like to see that in the Windows app (without manually going there and refreshing the window / rescanning the directory).
It holds for MS file explorer as well, I'd like it to show accurate file sizes and file dates, when I browse files that are located on the Linux side.
A particular use case I had was developing Ruby Sketchup extensions. Now, Sketchup can only run in Windows environments, so my plugin code needs to be available on that system. However, the tooling I use (to edit, to generate the Ruby code), I prefer to run in a good Linux environment. I was not able to set that up in a straight forward way (I ended up with something like a deploy, from Linux to Windows).
I don't expect "stellar file system performance" in these scenarios. It's more about seeing changes done to a handful of files (on the Linux side) being accurately reflected on the Windows side, within a second or two.
If you use VSCode, this workaround may be useful: I have switched to container based development. So, all your work files are located on the container, not on the windows "native" partition. The key which made this workaround useful for me is that you can still access the container source files via windows explorer as a psudo-network share. I can still use SourceTree (a windows, GUI based git client) by passing it the git repo location. Having a gui based git-client was the main reason I wanted my source files on the windows file system, so this workaround effectively solves everything for me.
If you want to access your docker containers files (mounted partitions) from windows, it is located at something like:
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\vsc-remote-containers\_data
This location is very poorly documented and a lot of false information floating around at places like stack overflow. only by luck did I stumble on it.I hope that helps you out too! Again, This is useful if you need to access the 'source' filesystem from windows while needing fast+fs notify support. You do need to watch out for filesystem permissions though, as in my case, my first attempt resulted in a slightly corrupt git folder because permissions defaulted to ROOT/ROOT. After I ensured the folder permissions include the linux user's user/group and re-downloaded the repository everything worked fine (and no git corruption has occured)
Edit: a big bonus side effect is that with vscode, the linux container's definition is now included in the git repository under a
.devcontainer
folder, ensuring that the entire dev stack is now reproducable using the git repo as the single source.
Thx for sharing. I will check this out
@dejayc this will work but it comes with a huge performance cost. CPU usage will increase significantly (30% on my system) as it needs to do a lot of active polling of the file system for changes. Additionally, the change detection is not nearly as fast as on the pure linux file system.
Using CHOKIDAR_USEPOLLING=true
on my relatively modest laptop doesn't increase CPU usage by even 1%.
Using
CHOKIDAR_USEPOLLING=true
on my relatively modest laptop doesn't increase CPU usage by even 1%.
It depends significantly on the number of files being watched. On my powerful desktop workstation CPU usage would rise sometimes to 50% depending on which project I have open.
One of the major factors here is that currently the maximum speed we could theoretically make WSL 2 run on the /mnt/c/ file system is the same speeds as WSL 1, which is still too slow for many users.
Just to add my two cents; while I do agree that it would still be too slow for many users, getting back into that range of being some 10x faster would probably be 'enough' for most of us. It wasn't a problem I personally had with WSL1, but it is a very real problem with WSL2 unfortunately. I'm able to move some of my projects over to the wsl side but not all of them, two reasons for this are:
It may be worth providing an official update over in #4197 since it's been over a year since the last update, and the discussion has gotten very fragmented with that issue locked. Or maybe just a meta discussion issue for these various WSL1 -> WSL2 filesystem regressions?
Well, tried all manner of things to get this working under Windows + WSL 2... still cant. Followed this blog post too: https://www.forevolve.com/en/articles/2020/02/04/speed-up-your-builds-and-watch-for-changes-to-up-to-375-percent-using-this-workaround-on-wsl2-ubuntu/
I tried hosting original project in Windows file system, and Ubuntu file system. I've tried starting VSCode from both Windows and WSL. Have tried VSCode in Remote-WSL mode, and regular. I've tried running my server from both WSL Terminal, as well as the VSCode built-in Terminal.
None of that is working/getting file changed events. Wtf Windows. Back to Ubuntu native. Grumble.
Is there a timeline for when this is going to be implemented/fixed? I make heavy use of OneDrive with all my scientific projects/code files hosted there . It's not really feasible for me to break up my directory structure and start putting files in WSL2.
When it will be fixed? I can't use Sublime 4 (file tree doesnt update)
I know that I am not adding a fix and that many people probably know about this but in case someone who has such problems is struggling because of difficult access to the filesystem of the OS running in WSL:
One can access the filesystem of the OS via Network connection
All currently running distributions (wsl -l) are accessible via network connection. To get there run a command [WIN+R] (keyboard shortcut) or type in File Explorer address bar \wsl$ to find respective distribution names and access their root file systems.
https://docs.microsoft.com/en-us/windows/wsl/compare-versions#performance-across-os-file-systems
Nevertheless, I would love to use my windows filesystem in my linux distro in WLS2 with full capabilities.
Is this being worked on? This issue renders many dev environments (and thus WSL as a whole) mostly useless for me and undoubtedly others.
The solution of prepending CHOKIDAR_USEPOLLING=true
to the pertinent command doesn't work in the case of more complex applications, such as running VS Code.
Same, any update on this? I use Eclipse with a Java application, through WSL2 and docker volumes shared I'm working on the Windows end the files aren't hot-reloading.
Any fix on eclipse and Java maybe from others? No issue with Hyper-v though (just a lot slower)
Any update or workaround for this? Looking to use https://github.com/cosmtrek/air for some docker containers with the code in windows but docker running WSL2 backend. No file updates from windows and therefore air is not hot reloading...
I add the rsync scenario developing with .NET:
dotnet publish
to create the new linux binariesrsync
every time the publish folder is updated.In my scenario, I want to share directory in windows and watch file changes in /mnt/d. Maybe I have to do the whole job all inside Linux.
Am I right to think that this issue is blocked by the lack of inotify support in FUSE ? Does WSL2 use FUSE under the hood ?
If I am not mistaken, it uses the 9P file system over VSOCK. That does not rely on FUSE; it is a linux kernel module
This paints me into a corner where I have to keep my app on an older version of Parcel that (optionally) uses a polling-based approach to detect file changes. The older version lets me edit files in VS Code on Windows, and the changes get picked up in a running docker container (on WSL2). The detection isn't awesome or quick, but it works. My bigger concern is that I'm stuck with old versions of libraries that aren't being maintained anymore.
I just adding my comment that I have also this problem and it means that I can't use Docker for developing my web app :(
This issue might be the reason, why remote development in VSCode does not work well on Windows. All tools that rely on the state of the file, like CMake, or the Git extension need manual refresh each time a file has changed. That's extremely annoying.
Will this problem be tackled?
Thanks :)
@jankap https://levelup.gitconnected.com/docker-desktop-on-wsl2-the-problem-with-mixing-file-systems-a8b5dcd79b22 should fix it although I haven't tried yet.
Also adding to this developing frontend via a development container with hot reload not working is terrible. It would be nice to have this fixed one day.
yep, parceljs/webpack 5 hmr not working
I have a node container and found an fix that works for me. I added the CHOKIDAR_USEPOLLING environment variable to the container in my docker-compose.yml. Now my watchers are working.
mynodecontainer:
[...]
environment:
- CHOKIDAR_USEPOLLING=1
Maybe it is not the best solution, but now i can develop with docker again.
I'm using Rollup and it can't detect changes even with polling.
What is the status of this issue?
This issue persists. Is there a timeline for when it will be resolved?
Also having this issue, whish it gets fixed but 2 years without progress make me doubt it...
BUILD TIME Comparism with Maven, Java + Docker when people tell me that WSL2 is faster when build from the linux subsystem..... well these numbers don't lie
- Powershell default setup (second time)
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25.722 s
[INFO] Finished at: 2022-06-02T18:57:23+02:00
- WSL2: Started build in WSL2 didn't move the data /mnt/c (first time)
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:15 min
[INFO] Finished at: 2022-06-02T19:05:52+02:00
- WSL2: did a fresh git-clone in /home/git (first time)
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:12 min
[INFO] Finished at: 2022-06-02T19:13:50+02:00
- WSL2: did a fresh git-clone in /home/git (second time)
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:52 min
[INFO] Finished at: 2022-06-02T19:22:03+02:00
- WSL2: fit-clone in /home/git & also clean maven artifacts in /home/.m2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47.331 s
[INFO] Finished at: 2022-06-02T19:45:44+02:00
So all compared WSL2 is still almost 2x slower than build directly from Windows ==> Hence this BUG should get more priority I'm not even counting the extra time with the Remote VSCode and terminal time you loose initially connection and starting the WSL2.
Very disappointing that this is still an issue after 3 years.
WSL2 is really close to being a perfect runtime environment for server apps being developed in Windows. Great job! One missing feature however is breaking a core part of the developer flow.
For sources stored on the Windows filesystem, any changes made by Windows applications such as Visual Studio do not trigger any file change notifications as far as Linux apps are concerned. This means that all "live rebuild"-type tools don't work (examples:
webpack --watch
,jekyll --interactive
, and Tilt.dev) when running under WSL2. This unfortunately renders many modern dev workflows unviable.Notes:
fs.inotify.max_user_watches
doesn't affect this, since the issue is about changes originating on the Windows side.Bug report template
Your Windows build number: 10.0.19033.1
What you're doing and what's happening:
This applies to all tools that listen for file change notifications, but as an example take
webpack
. Repro steps:c:\repro
), and then add these three files to itsudo apt-get install nodejs npm
cd /mnt/c/repro
npm i
npm run build:watch
. Wait a few seconds until it completes the first build. It will now be waiting for further changes to your source files.c:\repro\index.js
and save some change to it. For example, change'Hello, world'
to'Hello, world 2'
.What's wrong / what should be happening instead:
Expected behavior: Webpack should see the change and rebuild. That is, you'll see it log information about another build, and the output in
dist/bundle.js
will be updated.Actual behavior: Webpack doesn't respond at all, because there's no file change notification.
Finally I understand that the fix for this is likely to be "add file watch capabilities to the Plan9 server", and you may feel this is already being tracked by #4064. However #4064 describes a more obscure symptom of this missing feature and makes it sound like an intermittent issue. What I'm reporting here is not intermittent at all, and is a pretty mainstream scenario (using tools like
webpack --watch
). Thanks!