Manipulate Linux/Ubuntu subsystem files from via the Windows environment

[BJobsis]

• A brief description: When I try to create a file from Windows10 (using Sublime Text, etc) into the Linux Ubuntu subsystem the file is created, but afterwards the file can't be seen from within the subsystem. I bypass this by creating (and cutting/pasting) the file via the NANO cmd.

• Expected results It would be nice (like in Apple's OSX terminal environment) to manipulate files from outside the Linux subsystem. In my opinion (as a developer) this would mean a big step forward in the usability of the Ubuntu subsystem.

*My Windows build number: W10 Build 1607 14393.576.

Regards, Barry Jobsis (barry.jobsis@gmail.com)

[aseering]

Thanks @BJobsis for reporting this! For what it's worth, I think this is the same basic issue as #1239 , though the way you have phrased it is more general.

If you (or anyone else reading this particular issue) has not already seen this blog post, I'd recommend reading it: https://blogs.msdn.microsoft.com/commandline/2016/11/17/do-not-change-linux-files-using-windows-apps-and-tools/

The intent, in short, is that you put your files in /mnt/c/ somewhere if you want to get at them from Windows.

[aseering]

@benhillis -- this UserVoice appears to indicate that this issue is WONTFIX. Could you (or could someone else at Microsoft) make sure this ticket and that UserVoice are flagged the same way?

[fpqc]

@aseering It looks like their thinking has moved from "wontfix" to "might eventually fix in RS3 or RS4", based on some hints I've seen dropped.

[benhillis]

@aseering - I'm not sure why this was ever marked as won't fix. This is a huge limitation of the current design and definitely something we're looking at improving. I'll get that user voice status changed. Thanks for the heads up.

[fpqc]

Lol, and that's about as close to a confirmation as we've seen!

[benhillis]

It's a hard problem to solve (mapping Windows permissions to Linux permissions or vise versa). Just ask anybody who's worked with Samba.

[fpqc]

@benhillis Yup, the Samba people blamed you guys for it though!! (ACL format is not stable between windows versions).

[benhillis]

I'm pretty sure I wasn't born yet when those decisions were made :)

[therealkenc]

But if one shrinks the problem space to /home, ie "development related scenarios", the permissions problem solves itself. Just ask anybody who's worked with OneDrive. 😃

Longer term, there is no reason why users or permissions need to be mapped in the first place (but I digress).

[pharmmd-rich]

do what I do, create a symlink to your documents folder and just work on stuff like that. If you are really brave, you could try symlinking your windows user directory to your home directory. I haven't tried that though, so it might break things.

[fpqc]

Will break everything. Don't do it.

[therealkenc]

Here, fixed that for you. This also represents the third way I alluded to a month ago in #1397 (comment). Options available to have VSCode open a file on a WSL path:

1. VSCode would have to have built in logic to understand WSL file paths.
2. WSL could do some magic to the command line before creating the NT process. [Ben] outlined some of the trouble with that in #1243
3. Orrrr... Use a Window Filesystem that has built in logic that understands WSL file paths.

I give you, nfs-ganesha on WSL and nfs-fuse in Windows via dokan-dev. That's VSCode editing .bashrc, launched from WSL home directory on L: drive (natch) in Windows Explorer. Code to follow when it sucks less.

[fpqc]

@therealkenc Lol, was waiting for someone to do this. I figured you could do it horribly by setting up a local Samba share and mounting it on a Windows network mount, but then I considered what a hassle it would be. I'm guessing you've essentially streamlined this using dokan?

Hope I'm doing it justice.

[therealkenc]

Samba is problematic because Windows is a CIFS server and those ports are spoken for. This is NFS not CIFS. There is a NFS client available for Windows, but Microsoft considers the technology all Enterprisey 'n stuff (as opposed to the thing your teenager uses to share XBMC videos throughout the house).

I considered a fusesmb port to dokan, with which you could choose a nonstandard port, but that was just a bridge too silly to contemplate seriously. That is thread #764 (message). In fact if I knew how to redirect the CIFS ports on Windows this hack probably wouldn't have happened.

[fpqc]

So KenC, does this automagically fix paths (so launching code.exe ~/blah opens the file with a path relative to the nfs mount?

[therealkenc]

You'd get automagic translation if you mount / on L:, because then there's simply nothing to translate. That's problematic for other reasons though. I'll start playing with path translation after (if) I get more pressing stuff addressed. Truth is I haven't looked into how the interop stuff deals with paths much; notably $PWD. As of this writing it is mostly useful (if that is the word) for having write-access to VolFS files from the CMD prompt not the WSL prompt.

It's good to know they have their top men on this however. Though I would prefer if it were someone with a very particular set of skills.

[fpqc]

@therealkenc By the way, you don't need dokany if you have Win10 pro. There is some incorrect info, but the NFS client is still available for installation in the optional Windows features.

By the way, I built ganesha and set up a pkgbuild that I think is right, but I was wondering what you did and how you set up your config to get it to work (this won't build locally given the screwed up state of my build tools, but I do have a server running Arch, so I built it there).

PKGBUILD.txt

[therealkenc]

Ganesha doesn't work out of the box because of #1530 and to a lesser degree #1492. It was a relatively large work-around (basically implementing a dcache in userspace). Rather than "going public" I shelved this for the time being. #1529 makes interop painful unless you disable fd caching entirely. And there's a bug in ganesha itself that needs to be worked out. I get the (possibly unfair) impression ganesha is used mostly for cluster stuff; which is understandable because Real Linux™ has a perfectly serviceable NFS kernel module.

Upshot is that I'm presently looking into porting libsmbclient to Windows, even if that means they might have to haul me off to a padded room.

Funny story about Win10 Pro. I tried to give myself a Pro upgrade for my home machine as a Christmas present, but Microsoft wouldn't take my money because no US address to match my credit card. So I ported nfs-fuse to Dokan instead.

[fpqc]

@therealkenc =.= Yes, Microsoft wouldn't take our money at work to upg to Enterprise. They wanted us to contact a partner or something so we gave up.

We should have lied & told them we were deficient in our licenses or something lol. Then they'd have to give us something we could buy to move back into compliance!

Anyway, will keep an eye on those bugs. And yes, you are probably insane to try to port libsmbclient back to Windows!

[kayakyakr]

Late, but glad to hear that the perception has shifted toward allowing windows to access VolFS.

Still hoping that DriveFS can eventually support all the calls expected. The use case of IDE of choice + WSL + Docker is a really good environment, but is still just barely not supporting a lot of the frameworks.

Stock Rails works! Ember-Cli does not :(

[therealkenc]

Okay; there is one solution using stuff in everyone's toolbelt. Apache2 on WSL works alright enough out of the box these days. Which means WebDAV works. For those not familiar, friendly instructions for Apache here and WebDAV here (among many other sources). Mounting a WebDAV drive on Windows 10 here.

It works better than passably well for file editing and copying -ish use cases. I dragged-and-dropped a 1.5GB iso image and performance wasn't bad to the naked eye. Doing something like find-in-files on a big tree with Windows VSCode does work, but won't win any benchmark competitions. I'd post another screenshot, but it looks a lot like the last one.

[BJobsis]

Is there someone at Microsoft who can confirm that this request will be implemented in the (i hope near) future?

[fpqc]

@BJobsis Not yet It's something that they are looking at how to do for Redstone 3 (post-April). It will not make it into this series of insider builds. Read Rich Turner's Microsoft twitter feed @richturnms . Also Ben Hillis (MS Developer) has said the same a few times on this issues board. Please look for yourself.

[markallasread]

Subscribed. This is really my only impediment in swapping to windows full time (which, admittedly, I did — but am now regretting as a result of this).

• Creating a file from windows on the wsl folder makes it invisible & gives it really weird properties. (Edit: I should now note I see this is not supported, and heavily recommended against — don't be me!)
• Mounting windows drives on linux (or viewing them) shows everything with 777 permissions (and makes my eyes twitch, as a long-time linux/macos user)
• I tried running my daemons from windows as an alternative (mkdocs, jekyll, etc) but they hate backslashes and pretend none of my files exist.

Altogether, these aren't showstoppers, but when I use my IDE/text editor in windows (atom/sublime text/etc) — and create a file, it takes a few hours before I realize my mistake and have to undo a few hours of work.

For now, I'm simply using dropbox as a go-between — yes, everything has 777 permissions & drives me batty, but I can solve that with a simple script when I provision my work.

[fpqc]

@davidandgoliath My understanding is that they plan on removing some of the limitations on editing lxss files from Windows in the future, but I have no idea how they plan to do so. There have been a few suggestions from the community, but however the Devs are going to do it, they'll make sure it makes sense. We do know for sure that this won't be in the insiders builds at least until RS2 RTM (End of April). It might take months after that before we see anything in the RS3 Insider builds. I do not know, and the devs are loath to speculate. Addressing 777 mounts will likely flow from however they decide to do this.

As for daemons, this is going to rely on two things: Finishing syscalls and kernel features required by systemd, and the integration of the WSL init daemon (which manages Linux-side of the LXSSBus and provides the Win32 control and console interfaces) as a standalone daemon service run by systemd. This target was alluded to in passing, I think, in Ben Hillis's blogpost about interop, but we have rather less information about it aside from the uservoice response and Ben's mention of WSL-init outside of PID1.

So keep your fingers and toes crossed.

Also, the 777 factoid is not relevant, since cmd can be launched interactively from WSL and used to edit /etc/shadow through the Windows-side interface. If someone knows you are running WSL, root is effectively equivalent to any other user.

[pharmmd-rich]

IMO it's such a low priority. You can mount any Windows folder you need inside your linux system at any point in the filesystem. I have my Windows Documents folder mounted in ~/Documents for instance. I personally would rather see them work on improving compatibility, adding support for starting linux services at WIndows startup, and performance improvements (compiling a project on Ubuntu native is currently much faster than on WSL). I can't think of a single scenario where you would need to do the opposite and edit files directly on the WSL FS.

At any rate, to fix editing they could easily create a virtual (.vhd) EXT4 drive for the FS and store it in the User folder in Windows. Double click the VHD, it will mount, and you can edit files.

[kayakyakr]

Agree with @pharmmd-rich: I think the goal right now should be getting DriveFS up to speed so it performs comparably with VolFS and supports all the ops necessary. It seems that's a goal of theirs too as the most recent build came with DriveFS performance improvements.

Think this is a good place to mention really hoping that #1357 is fixed soon as it's the last hard block (that I know of) to my swapping to windows development full time. Though DriveFS performance is a soft block.

Linux services are better handled in Docker than WSL, imo.

[markallasread]

The underlying issue is that when I commit my code to repos, my permissions (and, by default, everything being 777 is committed as well). Perhaps this is more part of my issue: https://github.com/Microsoft/BashOnWindows/issues/352

[aseering]

@davidandgoliath -- as a workaround, that behavior is configurable for git via git config core.fileMode false. I would expect at least some other version-control systems to have similar workarounds, though I don't know them offhand.

[aseering]

Though, as someone who manages both code and executable scripts in git (so file permissions do matter), I actually do wish that DrvFS had better support for UNIX/Linux semantics. It's not a blocker for anything that I do, but it is a nuisance for some things.

[rtosman]

I am trying to get nfs-ganesha / nfs-fuse solution going on Anniversary edition. Here is the result of attempting to start ganesha.nfsd in WSL:

06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] init_export_root :RW LOCK :F_DBG :Unlocked 0x11795d0 (&export->lock) at /home/rallen/utilities/nfs-ganesha/src/support/exports.c:2290 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] init_export_root :RW LOCK :F_DBG :Unlocked 0x1179e88 (&obj->state_hdl->state_lock) at /home/rallen/utilities/nfs-ganesha/src/support/exports.c:2291 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] init_export_root :EXPORT :DEBUG :Added root obj 0x1179a38 FSAL MDCACHE for path / on export_id=0 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] foreach_gsh_export :RW LOCK :F_DBG :Unlocked 0x7c4140 (&export_by_id.lock) at /home/rallen/utilities/nfs-ganesha/src/support/export_mgr.c:718 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] nfs_Init :NFS STARTUP :INFO :krb5 keytab path successfully set to 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] nfs_Init :NFS STARTUP :INFO :gss principal "nfs" successfully set 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] nfs_Init_svc :DISP :DEBUG :NFS INIT: Core options = 15 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] fridgethr_init :RW LOCK :F_DBG :Init cond 0x117a180 (&frobj->deferment.block.cond) at /home/rallen/utilities/nfs-ganesha/src/support/fridgethr.c:158 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] nfs_Init_svc :DISP :INFO :NFS INIT: using TIRPC 06/04/2017 14:45:41 : epoch 58e6b705 : RALLEN4-W7 : nfs-ganesha-8951[main] nfs_Init_svc :DISP :FATAL :Cannot get udp netconfig, cannot get an entry for udp in netconfig file. Check file /etc/netconfig...

Any ideas on what I am missing here?

[kayakyakr]

Straight anniversary? That is so far behind the capabilities of the Creator's Update that it might be a better idea to just wait the 5 days until CU goes official and then try again.

For the record, I've been using an apache2/webdav methodology with moderate success (have to tell everything to ignore .DAV folders and it's slow, but it was fairly simple to set up and works. will also require the CU).

[rtosman]

Yeah POAE, unfortunately, corporate IT won't let me download insider updates because of the telemetry requirements.

Good to know webdav is working for you.

[aseering]

@rtosman -- Creators (non-Insider) has actually already been released as a .iso, though it won't be available via Windows Update for a few more days. You can get it here:

https://www.microsoft.com/en-us/software-download/windows10ISO

Unfortunately, it may not help in this particular case: WSL doesn't yet support FUSE. I don't see a global ticket for it, but specific applications have requested it as early as #412 .

[rtosman]

@aseering I am confused as to how @therealkenc was able to get it running back in December then. Isn't the fuse requirement only for the Windows side and supplied via doken?

[aseering]

@rtosman -- ah, yes, in that direction it should work. (Sorry, I had lost track of the objective of this ticket.) The WSL limitation just affects mounting FUSE filesystems within WSL.

[rtosman]

Cool. Thanks for the input. I'll get this working somehow :-)

[therealkenc]

Ganesha won't fly on Creators because of #1530 which remains open, and probably will remain so for a while. User Voice, or something.

[rtosman]

Thanks @therealkenc I am curious, what is your embarrassingly bad work-around?

[therealkenc]

I mentioned it in the other issue. I implemented a dentry cache in userspace to emulate open_by_handle_at(). If you look at the screencap up thread, you can see my trace statements looking up entries in the cache. It was embarrassingly bad, because I just allocated a fixed 64k array to store the entries and made zero effort towards a proper LRU ejection mechanism. That is doable, but would be wasted effort because (a) the WSL team will get to open_by_handle_at() eventually making it all throw-away code and (b) both Ganesha and Dokan have bugs and limitations that would make the end result uninspiring to end-users anyway. I outlined some of the issues in winfsp#74.

[rtosman]

@therealkenc Thanks for the update. I definitely need to find something usable for editing files on volfs from Windows space. CIFS would definitely be ok, but looking into GvFlt it looks like it still isn't officially part of Windows yet (someone please correct me if I am wrong).

[therealkenc]

It isn't; though the theory is that it will be part of Windows some time in the foreseeable future. GvFlt also doesn't support CreateSymbolicLink() or FindFirstChangeNotification(), though the theory is they might be amenable to adding support. In other words, all very theoretical on multiple levels, starting with the part about no SMB client library.

[ghost]

My dirty workaround with this is actually setting up sshd a new(since pre-installed does not work) and then using a sftp client(WinSCP) for file manipulation outside of WSL - Works like a charm! :) For now...

How to on the ssh part i found while going through gazillions searches on google: https://superuser.com/questions/1111591/how-can-i-ssh-into-bash-on-ubuntu-on-windows-10/1114162#1114162

Since doing this speed is nice(jumps between 35-40mb sec - source & target drive both ssds):

WSL File permissions and data is preserved perfectly when new files is introduced(same goes for data changes to exsisting files):

NOTE: Also WinSCP auto uploads file while editing. And you can setup folder syncronizasion with WinSCP(if you work from another drive but still want to test stuff from WSL and do everything practical from there etc.)

WinSCP also has an explorer interface(if the standard ftp look is off-putting): https://winscp.net/eng/docs/ui_explorer

So idea to devs for solving this without comprimising the linux filesystem metadata make a tool that let one configure and "mount" folders through ssh(either as mapped drives or folders in your windows "User" folder). That way one can configure diferent folders to mount and moving files around would be done the right way.

[BJobsis]

I tested again to see if my originally reported problem (When I try to create a file from Windows10 (using Sublime Text, etc) into the Linux Ubuntu subsystem the file is created, but afterwards the file can't be seen from within the subsystem.) still exists. I hoped that with the Creators Update this was solved. So for development I'll stay on OSX for a while.

[pharmmd-rich]

That will likely never be fixed. IMO it is not an issue when /mnt/drive is available.

[benhillis]

@pharmmd-rich - I wouldn't be so certain of that. It's high on our list of things we'd like to fix, but it's a difficult problem.

[pharmmd-rich]

@benhillis which part? Is work actually being done in this area? I guess I just don't understand the use case..

[benhillis]

@pharmmd-rich - It's by far the most common issue that pops up from people using WSL. There are some other implications outside of WSL as well.

[pharmmd-rich]

@benhillis wouldn't a better solution be to move to a .VHD format and use samba to expose the filesystem to Windows via network sharing? If users want to modify/set perms they can do it in bash. The reason I ask is because there are 2 very large issues that need to be addressed.

1) I don't know how WSL itself handles filesystem stuff, but I know the current implementation is quite slow. I am sure optimizations can be made, but will it ever be faster than using Linux in a VM?

2) Depending on how it's implemented, third party tools, older versions of Windows, etc. can still corrupt the installation. I've seen stuff like logoff scripts that copy the entire user folder to a Windows 2000 server and then copy it back on logon.

Everything understands the Windows file system, but if you start throwing Windows 10 specific extensions for use with WSL in there without a viable way of falling back, things could get nasty. Of course, I don't know how you guys are planning to implement this, so I won't comment further in this issue. I just hope you guys have extended use cases covered. I have no problem with the feature itself, (not that my opinion or acceptance is wanted and/or needed anyway ;))

[therealkenc]

I wouldn't be so certain of that. It's high on our list of things we'd like to fix, but it's a difficult problem.

Higher or lower than CUDA? [Don't answer that. Inside voice.]

I've made some progress (of the time-sink variety) since I last posted in this ticket, but the result never manages to reach feature parity with what we already have. Maybe I'll take another run once it starts raining here again. Happy it is still on the team's burner.