moving Linux filesystem

KiaraGrouwstra commented 8 years ago

I figured I was about to install a bunch of things on my shiny new Linux subsystem, then realized my SSD would soon run out of disk space at that rate. Hoping I could improvise a solution, I tried moving the folder C:\Users\USER\AppData\Local\lxss to my D: drive and creating a (hard) symlink to it from the original location. This backfired though, as I was no longer able to boot bash afterwards (nor after moving it back or disabling/re-enabling the feature), which resulted in either no response (if opened from the start menu) or with Error: 0x8007001f (if called from a command prompt). I'd be glad to know if anyone has successfully managed to move their Linux subsystem to save SSD space.

fragtion commented 8 years ago

Same thing happened to me. Tried with junction first, then symlink

dreyks commented 8 years ago

This would be really nice, as I generally keep my system disk not so big

lidangzzz commented 8 years ago

Same issue. Hope it could move to disk D totally.

fpqc commented 8 years ago

Since directly editing the lxss user home from Windows software is not supported right now, why not have the lxss filesystem (outside of the /mnt stuff) just be a drive image, similar to the option in RDS in Windows Server 2012 R2 to make user home directories mountable images? That would cure the trouble of being able to move the folder around (since you could just point bash.exe to the image), and also would dissuade users from directly editing files in it (which breaks it).

I mean, users could even do this now by creating the \appdata\local\lxss folder, mounting a vhd to it, and then installing bash.exe directly.

fpqc commented 8 years ago

I uninstalled via lxrun /uninstall /full /y but recreated the folder %localappdata%\lxss, created a 20GB NTFS-formatted lxss.vhdx in diskmgmt on my side hdd (W:), and mounted it to %localappdata%\lxss with diskpart's mount command, then reinstalled WSL via bash.exe. So far, it's working.

I have a task set up right now to mount the vhdx on boot via the taskschd.msc running diskpart together with a script as System user, should continue to work after a reboot.

The script: select vdisk file="W:\lxss.vhdx" attach vdisk `assign mount="C:\Users\fpqc\appdata\local\lxss"``

Also, another benefit of this is that you can easily back up your lxss install (if you are going to mess around and try something that might break it), simply by copying the vhdx.

KiaraGrouwstra commented 8 years ago

Great, thanks! Haven't tried yet but looks good. Marking as solved.

onomatopellan commented 8 years ago

Since build 14388 the vhdx method did stop working. :( I have been using an external bash.vhdx for days and attached to an empty %AppData%/Local/lxss folder WSL worked great. Now when running bash.exe it says Error: 0x80070003

It seems now it tests the lxss folder and if it wasn't created by lxrun.exe then refuses to run.

I hope someone finds another way to export the WSL filesystem outside of the system partition. Compiling a project like Cyanogenmod13.0 (needs +60Gb of disk space) is just impossible for me without it.

fpqc commented 8 years ago

ooh good catch man I'm not on 14388 yet but I hope they fix this.

benhillis commented 8 years ago

Does Cyanogenmod13.0 compile without issues on DriveFs? (/mnt/d/...)

onomatopellan commented 8 years ago

@benhillis On build 14372 there were some problems on repo sync. https://github.com/Microsoft/BashOnWindows/issues/124

I'm going to try now on 14388.

Edit: Same error as 14372. On VolFs repo sync worked well though.

fpqc commented 8 years ago

@benhillis eep my whole setup got nuked. I guess I will find a way to get my home folder and configs back >.>.

onomatopellan commented 8 years ago

I managed to move lxss folder to another partition in build 14390. It's not as elegant and flexible as the vhdx method but at least it works. First uninstall your current WSL with lxrun /uninstall /full. Before installing WSL again you need to move %LocalAppData% folder to another partition and then create in the AppData folder a junction pointing to the new location. mklink /J C:\Users\onoma\AppData\Local E:\Local After that lxrun /install will work like always.

%LocalAppData% is a folder always being used by the system so you need to move/copy it from "outside". I created another admin user and did it from there.

KiaraGrouwstra commented 8 years ago

I haven't tested onomatopellan's new workaround yet, but I'd be inclined to consider this an open issue until Microsoft would start considering and supporting this use-case. Not every SSD-owner will have found this thread. :(

onomatopellan commented 8 years ago

Yes, we need an option to move or install lxss folder in a desired location.

My method is working but it killed Edge browser. Every time I open Edge it crashes because it depends on LocalAppData folder. Also didn't survive to an upgrade to a newer build. Next step will be trying moving the entire AppData folder instead.

I'm trying to understand how lxss folder works. It can't be a junction anymore (thus VHD method wont' work) and only lxrun.exe can create it. Folder can't be copied or moved but can be renamed and like this I have various lxss folders (lxss.xenial, lxss.trusty, lxss.base) with different environments. To change from trusty to xenial I just have to rename lxss.xenial to lxss and it works. So the special attributes of WSL files are conserved in their original lxss folders. And that's what I can't understand. What makes this folder special? There are no reparse points, its attributes are just hidden and system, there aren't alternate data streams either... so where is the special info stored?

benhillis commented 8 years ago

I feel your pain as a 100 GB SSD user. I've kicked of a thread with some of the developers to think about how we can support this Post Windows 10 Anniversary Update.

Just so I have the full story, is the install location the problem or would creating a separate VolFs mount on a different volume serve your purpose? Apt packages etc would still end up on your main OS volume but you could use all the features of VolFs on a larger drive.

KiaraGrouwstra commented 8 years ago

Hm. I think I could definitely fill some GB just from installing things, so going by that, the ability to allow having the whole thing off the SSD would be nice. I dunno what VoIF stands for though!

onomatopellan commented 8 years ago

Great to hear you are considering this. I'd vote for an entire different install location. And if it was possible to install on a VHDX file it would be the best. Being able to backup an entire WSL environment so easily has been the best feature until build 14388 was released.

All that said as second option mounting VolFs folders on external partitions sounds sweet.

@tycho01 VolFs folders are the / and /home type folders and have linux-like attributes. DrvFs are the /mnt/* folders.

fpqc commented 8 years ago

@benhillis Actually, the earlier workaround solution of installing on a vhdx was working really well for me (and apparently a bunch of other people). Not really sure how it broke, but what would be ideal is to fix things so we can do that again (maybe automate it?) as well as allow us to create other vhdx files "formatted" for VolFS use that can be mounted.

rayncc commented 8 years ago

I'll vote for this option. The lxss is killing my disk space :(

zidongtuili commented 8 years ago

@onomatopellan

I might be wrong, but maybe you can consider just create lxss folder under appdata\Local and only link this folder to another directory, so that Edge won't crash? I'm not sure if you link the whole local folder because the bash system requires, or you didn't have the lxss folder before creating the subsystem? (Sorry I just realised you meant that lxss can't be linked? I just tried and it said cannot create link when file already exists)

Although vhdx is more elegant, mklink seems to be faster right? My poor table has very slow disk i/o (I'm using a sd card, which is slow and sometimes windows just can't detect it, maybe someone can help me make it stable), and I do hope I can have a faster way.

After reading this post, I'm considering moving Office 365 (or maybe the whole program files) :) No idea why Edge crashed actually, I thought mklink worked the same way as linux and Edge should put its files wherever the address is, regardless the actual location.

onomatopellan commented 8 years ago

@zidongtuili Exactly, lxss can't be a link anymore. That's why I tried moving and linking Local folder. I think Edge crashed because the method I used to copy the LocalAppData folder wasn't the best. I changed the way I copy the "Local" folder now (I use robocopy after booting to command prompt) and I had better luck. Edge works like it should and WSL installs without problem. Now I only need to know if the link will survive to a new build install. I'll try with next Insider build.

utick commented 8 years ago

@onomatopellan I tried your solution, use another admin user and robocopy the Local folder to another parition, then made a junction link. Everything seems ok after reboot and I installed WSL successfully. But one day later the system goes wrong. All of the ump apps disapeared and the setting app crashed. The command i used:

robocopy C:\Users\XXX\AppData\Local D:\Local /e /xj /copyall
mklink /j "C:\Users\XXX\AppData\Local" "D:\Local"

onomatopellan commented 8 years ago

@utick Is that user a Local account or a Microsoft Account? It's weird everything did work and suddenly didn't. Sounds like if Windows changed the drive letter where you stored the Local folder. Is that a fixed partition or a removable one? It also seems you used the old method, creating another user and copying from there in desktop. That method gave me trouble so now I use native Command Prompt.

Create a new admin local account for testing. Let's call it Test for example.
Login to Test and see which drive will store the Local folder. In my case is E:
Now restart to Advanced Startup (Shutdown button -> Restart with SHIFT key pressed)
Select Troubleshoot -> Advanced Options -> Command Prompt.
Login to Test and a command prompt window will appear. In this environment drive letters could have changed so we need to know which one is the windows partition and which one is the E: where we want to copy Local folder.
Type diskpart and then list volume In my case the windows partition is D: and the other partition here is F:
Type exit to exit from Diskpart. After this type:
mkdir F:\Local
cd D:\Users\Test\AppData
robocopy Local F:\Local /E /COPYALL Make sure there are no errors.
rename Local Local.old
mklink /J Local E:\Local This is important, the symlink needs to point to the drive letter you saw on the Desktop, nor the one you see on command prompt.
exit now Continue Windows 10 restart and login to Test and everything should work like always.

EDIT: I forgot something important. After this trick lxrun.exe cant create a new user because it can't access Temp folder. Open an admin CMD and type SET TEMP=E:\Local\Temp pointing to the new Temp location. After that lxrun /install to begin the bash install.

fmckeogh commented 8 years ago

I can confirm that on build 14393, creating a VHDX for the entire AppData/Local folder and mounting on boot works fine without any issues, it might be worth trying for anyone that encounters issues with @onomatopellan 's method.

fpqc commented 8 years ago

Great @chocol4te! Are you sure that works again?

fmckeogh commented 8 years ago

@fpqc Moving the lxss folder itself did not work, but moving one directory up did the trick. Unfortunately I can only login as root, but I doubt security is really a priority this early in the feature. A rough guide for any readers would be:

Completely uninstall the subsystem with lxrun.
Create another admin account, fully log out of your current account and sign in to your new account.
Navigate to the original AppData folder in your normal account folder, rename "Local" to "LocalOld", and create a new empty "Local" folder.
Open Disk Management and create a new virtual disk on your target drive. Select VHDX, mount in the Local folder you just made, format as NTFS. Once done, if it has not mounted automatically, select Attach VHD.
Copy all files from the old folder to your new mount point, click retry for any prompts for administrator, and skip for any other prompt (I have not experienced any issues doing this, but it would be wise to check with someone more experienced.).
Create two files, I recommend in the same directory as your VHDX file. I chose "local" as the name for both. Create a .txt file with the following contents:

select vdisk file="directory_to_vhdx"
attach vdisk
assign mount="C:\Users\yourusername\AppData\local"
exit

and a .ps1 file with this contents:

diskpart /s local.txt
exit

Open Task Scheduler and click Create Task. Name it whatever you want, but change the security options to be run by SYSTEM. At a trigger to run "At Startup". In actions select "Start a program" and e enter "PowerShell" as the program/script. Add the following as the arguments:

.\local.ps1

and "start in" the directory of your ps1, txt and VHDX file.

Reinstall the subsystem as normal.

lodo1995 commented 8 years ago

Nice workarounds. An lxrun option to set the Linux root dir would be much better, though. I hope Microsoft will do something on this.

qingxbl commented 8 years ago

Maybe I will bring a good news today. Lxss will run happily by mounting a vhdx to lxss and creating some dirs.

dreyks commented 8 years ago

@qingxbl what build do you have?

qingxbl commented 8 years ago

It's much simpler than I thought. Here is a complete instruction.

@dreyks Microsoft Windows [Version 10.0.14393], RS1 release.

Installation

Uninstall if you want.
Delete or move %LocalAppData%\lxss.

Note: copy / robocopy %LocalAppData%\lxss to other directory is unsafe. But move %LocalAppData%\lxss is safe.
Create a junction. The target path must be end with \. Yes, a junction to wherever you want, a mounted vhdx, another directory in other driver, etc.
```
mklink /j "%LocalAppData%\lxss" "X:\wherever\you\want\"
```
Install Ubuntu On Windows as usual, lxrun /install or bash.
You will get an error after entering UNIX username. Relax, we will pass it.
Find out your magic char. See below.
Open a file explorer window and enter %LocalAppData%\lxss.
Rename directories cache, data, mnt, rootfs to ჻cache, ჻data, ჻mnt, ჻rootfs. Yes, prepend a ჻! I don't know why.
```
move cache ჻cache
move data ჻data
move mnt ჻mnt
move rootfs ჻rootfs
```
Create directories ჻home, ჻root, rootfs.
```
mkdir ჻home
mkdir ჻root
mkdir rootfs
```
Run lxrun /setdefaultuser. It will work this time.
Run bash. Enjoy.

The Magic Char

Run Process Monitor. Press OK button in Process Monitor Filter.
Run lxrun /setdefaultuser, type username and press Enter.
Stop capture in Process Monitor (the 3rd button in toolbar or Ctrl+E).
Ctrl+F, search for rootfs. And you will see a log like this. The char before rootfs is the magic char.
The magic char will change every reboot. You need to rename directories that begins with the magic char before using WSL after rebooting.

utick commented 8 years ago

@qingxbl Where did you got this method from?

qingxbl commented 8 years ago

@utick I figure out myself after doing some research using Process Monitor

dreyks commented 8 years ago

That's hell of a job =) Thanks a lot, will try today

zidongtuili commented 8 years ago

@qingxbl Great work. So this means that we need to keep folders with names like ჻ xxx? Just googled this is "georgian paragraph separator (U+10FB)". Do these folders also exist in original lxss folder (without using vhd)? I just wonder if there's an update and suddenly we have to something else, as I'm just doing this for fun and my tablet is very slow.

onomatopellan commented 8 years ago

@qingxbl Step 11 and 12 doesn't work. Error: 0x80070003. :( This is on build 14393.51.

fpqc commented 8 years ago

@onomatopellan maybe meant 14901?

onomatopellan commented 8 years ago

@fpqc 14905 is out. I'll try there too.

qingxbl commented 8 years ago

Sorry for my poor english.

@zidongtuili Yes, you must keep these ჻xxx folders AND xxx folders at the same time. These folders doesn't exist in original lxss folder. But if lxss is a Reparse Point, these ჻xxx folders will be used instead of xxx. xxx folders will be create if missing, so just keep it.

@onomatopellan I'm using 14393.51, too. After step 10, lxss must contains following folders.

჻cache
჻data
჻home
჻mnt
჻root
჻rootfs
cache
data
home
mnt
root
rootfs

჻rootfs is the real rootfs for ubuntu, rootfs is empty.

onomatopellan commented 8 years ago

@qingxbl It doesnt' work. It could be ჻ doesn't mean the same in my Windows. Which Windows Language are you using? If you type chcp on CMD what it shows?

edit: steps are OK

qingxbl commented 8 years ago

The magic char ჻ will CHANGE after reboot!!!! I think this is the reason why vhdx method is broken after build 14388. M$ uses an uninitialized variable to construct file path.

Run Process Monitor. Press OK button in Process Monitor Filter.
Run lxrun /setdefaultuser, type username and press Enter.
Stop capture in Process Monitor (the 3rd button in toolbar or Ctrl+E).
Ctrl+F, search for rootfs. And you will see a log like this. The char before rootfs is the magic char.

onomatopellan commented 8 years ago

@qingxbl Thanks a lot. Following your last instructions I discovered magic char was  and now following your steps it works perfectly. I didn't try to reboot yet though.

EDIT: Indeed, magic char changes every reboot. Just renaming the folders with the new magic char and works again for the entire session.

zidongtuili commented 8 years ago

Is it possible to move the /usr or /(root) to another directory. Even in this case we still have lxss in C:, we can have a very small folder. Sorry this may be a naive question, as I never did things like that.

onomatopellan commented 8 years ago

@zidongtuili Not now. In future builds we should be able to mount VolFs folders (/home, /usr...etc) in other partitions outside of C: .

zidongtuili commented 8 years ago

@onomatopellan I mean not from the windows side, but when using linux, there could be a way to change the default directory of /usr /bin etc.

onomatopellan commented 8 years ago

@zidongtuili It doesn't matter in that case because you can't move /usr folder outside of lxss folder. All the linux folders need to be inside lxss as it's a special folder and the only one with extended attributes.