microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.42k stars 823 forks source link

WSL 2 should automatically release disk space back to the host OS #4699

Open dsmaher opened 4 years ago

dsmaher commented 4 years ago

Is your feature request related to a problem? Please describe. I occasionally have workloads that consume large amounts of disk space for temporary use in /tmp. For example, transcoding large video files or processing large datasets exported from a database. Since WSL2 stores its filesystem on a .vhdx, that file grows when the dataset is processed and never releases that space. Also, it doesn't appear to re-use released disk space in the .vhdx when files are deleted... it seems to prefer to grow the file rather than re-use existing empty space. (I haven't explicitly tested that theory, but my workload deletes temporary files as they are used and I never had enough of them existing simultaneously to reach 250GB, but the size of the .vhdx eventually expanded to that size.

When I'm done with that workload, I have to:

wsl --shutdown
optimize-vhd -Path .\ext4.vhdx -Mode full

This is annoying since it basically means that my system backups include a huge .vhdx file that is mostly empty. Also, until about 2 weeks ago, it was consuming the entire remainder of my system disk. I had only 50GB free, which was enough to handle the workload, but since it continued to grow even after files were removed, that didn't matter. (I figured it was time to upgrade the disk size anyway)

Describe the solution you'd like Automatic compaction of the .vhdx, or a way to do so while WSL2 is still running so that I can schedule it for frequent cleanups.

Describe alternatives you've considered

qwertynik commented 2 years ago

For Windows 10 Home (alternative Optimize-VHD cmdlet):

wsl --shutdown
diskpart
# open window Diskpart
select vdisk file="C:\WSL-Distros\…\ext4.vhdx"
attach vdisk readonly
compact vdisk
detach vdisk
exit

Thanks to @davidwin for the tip #4699 (comment).

@merkuriy Any idea if this deletes any useful files from WSL2? Is running this similar in effect to cleaning up the Recycle Bin on Windows?

Looking forward to your response and reclaiming some 'space' back 😄

Thanks!

diegoclair commented 2 years ago

For Windows 10 Home (alternative Optimize-VHD cmdlet):

wsl --shutdown
diskpart
# open window Diskpart
select vdisk file="C:\WSL-Distros\…\ext4.vhdx"
attach vdisk readonly
compact vdisk
detach vdisk
exit

Thanks to @davidwin for the tip #4699 (comment).

@merkuriy Any idea if this deletes any useful files from WSL2? Is running this similar in effect to cleaning up the Recycle Bin on Windows?

Looking forward to your response and reclaiming some 'space' back 😄

Thanks!

@qwertynik I tried it now and it looks that do not delete any useful files. And it free some space!

qwertynik commented 2 years ago

For Windows 10 Home (alternative Optimize-VHD cmdlet):

wsl --shutdown
diskpart
# open window Diskpart
select vdisk file="C:\WSL-Distros\…\ext4.vhdx"
attach vdisk readonly
compact vdisk
detach vdisk
exit

Thanks to @davidwin for the tip #4699 (comment).

@merkuriy Any idea if this deletes any useful files from WSL2? Is running this similar in effect to cleaning up the Recycle Bin on Windows? Looking forward to your response and reclaiming some 'space' back 😄 Thanks!

@qwertynik I tried it now and it looks that do not delete any useful files. And it free some space!

Thanks for confirming @diegoclair. That instills some confidence in the commands. Hope there are no bitter surprises awaiting :)

germangp088 commented 2 years ago

OK, ran:

image

To get rid of the docker images that had been accumulating. I have my hard drive back, and Docker is now pulling down images (good) but it's failing with:

ERROR: Service 'postgres' failed to build: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.65.1:53: read udp 192.168.65.3:39176->192.168.65.1:53: i/o timeout

Which I'm now troubleshooting.

Edit: since the IP addresses are local, poking around in Docker networking config and restarting docker fixed it.

Thanks! this worked for me after tried everything else for hours!

berlincount commented 2 years ago

What I absolutely like best is the way you can do it on VirtualBox: the disk image is a sparse file (i.e. with holes in it) like the vhdx file, but when you have your linux configured correctly (i.e. LVM, block devices, etc) the simulated SSD forwards the deletes and trims from ext4 to the underlying blockdevice through to VirtualBox, which re-punches the hole into the underlying disk image, releasing the space on the host operating system. I consider this the optimum solution, as it releases the space at runtime already.

mahdibx commented 2 years ago

some programs write files constantly then clear them, yet wsl expands the disk continuously anyway. It should first take available space in the virtual disk before trying to expand it.

philippe-granet commented 2 years ago

Is there a solution to compact disk without admin rights? diskpart or optimize-vhd commands needs admin rights and admin account is blocked on my Corporate computer.

svet-am commented 2 years ago

I got tired of trying to do this manually so I wrote a PowerShell script to do this more-or-less automatically. It's not perfect and I could use some help cleaning it up but it's here in case anyone else wants it: https://github.com/svet-am/wsl-vhd-compactor/blob/main/compact_vhds.ps1

It uses diskpart directly to get around the issue where optimize-vhd might not be available on a particular host.

qwertynik commented 2 years ago

I got tired of trying to do this manually so I wrote a PowerShell script to do this more-or-less automatically. It's not perfect and I could use some help cleaning it up but it's here in case anyone else wants it: https://github.com/svet-am/wsl-vhd-compactor/blob/main/compact_vhds.ps1

It uses diskpart directly to get around the issue where optimize-vhd might not be available on a particular host.

Thanks for sharing the script @svet-am. Any ideas if this deletes files on Docker? As per this comment, https://github.com/microsoft/WSL/issues/4699#issuecomment-574835154 working on the vhdx file led to important files being deleted.

mikemaccana commented 2 years ago

I got tired of trying to do this manually so I wrote a PowerShell script to do this more-or-less automatically. I

@svet-am a few of us have been working on https://github.com/mikemaccana/compact-wsl2-disk, you're welcome to contribute there if you like, or not, that's fine too.

svet-am commented 2 years ago

Thanks @mikemaccana - wasn't aware of your project when I wrote mine. I'll contribute over on yours as the need arises. Yours is a bit less brute force than mine anyway.

theAkito commented 2 years ago

The optimize-vhd -Path .\ext4.vhdx -Mode full workaround does not work anymore, on my computer. I am stuck with a 69GB ext4.vhdx on my NVMe SSD and I don't know what to do.

Chris2011 commented 2 years ago

The optimize-vhd -Path .\ext4.vhdx -Mode full workaround does not work anymore, on my computer. I am stuck with a 69GB ext4.vhdx on my NVMe SSD and I don't know what to do.

I moved the vdisk to an external disk. I followed these steps: http://nuts4.net/post/moving-wsl2-vhdx-file-to-a-different-location

svet-am commented 2 years ago

The optimize-vhd -Path .\ext4.vhdx -Mode full workaround does not work anymore, on my computer. I am stuck with a 69GB ext4.vhdx on my NVMe SSD and I don't know what to do.

I moved the vdisk to an external disk. I followed these steps: http://nuts4.net/post/moving-wsl2-vhdx-file-to-a-different-location

Did you try ring diskpart method rather than optimize-vhd? I just tried and the diskpart method still works for me even with very large (100GB+) VHD.

Chris2011 commented 2 years ago

I also had problems with both neither diskpart nor optimze-vhd worked for me. So I moved the file to my external SSD and I can live with that.

theAkito commented 2 years ago

Did you try ring diskpart method rather than optimize-vhd?

Yes, I tried the workaround for the workaround, but it showed the same symptoms. Compacting goes up to 10% instantly, then slowly progresses to around 20% and then instantly finishes with 100%. Then, I check the disk space and nothing changed, at all.

I moved the vdisk to an external disk. I followed these steps: http://nuts4.net/post/moving-wsl2-vhdx-file-to-a-different-location

The problem with that is, that the disk would keep growing infinitely. So, it does not solve any problem, except maybe my SSD dying quickly. I also have some spare space on hard drives, but I won't waste hundreds of GB, because Windows is too stupid to compact a native virtual disk.


When not even the workaround for the workaround works for some users, how does this not have a huge priority? How does the majority of the user base use Docker Desktop on Windows without pulling their hair out after a couple of months of accumulating space for no reason? Even if the workarounds work, this situation is awful.

The outcry about this issue is not adequately representing the urgency of this situation. I probably would need to purge Docker Desktop or just plainly force remove the vhdx, soon. Which would also be the case for anyone else, who cannot even get the workaround for the workaround to work...

sarim commented 2 years ago

I personally removed docker desktop long ago, after having to diskpart the disk a few times, also other reasons. Now I just run podman (rootless container). diskpart is still working for me. But you can try recreating the distro, wsl --export to export it, then wsl --import to import it again. This export / import makes the whole distro/container a tar archive, so it'll not take any extra disk space, I hope.

As for the main issue, I guess hyperv's commands / technology were not really made for ext4? In that case error/bug report should be reported to hyperv team I guess.

qwertynik commented 2 years ago

@craigloewen-msft Although there are many solutions suggested here, not certain how it would impact the distro - could there be any data loss, etc. Like me, there would be many who are not fully aware of the internals and would be hesitant to try out the solutions here.

Can the WSL team look into this and suggest an official way to resolve this issue?

theAkito commented 2 years ago

Although there are many solutions suggested here, not certain how it would impact the distro - could there be any data loss, etc.

I think someone mentioned possible data loss, somewhere in this long thread. Using one of the provided workarounds is definitely not super safe. Apply at your own risk.

Can the WSL team look into this and suggest an official way to resolve this issue?

That's what we are waiting for, since 2019. 😄

richyeiv commented 2 years ago

In my case the problems were related to disk space. When you export the .vhdx is removed and replaced by a .tar. On import and recreate the vhdx you need at least twice the space (vhdx + tar). At first it gave the 'unespecified error' message then when I tried to open the .tar it said the file was corrupted. In the end I moved the file to another drive with more space and imported in the same directory where the .tar was located: wsl --import Ubuntu .\ .\Ubuntu.tar It works for me.

mklueh commented 2 years ago

Having permission issues with "optimize-vhd -Path .\DockerDesktop.vhdx -Mode full" despite having launched PowerShell as admin

Zaffer commented 2 years ago

For WSL2 Ubuntu it is recommended to use Docker Desktop. If you read here they specifically mention VHD space concern:

"`If you have concerns about the size of the docker-desktop-data VHDX, or need to change it, take a look at the WSL tooling built into Windows."

Instructions here: https://docs.microsoft.com/en-us/windows/wsl/tutorials/wsl-containers

iuriguilherme commented 2 years ago

To all of you copying and pasting commands from github issues (yes, you!): STOP.

If you have a large, inefficient virtual hard drive that you need to optimize, will you please read the manual of the tool, it's available in the official website: https://docs.microsoft.com/en-us/powershell/module/hyper-v/optimize-vhd

What the OP is complaining about is the fact that with WSL the hard drives keep growing and no automatic effort is made to keep them trimmed. In order to perform any operations on those virtual hard drive files, you have to go through the checklist:

Be aware that this operation, if done incorrectly can lock you out of your virtual machine and if you're not an expert in fixing volumes, you may have to start over from scratch. Use the options -WhatIf to check what will happen and when you're sure it's the right command, run it with -Confirm so it asks you for confirmation before each operation.

You may want to consider if you really need the Full mode. In most circumstances, the Retrim is all you need. If that doesn't claim back space in the underlying NTFS partition, then the Quick mode will reclaim all unused space without seeking for blocks filled with zeroes. You would only use the Full option if you really need to compact the virtual hard drive as much as possible and you have already used UNIX tools inside the virtual machine to prepare and fill empty spaces with zeroes beforehand. After you do the initial trim of the disk using the windows tool, weekly management can be done with something easier such as the Pretrimmed option, which doesn't require you to do a wsl.exe --shutdown first.

So all things considered, the actual command in this example would be:

  1. Check if everything will work as expected
Optimize-VHD -Path "C:\Users\user\AppData\Local\Packages\TheDebianProject.DebianGNULinux_49vf1zgz67zfv\LocalState\ext4.vhdx" -Mode Retrim -Confirm -Whatif
  1. Actually running it
wsl --shutdown
Optimize-VHD -Path "C:\Users\user\AppData\Local\Packages\TheDebianProject.DebianGNULinux_49vf1zgz67zfv\LocalState\ext4.vhdx" -Mode Retrim -Confirm
  1. Seeing it doesn't do what you want and come back to github to complain about it
tomas-pritrsky commented 2 years ago

bump. I see no reason why docker should consume disk space to infinity.

dbjungle commented 2 years ago

Is there a process to manually migrate the WSL2 VHDX disk to a smaller VHDX? I'm not comfortable with the default max size of 250GB and I'd like to change it to something smaller even if there is some effort involved.

andymcblane commented 2 years ago

Also ran into this issue, after seeing 0 bytes free on C:\

arnavmehta7 commented 2 years ago
detach vdisk

Thankyou!

zolbayars commented 2 years ago

I ran

Optimize-VHD -Path "C:\Users\...\LocalState\ext4.vhdx"  -Mode Quick -Confirm

and it's stuck at 97% (even after 5 hours). Shall I just wait? or is there any other thing I need to do? 🤔

image

okibcn commented 2 years ago

There is a huge problem with the ever-growing size of the vhdx image. So far the best results I've found are:

  1. Inside your WSL2 distro run sudo fstrim / -v.
  2. In powershell run wsl --shutdown
  3. Also in powershell run Optimize-VHD "C:\Path/to/ext4.vhdx" -Mode Full

However, there are huge differences between the actual size of the filesystem reported by df or ncdu / -x inside WSL2 that reports barely 900 MB and the actual size of the VHDX file of more than 2300 MB.

I also tried the traditional way of filling the whole filesystem with zeroes and cleaning it again, but it didn't work. This is by running dd if=/dev/zero of=./junk ; sudo sync ; rm -f ./junk in WSL2

kevlarr commented 2 years ago

I ran

Optimize-VHD -Path "C:\Users\...\LocalState\ext4.vhdx"  -Mode Quick -Confirm

and it's stuck at 97% (even after 5 hours). Shall I just wait? or is there any other thing I need to do? 🤔

image

I ran into something similar (or identical?) - stuck at 10% for probably 5 hours, and it's always been fairly quick before. I finally just hit CTRL-C and it immediately printed a message basically saying "done and successfully compacted", and lo and behold I had all that disk space reclaimed in Windows.

🤷

vbrozik commented 2 years ago

I finally just hit CTRL-C and it immediately printed a message basically saying "done and successfully compacted", and lo and behold I had all that disk space reclaimed in Windows.

Thanks for sharing your experience. Did you check the file system integrity (fsck) after that?

kevlarr commented 2 years ago

@vbrozik No, but I'd like to - are there good instructions somewhere for doing that? I'm not super experienced with linux in general, and I'm not particularly sure how to use that tool while it's mounted and running.

vbrozik commented 2 years ago

@kevlarr The quick dirty way is to do the following:

$ fsck -nvf "$(df --output=source / | tail -n1)"
fsck from util-linux 2.37.2
e2fsck 1.46.5 (30-Dec-2021)
Warning!  /dev/sdd is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (64896683, counted=64896612).
Fix? no

Free inodes count wrong (16671680, counted=16671653).
Fix? no
...

Free blocks count wrong and Free inodes count wrong are normal when checking a mounted file system.


The proper way would be to run a different machine, attach the VHD disk image there as a block device and run fsck there.

For that you can use for example a Hyper-V machine or access the VHD disk image content as a raw disk image from another WSL2 distro using affuse - see: How to mount a virtual hard disk?, https://github.com/sshock/AFFLIBv3

okibcn commented 2 years ago

The only fail-safe way to really compact a WSL distro to its minimal size is by running this export-import PowerShell script:

$wsl_distro = "Arch"      # <<<<<<<<<<<<<<<<  YOUR DISTRO NAME HERE

$wsl_path = (Get-ChildItem HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss | ForEach-Object {Get-ItemProperty $_.PSPath} | Where-Object { $_.DistributionName -in "$wsl_distro" }).BasePath.Substring(4)
$tmp_folder = "$Env:TEMP\wslclean"
mkdir "$tmp_folder"
wsl --shutdown
cmd /c "wsl --export ""$wsl_distro"" - | wsl --import wslclean ""$tmp_folder"" -" 
wsl --shutdown
Move-Item "$tmp_folder/ext4.vhdx" "$wsl_path" -Force
wsl --unregister wslclean
rm -Recurse -Force "$tmp_folder"

I have tried other solutions like DISKPART or Optimize-VHD described in previous posts and they can't reduce the size to its minimum in most cases, not even after running a sudo fstrim / or dd if=/dev/zero of=./junk ; sudo sync ; rm -f junk inside the distro before optimizing the VHDX.

I have tried simpler solutions such as

$wsl_distro = "Arch"      # <<<<<<<<<<<<<<<<  YOUR DISTRO NAME HERE
$wsl_path = (Get-ChildItem HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss | ForEach-Object {Get-ItemProperty $_.PSPath} | Where-Object { $_.DistributionName -in "$wsl_distro" }).BasePath.Substring(4)
wsl --shutdown
wsl --export "$wsl_distro" "$wsl_path/compacted.vhdx" --vhd
Move-Item "$wsl_path/compacted.vhdx" "$wsl_path/ext4.vhdx" -Force

But this option did not always end as small as the solution proposed above in this message.

kevlarr commented 2 years ago

Thanks @vbrozik, I'll give that a try

vbrozik commented 2 years ago

@okibcn Thank you for testing another method.

The only fail-safe way to really compact a WSL distro to its minimal size

Did you see a big difference in the compaction (this tar-untar method vs Optimize-VHD)?

wsl --shutdown

I do not think you have to terminate all your distributions and possibly docker. IMHO wsl --terminate $wsl_distro would be enough.

cmd /c "wsl --export ""$wsl_distro"" - | wsl --import wslclean ""$tmp_folder"" -"

okibcn commented 2 years ago

@vbrozik, the export-import method achieves the minimum size, by far. Optimize-VHD can't reduce size on many occasions. In some of my tests, the export-import method sometimes compacts the VHDX to about 20% the size of what Optimize-VHD or DISKPART could do. And in all tests, the export-import method was always the winner.

Regarding the wsl --shutdown, I haven't tried to terminate only one instance. You can try. In any case, if you have more than one distro, you can compact them all. Actually, I was thinking of creating a function to add to the $PROFILE. The function would compact all distros installed. It would be very easy to generalize the script into a function.

Regarding using cmd for the core of the export-import method, it is a requirement for the binary pipes. PowerShell can't pipe binary streams as it can pipe only objects. There are ways to emulate sh pipes in PowerShell, but they are more complex. Since pipes work perfectly inside the cmd interpreter, there is no need to add any complexity, you know, the KISS rule.

The export-import keeps all the attributes, and the resulting filesystem is the same, exactly the same, but fully compacted.

Querela commented 2 years ago

Is there a way to export/import using an external device? If the vhdx file has grown over e.g. 150GB on a 256GB SSD (laptop) it failed to export to a connected storage device (> 250GB, USB) due to insufficient space. It seems to first cache it locally but I could not find where or how to disable this. So you probably sometimes can't really use the export/import trick in those cases? This also fails for doing a simple backup... So best to periodically reduce the size before it is too late...

okibcn commented 2 years ago

@Querela, the import-export method I use doesn't require a temporary file, but it requires extra space as the new VHDX file is created along with the oversized one. You can use an external drive if you wish. Have you tried a piped export? Something like this PowerShell instruction:

cmd /c "wsl  --export DISTRO_NAME - | 7z a -si  ""Z:\External\path\dump.tar.7z"" "

This way you end up using a small portion of disk space, as your distro dump will end up compressed without temporal files in the drive and you can move it around to another computer if you like.

After checking your compressed filesystem, you can safely remove your oversized distro and release the space by doing a:

wsl --unregister DISTRO_NAME

Now, you can import the dump again by doing:

cmd /c "7z x -so -bsp2 ""Z:\External\path\dump.tar.7z""  | wsl  --import DISTRO_NAME ""C:\path\to\distro"" -"

Note the import function sometimes get stuck. Don't panic! If that happens, just press Ctrl+C and run the import instruction again.

The actual size of the data in your distro is something you can see by typing df / inside the distro. You should end up with a compact VHDX for that distro slightly larger than that value. Regarding the temporary compressed file, in my tests, 7z creates a dump.tar.7z of about 20% of the size of the data contained in your distro, however, your mileage may vary depending on the kind of data your distro contains.

I use 7z, but any other compression app would work as long as it allows piping standard input and output.

Querela commented 2 years ago

Hmm. I'm not sure whether I piped it or used a path parameter. I will need to check this later but if I remember correctly I tried to used piping and compression.

qwertynik commented 2 years ago

Microsoft has a tutorial to expand the size of WSL2's VHD here. There's something about memory reclaim too here

Is it that most of us here are not able to get to Microsoft's tutorial on how to shrink the VHD size or there's none yet?

tomas-pritrsky commented 2 years ago

Is it that most of us here are not able to get to Microsoft's tutorial on how to shrink the VHD size or there's none yet?

No, this discussion went a little off topic. The title of this issue says "WSL 2 should automatically release disk..." So if I dare bring it back to the topic at hand - how to achieve this automatically? Can Microsoft fix it so that it is done automatically please?

qwertynik commented 2 years ago

No, this discussion went a little off topic.

Yes, it did. However, before we get there, knowing a safe and foolproof way to reclaim the hard disk manually would help.

There are some methods recommended earlier, like diskpart, and import-export - however not certain about the risks involved in using them. Optimize-VHD is a related cmdlet but it isn't available in the home edition though.

qwertynik commented 2 years ago

Seems unusual to continue to wait on something as fundamental as this in paid software.

Is there any other platform to attract the Microsoft team's attention to this? Had tweeted here but there isn't a response yet. If there's no other way, collectively liking the tweet could help. Or, someone with a larger following could Tweet something similar.

tusharsnx commented 1 year ago

@benhillis @craigloewen-msft Is there a possibility to have this feature within wsl ? with something like:

$ wsl --optimize-space <distro-name>

which would run compact operation on distro's ext4.vhdx.

Deoptim commented 1 year ago

To optimize disk space, you need to defragment the image and then write zeros (0x00) to the empty space - this is the format of the .vhdx files (not WSL2).

I wrote a small batch script for this process:

@ECHO OFF
setlocal enabledelayedexpansion

SET "mountpoint=/"
SET "vhdx_file=%LOCALAPPDATA%\Packages\CanonicalGroupLimited.Ubuntu20.04LTS_79rhkp1fndgsc\LocalState\ext4.vhdx"

:: BatchGotAdmin
:-------------------------------------
REM  --> Check for permissions
    IF "%PROCESSOR_ARCHITECTURE%" EQU "amd64" (
>nul 2>&1 "%SYSTEMROOT%\SysWOW64\cacls.exe" "%SYSTEMROOT%\SysWOW64\config\system"
) ELSE (
>nul 2>&1 "%SYSTEMROOT%\system32\cacls.exe" "%SYSTEMROOT%\system32\config\system"
)

REM --> If error flag set, we do not have admin.
if '%errorlevel%' NEQ '0' (
    echo Requesting administrative privileges...
    goto UACPrompt
) else ( goto gotAdmin )

:UACPrompt
    echo Set UAC = CreateObject^("Shell.Application"^) > "%temp%\getadmin.vbs"
    set params= %*
    echo UAC.ShellExecute "cmd.exe", "/c ""%~s0"" %params:"=""%", "", "runas", 1 >> "%temp%\getadmin.vbs"

    "%temp%\getadmin.vbs"
    del "%temp%\getadmin.vbs"
    exit /B

:gotAdmin
    pushd "%CD%"
    CD /D "%~dp0"
:-------------------------------------- 

echo (
for %%A in ("%vhdx_file%") do (
    set "before_size=%%~zA"
)

:loop
wsl.exe --list --running >nul 2>&1
if '%errorlevel%' EQU '-1' (
    echo Wating for running WSL2... ^(the user must run WSL2 himself^)
    timeout /t 5 /nobreak >nul 2>&1
    goto loop
) else if '%errorlevel%' EQU '0' ( goto continue )

:continue
set /a count=3
:loop2
wsl -u root sudo findmnt %mountpoint% -n -v -o SOURCE
if '%errorlevel%' EQU '1' (
    if %count% GEQ 0 (
        echo Wating for mountpoint...
        timeout /t 5 /nobreak >nul 2>&1
        set /a count=%count%-1
        goto loop2
    ) else (
        echo ERROR! The wait for mountpoint has expired!
        pause
        exit
    )
) else if '%errorlevel%' EQU '0' ( goto continue2 )

:continue2
FOR /F "delims=" %%X IN ('wsl -u root sudo findmnt %mountpoint% -n -v -o SOURCE') DO (
  SET "device=%%X"
)
wsl -u root sudo e4defrag %device%
wsl -u root sudo mount -o remount,ro %device%
rem wsl -u root sudo zerofree -v %device%
wsl -u root sudo e2fsck -f -n -E discard %device%
wsl -u root sudo mount -o remount,rw %device%
wsl --shutdown

:loop3
wsl.exe --list --running >nul 2>&1
if '%errorlevel%' EQU '0' (
    echo Wating for exit from WSL2...
    timeout /t 5 /nobreak >nul 2>&1
    goto loop3
) else if '%errorlevel%' EQU '-1' ( goto continue3 )

:continue3
:loop4
2>nul (
    >>%vhdx_file% echo off
) && (
    echo File is unlocked.
    echo Continue to diskpart.
) || (
    echo The file %vhdx_file% is still locked!
    echo Wating when WSL2 is unlock the file...
    timeout /t 5 /nobreak >nul 2>&1
    goto loop4
)

SET DiskPartScript="%TEMP%DiskpartScript.txt"
ECHO select vdisk file="%vhdx_file%" > %DiskPartScript%
ECHO attach vdisk readonly >> %DiskPartScript%
ECHO compact vdisk >> %DiskPartScript%
ECHO detach vdisk >> %DiskPartScript%
ECHO exit >> %DiskPartScript%
diskpart /s %DiskPartScript%

echo (
for %%A in ("%vhdx_file%") do (
    set "after_size=%%~zA"
)

echo File %vhdx_file% has a size of:
echo Before optimizing: !before_size! bytes
echo After optimizing: !after_size! bytes

ENDLOCAL

pause

(you can also use the zerofree utility to delete empty space (instead of the e2fsck utility) - for this erase "rem")

The path on the vhdx container in the vhdx_file variable is may different for you (it depends on your distribution), change it to your own.

This script is also compatible with AlpineWSL:

SET "mountpoint=/mnt/wsl/Alpine"
SET "vhdx_file=d:\Alpine\ext4.vhdx"
okibcn commented 1 year ago

@Deoptim I see several problems with your approach:

  1. zerofree works in debian/ubuntu, but it is not compatible with ext4 in some distros such as Arch, so it is not a general solution.
  2. The beauty of ext4 is that you don't need to defrag, so e4defrag is not going to help a lot and shares the same cross-distro compatibility problems described for zerofree.
  3. Using the diskpart method is ineffective in too many cases. And, even in some cases, it ends up generating larger images.
  4. It requires elevated privileges, and some users use WSL as a user in machines without administrator rights.
  5. You need to know the name and location of the ext4.vhdx file, so it works for only one distro at a time.

I ended up with this simple function without any of those problems.

WSLCOMPACT

INSTALL

EDIT: I made a full script with additional functionality such as test mode displaying the estimated target size without compacting. I need GitHub stars ⭐⭐⭐⭐... for Scoop organization to accept it in its repository of apps. You find it at: https://github.com/okibcn/wslcompact

However you can have the lite version. Just edit the default profile with notepad $PROFILE in powershell, and add the following function anywhere:

function wslcompact($distro){
  $tmp_folder = "$Env:TEMP\wslcompact"
  mkdir "$tmp_folder" -ErrorAction SilentlyContinue | Out-Null
  Get-ChildItem HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss\`{* | ForEach-Object {
    $wsl_=Get-ItemProperty $_.PSPath
    $wsl_distro=$wsl_.DistributionName
    $wsl_path=if ($wsl_.BasePath.StartsWith('\\')) {$wsl_.BasePath.Substring(4)} else {$wsl_.BasePath}
    if ( !$distro -or ($distro -eq $wsl_distro) ){
      echo "Creating optimized $wsl_distro image."
      $size1 = (Get-Item -Path "$wsl_path\ext4.vhdx").Length/1MB
      wsl --shutdown
      cmd /c "wsl --export ""$wsl_distro"" - | wsl --import wslclean ""$tmp_folder"" -" 
      wsl --shutdown
      Move-Item "$tmp_folder/ext4.vhdx" "$wsl_path" -Force
      wsl --unregister wslclean | Out-Null
      $size2 = (Get-Item -Path "$wsl_path\ext4.vhdx").Length/1MB
      echo "$wsl_distro image file: $wsl_path\ext4.vhdx"
      echo "Compacted from $size1 MB to $size2 MB"
    }
  }
  Remove-Item -Recurse -Force "$tmp_folder"
}

Close the PowerShell terminal and reopen it again to ensure the updated profile is active, or just type . $PROFILE.

USAGE

The usage is straightforward. Calling wslcompact without arguments compacts all the WSL images. Or you can compact a single one passing its name as an argument, for instance wslcompact Ubuntu. It ensures a minimal size and you end up with contiguous files for faster access in old HD-based systems. The list of names of the installed distros is accessible by typing wsl -l in any powershell terminal.

if your C: drive doesn't have enough temporal free space, just change the TEMP folder before calling the function. So, instead of a simple wslcompact, just do:

$env:TEMP="Z:\your temp\folder"
wslcompact

The new TEMP folder will be active only for that PowerShell terminal session, so no problem at all for the rest of the system and it won't leave garbage.

theAkito commented 1 year ago

I ended up with this simple function without any of those problems.

I want to try out your procedure, but I already see a huge issue.

$tmp_folder = "$Env:TEMP\wslclean"
mkdir "$tmp_folder" -ErrorAction SilentlyContinue | Out-Null

This is a no-no.

The function should probably take an optional temporary folder path as an argument, as well.

okibcn commented 1 year ago

@theAkito if your temp folder is a no-no then just change the TEMP folder before calling the function. So, instead of a simple wslcompact, just do:

$env:TEMP="Z:\your temp\folder"
wslcompact

The new TEMP folder will be active only for that PowerShell terminal session, so no problem at all for the rest of the system.

theAkito commented 1 year ago

@okibcn

So, the first try did not succeed. Your function assumes, that every path starts with \\?\. It should check, whether this is the case, before cutting the prefix.

Second try succeeded a bit.

The compacted VHD was unmovable and, since it did not get moved, the wslclean folder would not be removed. The reason for the inability to move: The process cannot access the file because it is being used by another process. Even though, WSL was shut down and not booted during the compacting process.

After manually moving the file, Windows Terminal did not find the distribution. So, I opened it directly and it went to installation mode, essentially deleting all data.

Did you try the function on one of your installations?