Add ntfszapfree to optimize sparse images

[felfert]

This PR adds an invocation of ntfszapfree in the final cleanup stage just before running sysprep. zapfree writes zeroes to the unused parts of the disk. This in turn helps qemu-img to drastically reduce the size of the resulting qcow2 image.

In order to see the effect, after the initial run of the qcow2 image in a hypervisor (which inflates the image quite extensive due to running windows-update) you now can run an additional convert in qemu-img like this:

qemu-img convert -O qcow2 w2k16-cloudbase-init.qcow2 w2k16-cloudbase-init2.qcow2

The result is quite impressive. Here are the sizes before and after running that additional convert:

-rw-r--r--. 1 root root 11G Jun  2 23:18 w2k16-cloudbase-init2.qcow2
-rw-r--r--. 1 root root 29G Jun  2 23:14 w2k16-cloudbase-init.qcow2

These are uncompressed images. With compression enabled this gets down to ~5G!

Cheers -Fritz

[ader1990]

Hello,

I also had a try at this: https://github.com/cloudbase/windows-openstack-imaging-tools/pull/140/files but closed the PR due to external dependency, which I would like to avoid (sdelete or dd).

There is another naive way to easily achieve the disk cleanup using PowerShell: https://sqlserverpowershell.com/2015/01/14/create-a-zerod-out-file-with-powershell/ It would be great if you could give it a try.

Thank you, Adrian Vladu

[ader1990]

On another hand, to achieve the optimized, are you running with purge_updates=True, I suppose? There is also the defrag/trim optimizations done: https://github.com/cloudbase/windows-openstack-imaging-tools/blob/master/WinImageBuilder.psm1#L939

That would be impressive if the sdelete will indirectly optimize the image and if we could find a native way (PS) to achieve it.

[felfert]

What is the reason why you don't want an external dependency? There is no copyright problem, because it is not included but downloaded at runtime. IMHO, implementing it in PS/C# is "reinventing the wheel" and error-prone and therefore a bad idea.

-Fritz

[ader1990]

What is the reason why you don't want an external dependency? There is no copyright problem, because it is not included but downloaded at runtime. IMHO, implementing it in PS/C# is "reinventing the wheel" and error-prone and therefore a bad idea.

-Fritz

Fair enough, but what I am trying here is to avoid an extra mandatory dependency problem.

I will make a review on the patch in order to avoid this issue, and not oblige users to download the tool (people should be able to run the code offline if they do not want to install updates) and make the sdelete run configurable and turned off by default.

[felfert]

Well, the code in UnattendResources/Logon.ps1already skips execution if sdelete.exe is not available. This just leaves a config-option in WinImageBuilder.psm1 which is easy. I will add a commit to this PR ...

[felfert]

Ok, PR should now have all the requested changes

[ader1990]

I will give it a run to make sure it works on my env and it should be good to merge. Thank you for the contribution, much appreciated!

[ader1990]

@felfert Found a small unit test issue, you need to add sdelete_cleanup = true here: https://github.com/felfert/windows-openstack-imaging-tools/blob/add_sdelete/Tests/fake-config.ini#L13

[felfert]

Ah yes of course :-)

[alexpilotti]

@felfert first, thanks for contributing this (and nice to see you again!).

The SysInternals' license limits a lot of things unfortunately, for example if forbids to "transfer the software or this agreement to any third party". This implies that we cannot download and run the software on behalf of the user w/o them accepting explicitly the license (I'll refrain from saying what I think of that license).

IMO the idea is good, I'm actually surprised that unallocated areas are not zeroed, we just need to find another tool.

[alexpilotti]

To unblock this, let's expect the user to have sdelete already in the path and let's add a "zero_unused_volume_sectors" boolean option to control it.

We can then replace sdelete with a simple open source project that zeroes the unused parts of the volume (basically just FSCTL_GET_VOLUME_BITMAP plus writing the zeros).

[alexpilotti]

One more thing to evaluate: a simple call to Optimize-VHD after the updates, before the conversion.

[ader1990]

@felfert I created a patch that also runs Optimize-VHD on the image: https://github.com/cloudbase/windows-openstack-imaging-tools/pull/265. Can you give it a try and check if it solves the qcow2 issue?

[felfert]

Yes. But I'm travelling right now. Will try on Friday

Am 4. Juni 2019 16:15:45 MESZ schrieb Adrian Vladu notifications@github.com:

@felfert I created a patch that also runs Optimize-VHD on the image: https://github.com/cloudbase/windows-openstack-imaging-tools/pull/265. Can you give it a try and check if it solves the qcow2 issue?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/cloudbase/windows-openstack-imaging-tools/pull/261#issuecomment-498690659

-- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[AppVeyorBot]

:white_check_mark: Build windows-openstack-imaging-tools 1.0.48 completed (commit https://github.com/cloudbase/windows-openstack-imaging-tools/commit/be4efc3b2c by @felfert)

[AppVeyorBot]

:white_check_mark: Build windows-openstack-imaging-tools 1.0.49 completed (commit https://github.com/cloudbase/windows-openstack-imaging-tools/commit/ac79f9b16a by @felfert)

[AppVeyorBot]

:white_check_mark: Build windows-openstack-imaging-tools 1.0.50 completed (commit https://github.com/cloudbase/windows-openstack-imaging-tools/commit/ebe0614a1e by @felfert)

[felfert]

To unblock this, let's expect the user to have sdelete already in the path and let's add a "zero_unused_volume_sectors" boolean option to control it.

We can then replace sdelete with a simple open source project that zeroes the unused parts of the volume (basically just FSCTL_GET_VOLUME_BITMAP plus writing the zeros).

I now have created an open-source replacement for sdelete and replaced all references of sdelete with this new tool named zapfree. As requested, the config switch was renamed to zero_unused_volume_sectors

[AppVeyorBot]

:white_check_mark: Build windows-openstack-imaging-tools 1.0.51 completed (commit https://github.com/cloudbase/windows-openstack-imaging-tools/commit/29a00caa2e by @felfert)

[felfert]

@ader1990 @alexpilotti Did you see my commits, replacing of sdelete with open-source ntfszapfree? This should unblock this PR

[alexpilotti]

@felfert thanks for creating the seekers replacement, we’re currently doing some tests, as soon as they finish we can merge!

[AppVeyorBot]

:white_check_mark: Build windows-openstack-imaging-tools 1.0.55 completed (commit https://github.com/cloudbase/windows-openstack-imaging-tools/commit/90803347bb by @felfert)

[AppVeyorBot]

:white_check_mark: Build windows-openstack-imaging-tools 1.0.57 completed (commit https://github.com/cloudbase/windows-openstack-imaging-tools/commit/a426aefe51 by @felfert)

[AppVeyorBot]

:white_check_mark: Build windows-openstack-imaging-tools 1.0.59 completed (commit https://github.com/cloudbase/windows-openstack-imaging-tools/commit/02566929e9 by @felfert)

[ader1990]

Looks good, execution tested on a 2016 golden image:

6/14/2019 4:03:35 PM - Cloud image from golden image generation started.
6/14/2019 4:03:46 PM - Copy Unattend Resources: Windows Server 2012 R2 SERVERSTANDARD...
6/14/2019 4:03:46 PM - Copying: C:\Users\windows-openstack-imaging-tools\UnattendResources G:\UnattendResources
6/14/2019 4:03:46 PM - Resources have been copied.
6/14/2019 4:03:46 PM - Copy Custom Resources: ...
6/14/2019 4:03:46 PM - Custom Resources at: G:\UnattendResources.
6/14/2019 4:03:46 PM - Setting wallpaper...
6/14/2019 4:03:46 PM - Wallpaper copied to the image.
6/14/2019 4:03:46 PM - Cached wallpaper for user Administrator has been replaced.
6/14/2019 4:03:46 PM - Wallpaper GPO copied to the image.
6/14/2019 4:03:46 PM - Wallpaper was set.
6/14/2019 4:03:46 PM - Downloading ntfszapfree...
6/14/2019 4:03:48 PM - Downloading Cloudbase-Init...
6/14/2019 4:04:07 PM - Waiting for WindowsGoldImage-Sysprep323236298 to finish sysprep.
6/14/2019 4:08:23 PM - Shrinking VHD to minimum size
6/14/2019 4:08:23 PM - Initial VHD size is: 30 GB
6/14/2019 4:08:30 PM - Current partition size: 29.998046875 GB
6/14/2019 4:08:30 PM - New partition size: 13 GB
6/14/2019 4:08:30 PM - Size increased: 13958643712
WARNING: Size Not Supported

Extended information:
The specified shrink size is too big and will cause the volume to be smaller than the
minimum volume size.

Activity ID: {4ba4fabb-7c02-4d59-9951-7a9a156a8d94}
6/14/2019 4:08:40 PM - Size increased: 14063501312
6/14/2019 4:08:52 PM - Final disk size: 13.099609375 GB
6/14/2019 4:08:52 PM - Optimize VHD E:\diskimages\zap.vhdx: file size before optimization is 23.12890625 GB
6/14/2019 4:09:06 PM - Optimize VHD E:\diskimages\zap.vhdx: file size after optimization is 12.03515625 GB
6/14/2019 4:09:06 PM - Cloud image from golden image generation finished.

[ader1990]

@felfert Thank you for the contribution, will add what we discussed in the comment sections as issues to improve on, like the sysprep wait and proper cleanup,