Windows .raw image is too large (30GB)

C00kiie commented 1 year ago

Hi, I tried to create a cloud image in .raw format, for openstack hypervisor, with the examples/create-windows-cloud-image.ps1 powershell script.

Here's my config.ini file:

[DEFAULT]
# The location of the WIM file from the mounted Windows ISO.
wim_file_path=H:\Sources\install.wim
# This is the complete name of the Windows version that will be generated.
# In order to find the possible options, use the Get-WimFileImagesInfo command
# and look for the Name property.
image_name=Windows Server 2022 SERVERSTANDARD
# The destination of the generated image.
image_path=V:\windows-image-build-qcow2\windows2022-image.raw
# Select between VHD, VHDX, QCOW2, VMDK or RAW formats.
virtual_disk_format=RAW
# This parameter allows to choose between MAAS, KVM, VMware and Hyper-V specific images.
# For HYPER-V, cloudbase-init will be installed and the generated image should be in vhd or vhdx format.
# For MAAS, in addition to cloudbase-init, the curtin tools are installed
# and the generated image should be in raw.tgz format.
# For KVM, in addition to cloudbase-init, the VirtIO drivers are installed
# and the generated image should be in qcow2 format.
image_type=HYPER-V
# This parameter can be set to either BIOS or UEFI.
disk_layout=BIOS
# The product key for the selected OS. If the value is default_kms_key and the Windows image is
# ServerStandard or ServerDatacenter (Core), the appropiate KMS key will be used.
product_key=""
# A comma separated array of extra features that will be enabled on the resulting image.
# These features need to be present in the ISO file.
extra_features=""
# A comma separated array of extra capabilities that will be enabled on the resulting image.
# These capabilities need to be present in the ISO file.
extra_capabilities=""
# It will force the image generation when RunSysprep is False or the selected SwitchName
# is not an external one. Use this parameter with caution because it can easily generate
# unstable images.
force=False
# If set to true, MAAS Windows curtin hooks will be copied to the image root directory.
install_maas_hooks=False
# Select between tar, gz, zip formats or any combination between these.
compression_format=""
# If this parameter is set, after the image is generated,
# a password protected zip archive with the image will be created.
# compression_format must contain zip in order for this parameter to be used
zip_password=""
# It will stop the image generation after the updates are installed and cleaned.
gold_image=False
# This is the full path of the already generated golden image.
# It should be a valid VHDX path.
gold_image_path=""
# This is a full path to the VMware-tools.exe version that you want to install.
vmware_tools_path=""
# If set to true, .NET Framework 3.5 will be installed before the windows updates.
# This feature applies only ot Windows server 2012 R2.
install_net_3_5=False
# This is the full path of a folder with custom resources which will be used by
# the custom scripts.
# The resources found at this path will be copied recursively to the image
# UnattendResources\CustomResources folder.
custom_resources_path=""
# This is the full path of the folder which can contain a set of PS scripts,
# that will be copied and executed during the online generation part on the VM.
# The PowerShell scripts, if existent, will be started by Logon.ps1 script,
# at different moments during image generation.
# The purpose of these scripts is to offer to the user a fully
# customizable way of defining additional logic for tweaking the final image.
# The scripts files can have the following names: RunBeforeWindowsUpdates.ps1,
# RunAfterWindowsUpdates.ps1, RunBeforeCloudbaseInitInstall.ps1, RunAfterCloudbaseInitInstall.ps1,
# RunBeforeSysprep.ps1, RunAfterSysprep.ps1.
# The script names contain the information on when the script will be executed.
# One can define only some of the hook scripts and it is not mandatory to define all of them.
# If a script does not exist, it will not be executed.
custom_scripts_path=""
# If set to true the Administrator account will be enabled on the client
# versions of Windows, which have the Administrator account disabled by default
enable_administrator_account=False
# Whether to shrink the image partition and disk after the image generation is complete.
shrink_image_to_minimum_size=True
# If set to true, a custom wallpaper will be set according to the values of configuration options
# wallpaper_path and wallpaper_solid_color
enable_custom_wallpaper=True
# If set, it will replace the Cloudbase Solutions wallpaper to the one specified.
# The wallpaper needs to be a valid .jpg/.jpeg image.
wallpaper_path=""
# If set, it will replace the Cloudbase Solutions wallpaper to a solid color.
# Currently, the only allowed solid color is '0 0 0' (black).
# If both wallpaper_path and wallpaper_solid_color are set,
# the script will throw an error.
wallpaper_solid_color=""
# If set, the animation displayed during the first login on Windows Client versions will be disabled.
disable_first_logon_animation=False
# If set to true and the target image format is QCOW2, the image conversion will
# use qemu-img built-in compression. The compressed qcow2 image will be smaller, but the conversion
# will take longer time.
compress_qcow2=False
# If set to true, during final cleanup, https://github.com/felfert/ntfszapfree will be used to zero unused space.
# This helps qemu-img to minimize image size. In order to benefit from this, an additional invocation
# of qemu-img convert must be performed after the initial run of the image has shutdown.
zero_unused_volume_sectors=False
# A comma separated list of extra packages (referenced by filepath)
# to slipstream into the underlying image.
# This allows additional local packages, like security updates, to be added to the image.
extra_packages=""
# Ignore failures from DISM when installing extra_packages, such as when
# updates are skipped which are not applicable to the image.
extra_packages_ignore_errors=False
# Enables shutdown of the Windows instance from the logon console.
enable_shutdown_without_logon=False
# If set to true, firewall rules will be added to enable ping requests (ipv4 and ipv6).
enable_ping_requests=False
# If set to true, use EUI-64 derived IDs and disable privacy extensions for IPv6.
# If set to false, the IPv6 protocol might not work on OpenStack or CloudStack.
# See https://github.com/cloudbase/windows-openstack-imaging-tools/issues/192
enable_ipv6_eui64=False
# If set to true, it will set the High Performance mode and some power mode
# and registry tweaks to prevent the machine from sleeping / hibernating.
enable_active_mode=False
[vm]
# This will be the Administrator user's, so that AutoLogin can be performed on the instance,
# in order to install the required products,
# updates and perform the generation tasks like sysprep.
administrator_password=Pa$$w0rd
# Used to specify the virtual switch the VM will be using.
# If it is specified but it is not external or if the switch does not exist,
# you will get an error message.
external_switch=external
# The number of CPU cores assigned to the VM used to generate the image.
cpu_count=1
# RAM (in bytes) assigned to the VM used to generate the image.
ram_size=2147483648
# Disk space (in bytes) assigned to the boot disk for the VM used to generate the image.
disk_size=32212254720
# If set to true and the disk layout is UEFI, the secure boot firmware option will be disabled.
disable_secure_boot=False
[drivers]
# The path to the ISO file containing the VirtIO drivers.
virtio_iso_path=V:\windows-image-build-qcow2\virtio.iso
# The location where the VirtIO drivers are found.
# For example, the location of a mounted VirtIO ISO. VirtIO versions supported >=0.1.6.x
virtio_base_path=""
# The location where additional drivers that are needed for the image are located.
drivers_path=""
[custom]
# Installs QEMU guest agent services from the Fedora VirtIO website.
# Defaults to 'False' (no installation will be performed).
# If set to 'True', the following MSI installer will be downloaded and installed:
# * for x86: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-qemu-ga/qemu-ga-win-100.0.0.0-3.el7ev/qemu-ga-x86.msi
# * for x64: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-qemu-ga/qemu-ga-win-100.0.0.0-3.el7ev/qemu-ga-x64.msi
# The value can be changed to a custom URL, to allow other QEMU guest agent versions to be installed.
# Note: QEMU guest agent requires VirtIO drivers to be present on the image.
install_qemu_ga=False
# Set a custom timezone for the Windows image.
time_zone=""
# Set custom ntp servers(space separated) for the Windows image
ntp_servers=""
[updates]
# If set to true, the latest updates will be downloaded and installed.
install_updates=True
# If set to true, will run DISM with /resetbase option. This will reduce the size of
# WinSXS folder, but after that Windows updates cannot be uninstalled.
purge_updates=True
# Clean up the updates / components by running a DISM Cleanup-Image command.
# This is useful when updates or capabilities are installed offline.
clean_updates_offline=False
# Clean up the updates / components by running a DISM Cleanup-Image command.
# This is useful when updates or other packages are installed when the instance is running.
clean_updates_online=True
[sysprep]
# Used to clean the OS on the VM, and to prepare it for a first-time use.
run_sysprep=True
# The path to the Unattend XML template file used for sysprep.
unattend_xml_path=UnattendTemplate.xml
# DisableSwap option will disable the swap when the image is generated and will add a setting
# in the Unattend.xml file which will enable swap at boot time during specialize step.
# This is required, as by default, the amount of swap space on Windows machine is directly
# proportional to the RAM size and if the image has in the initial stage low disk space,
# the first boot will fail due to not enough disk space. The swap is set to the default
# automatic setting right after the resize of the partitions is performed by cloudbase-init.
disable_swap=True
# In case the hardware on which the image is generated will also be the hardware on
# which the image will be deployed this can be set to true, otherwise the spawned
# instance is prone to BSOD.
persist_drivers_install=True
[cloudbase_init]
# This is a switch that allows the selection of Cloudbase-Init branches. If set to true, the
# beta branch will be used:
# https://cloudbase.it/downloads/CloudbaseInitSetup_<arch>.msi, where arch can be x86 or x64
# otherwise the stable branch will be used:
# https://cloudbase.it/downloads/CloudbaseInitSetup_Stable_<arch>.msi, where arch can be x86 or x64
beta_release=False
# Serial log port for Cloudbase-Init.
# If set to null, the first serial port (if any) from the generation VM will be used
serial_logging_port=COM1
# If set, the Cloudbase-Init msi at this path will be used.
# The path needs to be a locally accessible file path.
msi_path=""
# If set, the cloudbase-init.conf is replaced with the file at the path.
cloudbase_init_config_path=""
# If set, the cloudbase-init-unattend.conf is replaced with the file at the path.
cloudbase_init_unattended_config_path=""
# If set, the Cloudbase-Init service will be run under Local System account.
# By default, a user named cloudbase-init with admin rights is created and used.
cloudbase_init_use_local_system=False
# If set, the Cloudbase-Init service startup type will be set to delayed-auto
cloudbase_init_delayed_start=False

I'm using windows 2022 trial evaluation image ,

I edited a few lines in the example, so it doesn't download virtio.iso file over and over, since I have low bandwidth, here's the full create-windows-cloud-image.ps1 file:

# Copyright 2016 Cloudbase Solutions Srl
#
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.

$ErrorActionPreference = "Stop"

$scriptPath =Split-Path -Parent $MyInvocation.MyCommand.Definition | Split-Path
git -C $scriptPath submodule update --init
if ($LASTEXITCODE) {
    throw "Failed to update git modules."
}

try {
    Join-Path -Path $scriptPath -ChildPath "\WinImageBuilder.psm1" | Remove-Module -ErrorAction SilentlyContinue
    Join-Path -Path $scriptPath -ChildPath "\Config.psm1" | Remove-Module -ErrorAction SilentlyContinue
    Join-Path -Path $scriptPath -ChildPath "\UnattendResources\ini.psm1" | Remove-Module -ErrorAction SilentlyContinue
} finally {
    Join-Path -Path $scriptPath -ChildPath "\WinImageBuilder.psm1" | Import-Module
    Join-Path -Path $scriptPath -ChildPath "\Config.psm1" | Import-Module
    Join-Path -Path $scriptPath -ChildPath "\UnattendResources\ini.psm1" | Import-Module
}

# The Windows image file path that will be generated
$virtualDiskPath = "V:\windows-image-build-qcow2\windows2022-image.raw"

# The wim file path is the installation image on the Windows ISO
$wimFilePath = "H:\Sources\install.wim"

# VirtIO ISO contains all the synthetic drivers for the KVM hypervisor
$virtIOISOPath = "V:\windows-image-build-qcow2\virtio.iso"
# Note(avladu): Do not use stable 0.1.126 version because of this bug https://github.com/crobinso/virtio-win-pkg-scripts/issues/10
# Note (atira): Here https://fedorapeople.org/groups/virt/virtio-win/CHANGELOG you can see the changelog for the VirtIO drivers
$virtIODownloadLink = "https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.217-2/virtio-win-0.1.217.iso"

# Download the VirtIO drivers ISO from Fedora

# Extra drivers path contains the drivers for the baremetal nodes
# Examples: Chelsio NIC Drivers, Mellanox NIC drivers, LSI SAS drivers, etc.
# The cmdlet will recursively install all the drivers from the folder and subfolders
$extraDriversPath = "V:\windows-image-build-qcow2\drivers"

# Every Windows ISO can contain multiple Windows flavors like Core, Standard, Datacenter
# Usually, the second image version is the Standard one
$image = (Get-WimFileImagesInfo -WimFilePath $wimFilePath)[1]

# The path were you want to create the config fille
$configFilePath = Join-Path $scriptPath "Examples\config.ini"
New-WindowsImageConfig -ConfigFilePath $configFilePath

#This is an example how to automate the image configuration file according to your needs
Set-IniFileValue -Path $configFilePath -Section "Default" -Key "wim_file_path" -Value $wimFilePath
Set-IniFileValue -Path $configFilePath -Section "Default" -Key "image_name" -Value $image.ImageName
Set-IniFileValue -Path $configFilePath -Section "Default" -Key "image_path" -Value $virtualDiskPath
Set-IniFileValue -Path $configFilePath -Section "Default" -Key "virtual_disk_format" -Value "RAW"
Set-IniFileValue -Path $configFilePath -Section "vm" -Key "disk_size" -Value (30GB)
Set-IniFileValue -Path $configFilePath -Section "drivers" -Key "virtio_iso_path" -Value $virtIOISOPath
Set-IniFileValue -Path $configFilePath -Section "updates" -Key "install_updates" -Value "True"
Set-IniFileValue -Path $configFilePath -Section "updates" -Key "purge_updates" -Value "True"
Set-IniFileValue -Path $configFilePath -Section "sysprep" -Key "disable_swap" -Value "True"

# This scripts generates a raw image file that, after being started as an instance and
# after it shuts down, it can be used with Ironic or KVM hypervisor in OpenStack.
New-WindowsCloudImage -ConfigFilePath $configFilePath

TL;DR: I created windows cloud image with create-windows-cloud-image.ps1 and got a 30GB raw image. Is this the normal behavior?

C00kiie commented 1 year ago

notes / questions:

I'm creating this image for openstack, and we can't use qcow2 images for storage backend compatibility purposes, so we have our hands tied to using .raw format.
Will this automatically launch instances in openstack, or will I have to use extra deployment tools such as packer to automate the deployment?

ader1990 commented 1 year ago

Hello,

There are a few examples on the examples folder, with one more suitable on what you might need: https://github.com/cloudbase/windows-imaging-tools/blob/master/Examples/create-windows-online-cloud-image.ps1

New-WindowsCloudImage is the "half-baked" image, where the WIM has been applied but no updates, no Cloudbase-init has been installed, sysprep has not been executed. You need a hypervisor to instantiate and run the VM with the backing virtual disk New-WindowsCloudImage has created, in order to finish configuring the image.

New-WindowsOnlineImage is the method to create a fully working image.

Thank you, Adrian Vladu

ader1990 commented 1 year ago

Closing the issue as the New-WindowsOnlineImage should be used to obtain a shrinked image.

cloudbase / windows-imaging-tools

Windows .raw image is too large (30GB) #382