clalancette / oz

Automated installation for guest images
GNU Lesser General Public License v2.1
310 stars 129 forks source link

need a way to telll qemu to use discard="unmap" so that FITRIM() will pass through, to reduce image sizes #301

Open cmurf opened 2 years ago

cmurf commented 2 years ago

Summary: The VM's need to have virtioblk set to use discard=unmap in order for guest issuance of fstrim to pass through to the host backing file. This has the effect of creating a sparse backing file, since the holes don't take up space when compressing, it reduces image compression time, storage and download resources.

Details:

  1. By mounting an oz created image on loop and using fstrim manually, I can demonstrate the large amount of garbage stuck in the raw file that should instead be holes.
# mount /dev/mapper/loop0p3 /mnt/
# ls -ls
5129956 -rw-r--r--. 1 root root 13958643712 Jul 22 15:03
Fedora-Workstation-Rawhide-20220721.n.0.aarch64.raw
# fstrim -v /mnt
/mnt: 8.5 GiB (9149927424 bytes) trimmed
# ls -ls
3293292 -rw-r--r--. 1 root root 13958643712 Jul 22 15:04
Fedora-Workstation-Rawhide-20220721.n.0.aarch64.raw

This is about 1.75GiB of garbage needlessly compressed, stored, downloaded, and decompressed - times every nightly and person who downloads and uses the image. It's a huge waste. Similar problem exists for Fedora Cloud Base image.

First mount
892856 -rw-rw-r--. 1 chris chris 4294967296 Jul 21 14:45 Fedora-Cloud-Base-36-20220721.0.x86_64.raw
fstrim /boot (partition 2)
856796 -rw-rw-r--. 1 chris chris 4294967296 Jul 21 14:45 Fedora-Cloud-Base-36-20220721.0.x86_64.raw
fstrim / (partition 5)
429316 -rw-rw-r--. 1 chris chris 4294967296 Jul 21 14:46 Fedora-Cloud-Base-36-20220721.0.x86_64.raw
  1. I found this file in koji along with the logs and image files for the aaarch64 Workstation image above. https://kojipkgs.fedoraproject.org//packages/Fedora-Workstation/Rawhide/20220721.n.0/images/libvirt-raw-xz-aarch64.xml

Also attaching that xml file. libvirt-raw-xz-aarch64.xml.txt

It contains this section

<disk device="disk" type="file">
<target dev="vda" bus="virtio"/>
<source file="Fedora-Workstation-Rawhide-20220721.n.0.aarch64.raw-xz"/>
<driver name="qemu" type="raw-xz"/>
</disk>

Instead this needs:

<driver name="qemu" type="raw-xz" cache="unsafe" discard="unmap"/>  [1][2]

Alternative: I'm told oz uses guestfs after installation to cutomize the image, so if it's easier to do the quiescing there, that's OK too.

[1] The cache="unsafe" is consistent with other Fedora builder and compose systems, improving performance considerably. [2] I'm not certain driver type "raw-xz" supports discard=unmap, type "raw" definitely does.