Support qcow2 as a storage backend

[ghost]

Are there any plans to have LXD support qcow2 images as a storage backend?

I am not much of a docker fan but when trying it a few times I noticed that it uses qcow2 images for the backend. The storage backend feature list seems to indicate that "dir" based storage tends to be the least feature-full method of storage for LXD and that ZFS being the preferred if you want particular storage features.

In a number of cases though there may be reasons for admins to not want to use ZFS on a Linux host machine. Images with the qcow2 format offer similar types of features to LVM/ZFS but are stored as a file on the file system instead

Thanks

[stgraber]

No real reason not to support it, though it'd certainly not be anywhere near the feature set of zfs/btrfs, at best I'd expect it to be somewhat similar to LVM.

[ghost]

In terms of ZFS file system support features, I agree qcow2 is not up there but it does have some very interesting image (block) support. I thought I might pop a comparison between the current storage methods based on qcow2 features from what I could see/find.

Feature	Directory	Btrfs	LVM	ZFS	QCOW2
Optimized image storage	no	yes	yes	yes	?
Optimized container creation	no	yes	yes	yes	?
Optimized snapshot creation	no	yes	yes	yes	?
Optimized image transfer	no	yes	no	yes	?
Optimized container transfer	no	yes	no	yes	?
Copy on write	no	yes	yes	yes	yes
Block based	no	no	yes	no	no/yes*
Instant cloning	no	yes	yes	yes	yes
Nesting support	yes	yes	no	no	?
Restore from older snapshots (not latest)	yes	yes	yes	no	yes
Storage quotas	no	yes	no	yes	yes

I am not too clued yet on the internals of LXD workings so not sure if the optimized options would apply. I don't see why not though since qcow2 does support image overlays similar to ZFS (instant cloning). Compression and encryption built in and could lay the foundation in supporting the planned qcow3 format.

^{* not 100% on this but as far as I can see qcow2 emulates a block device but is stored as an image file on a host system.}

[stgraber]

Yeah, I suspect we can do quite a few of the things we need by using qcow2 overlays, the obvious downside being that we'd need to keep the source image so long as it's got any downstream user, but that's pretty similar to zfs filesystems in that regard.

I suspect the main problems will be similar to what we have with LVM, specifically that live grow/shrink is painful due to having to deal with both block+filesystem and that we can't get low overhead live migration as there is no fs-level delta support and the base image hash will be different between hosts requiring a re-send.

[voxsoftware]

Why not simply use ZFS on RAW file.

truncate -s160G /root/disk.img
zpool create disk0 /root/disk.img

[ivzhh]

Why not simply use ZFS on RAW file.

truncate -s160G /root/disk.img
zpool create disk0 /root/disk.img

Or maybe zvol.

[1000283]

It could be useful to import from a .qcow2 file into LXD's storage backend, be it ZFS or what not. (I'm searching for a working template for Slackware - hard to find - and thinking about installing it on a KVM VM and using the disk as baselite for the rootfs.)

[stgraber]

lxc image import for VMs takes a tarball as first argument (for metadata) and a QCOW2 as second argument for the rootfs, so we effectively already support that.

[Wyk72]

I use qcow2 as a LXC "backend" with simple mount scripts & some little helpers programs (qemu-nbd), for particular use-cases.

One was to boot an Alpine Linux USB key in run-from-ram mode with containers and KVM machines alike.

This way I can keep a zstd-compressed qcow2 image file (ext4 "no journal" formatted) with inside the rootfs of the LXC container as read-only on the USB key.

Then I use that read-only qcow2 file as "backing" for a new .qcow2 file that is copied into RAM and gets written with LBU whan I shut down the machine.

This way I keep the amount of data written to the USB kay as small as possible.

It boils down to very simple scripts, but it can be pretty fast, got a lot of performance out of ancient 50€ PCs (i5-4570 w 4gb of ram), and just a old and crusty 8GB USB key.

A little example (I use Alpine Linux as my main OS):

Create qcow2 file, keep cluster_size small to waste less space (hurts performace a bit, but not much)

qemu-img create -f qcow2 /qcow2/issab-lxc.qcow2 15G -o cluster_size=32K

use nbd & qemu-nbd to mount it

modprobe nbd qemu-nbd --connect /dev/nbd0 /qcow2/issab-lxc.qcow2 --aio=io_uring --cache=writethrough

format the qcow2 image with ext4 (no journal, since it will be read-only)

mkfs.ext4 -O ^has_journal /dev/nbd0

mount it somewhere - fs type needs to be specified, at least on Alpine

mount -t ext4 /dev/nbd0 /mnt

copy the containers rootfs into it with rsync

rsync -avxHAX /var/lib/lxc/ /mnt/

umount it

umount /mnt

disconnect nbd from the file

qemu-nbd --disconnect /dev/nbd0 rmmod nbd

convert&compress the qcow2 file into another qcow2 image with zstd

qemu-img convert -f qcow2 -O qcow2 /qcow2/issab-lxc.qcow2 /qcow2/issabel-lxc.qcow2 -c -p -o compression_type=zstd,cluster_size=32k,extended_l2=on

.. now you can copy this compressed qcow2 file to the read-only storage (usb key, a network share, it just need to be readable/reachable).

To use those LXC containers "golden images" you just need to create a new qcow2 file that uses the read-only file ad backing file i.e.

create

qemu-img create -f qcow2 -F qcow2 -b /media/usb/issabel-lxc.qcow2 issabel.qcow2 -o cluster_size=32K,extended_l2=on

this new file "issabel.qcow2" will be initially just a few Kbytes in size, will be filled new data as it is the "r/w" part.

To use it in LXC I put this into my local.d openrc scripts:

connect the qcow2 file to the nbd0 device

qemu-nbd --connect /dev/nbd0 /root/issabel.qcow2 --aio=io_uring --cache=writethrough

create the path: since we run-from-ram /var/lib is empty....

mkdir -p /var/lib/lxc/issabel

mount the lxc rootfs there

mount -t ext4 /dev/nbd0 /var/lib/lxc/

start the container

/etc/init.d/lxc.issabel start

That's about it. It's a DIY solution but it has proven to be a good one.

I could have used overlayfs or some other trick, but this keeps the LXC rootfs compressed and all in one file, easy to move around.

Of course you can rebase the whole thing with just one command if you need to AND you have snapshots too (never used but can be interesting).

It's like using the QEMU/qcow2 "engine" for other purposes. A cute trick IMHO.