Make the templates dependable on each other, layered templates

[david-caro]

Moved from https://bugzilla.redhat.com/show_bug.cgi?id=1283344

If we could use snapshots or similar and allow dependencies between templates, the disk space required on the client (and alas, the amount of data to download) would drop a lot, for example having a base centos-7 template, and a host and engine template that depend on it, but are just a snapshot on top of it, so they only occupy the extra space of the installation and configuration instead of duplicating it completely.

[nvgoldin]

Hi, the below is a draft proposal me and @gbenhaim worked on, feel free to comment. It does not include all details yet. This is also dependant on some abstraction needed to the current Template mechanism, I'll open a different ticket for that adding the planning.

Layered images

Abstract

The general concept is to allow using layered images in Lago by using qemu-img 'backing_file' functionality[1] . The layered images will be automatically resolved for the end-user, without needing to explicitly define the images chain.

Terminology: Layered image - an image that has a none-empty 'backing_file' parameter. Base image - an image with an empty 'backing_file'. Images server - a web server storing image files.

Resolving layered images

The resolution of which images need to be downloaded will be done locally without storing explicit metadata files on the server. The metadata needed to resolve the images chain will be extracted from the qcow file. On the images server, the qcow images will be stored so the 'backing_chain' parameter holds the needed metadata to resolve if it refers to another image. The expected format for 'backing_file' parameter in a qcow image stored at the server would be similar to: lago::SHA512::IMAGE-NAME

Where 'SHA512' is the hash for the next image in the chain. Once downloaded and stored locally, lago will rename the image filename to a combination of the hash and name, after resolving all images needed in the chain, it will change the 'backing_file' parameter to point to the locally downloaded image.

Example 1

Assuming the image server holds the following qcow2 files, each having the defined backing_file parameter: filename=base-img.qcow2,hash=hash1,backing_file= filename=second-img.qcow2:hash=hash2,backing_file=lago::hash1::base-img filename=third-image.qcow2:hash=hash3,backing_file=lago::hash2:second-img

And the following LagoInitFile: domains: host1: .... template_name: third-image .... When running 'lago init', the following image chain will be created(where '<-' indicates the backing_file parameter):

host1_node <- third-image <- second-image <- host1-img

Assuming the images cache directory is /var/lib/lago/store, these will be the new images and the backing_file parameter:

/var/lib/lago/store/hash1_base-img.qcow /var/lib/lago/store/hash2_second-img.qcow, backing_file=/var/lib/lago/store/hash1_base-img.qcow /var/lib/lago/store/hash3_third-image.qcow, backing_file=/var/lib/lago/store/hash2_second-img.qcow

[david-caro]

I had some code that implemented a preliminary version of the templates on top of https://github.com/david-caro/lago-images/

It used the images metadata that's in the lago repository metadata to link images one to the other making the resolution trivial and allowing parallel downloads.

I'll try to find the code, maybe it's on the templates.ovirt.org server somewhere, not sure, I'll check (I really regret not pushing it anywhere :/ not sure what was I thinking)

[david-caro]

it's actually at the lago-project one :+1: here: https://github.com/lago-project/lago-images/commit/82dadca1eccb1042adbb2acf1bf95e7a0ce5f602

The dependency linking is done through the 'base' header in the image spec, if it starts with 'libguestfs:' then it pulls it from libguestfs and no layer is done for that one as there's no way to layer on top of those, but if it does not have that header, then it will be layered on top of the matching one, see the el7-iscsi there for an example.

This only solves the image generation on the server side though, on the cli side I had no code yet (I manually downloaded those images and stashed them into the lago cache dir to test them and they worked well)

[david-caro]

Btw. there was no layered support on virt-builder side at the time, they were thinking about adding something but they were not sure. Might be nice to know if they implemented it, if not, the repos will stop being fully virt-builder capable (afaik we never used that feature, so might not be a big issue to drop it).

[david-caro]

Also, be careful with resizing images, as resizing a layered image pulls to it all the size of the base image, making it as if it were not layered. We thought on alleviating that issue by generating multiple sized layered images, like el7-nfs-small with 8G of free space, el7-nfs-medium with 20G and el7-nfs-big with 50G or similar...

[ifireball]

On 2 November 2016 at 18:49, Nadav Goldin notifications@github.com wrote:

Example 1

Assuming the image server holds the following qcow2 files, each having the defined backing_file parameter: filename=base-img.qcow2,hash=hash1,backing_file= filename=second-img.qcow2:hash=hash2,backing_file=lago::hash1::base-img filename=third-image.qcow2:hash=hash3,backing_file=lago::hash2:second-img

I wonder, why do we need the "lago::" prefix in the backing_file field?

It seems nothing else in this proposal is Lago-specific, so maybe we should do things in such a way that will allow other projects to adopt this "standard".

It will be exciting if at some point virt-builder and oVirt will support layered images too...

Barak Korren bkorren@redhat.com RHEV-CI Team

[eedri]

Please move it to To Do (backlog) while we discuss the ETA and scope, we might want to finish packaging and its blocking items first.

[nvgoldin]

@david-caro - thanks for the finding the code! I looked through it, I think that if we go in the path of https://github.com/lago-project/lago/issues/359, I would rather see the resolution done locally, thus decoupling the dependency from doing it on the server side. This will also allow to use different remote servers to fetch the images from and building different layers. thanks for the info about virt-resize, will need further inspection. @ifireball

I wonder, why do we need the "lago::" prefix in the backing_file field?

the thought was that if a user downloads such an image and doesn't use lago, at least he would have some idea where to look, though it makes sense to have the plain hash there.

@eedri - moved to to do.

[david-caro]

@nvgoldin I don't think that having the info computed on the server is incompatible with pulling the images from different servers, you can still get the images from anywhere. And instead of having to download the complete image each time in order to know which is the next, you have all the info from a small index file/api call.

[nvgoldin]

@david-caro - good point. the only advantage in knowing the exact chain in advance, i can think of, is if lago would optimize the pulling with parallel downloads, as: If you never had this image locally, you are going to download it either way. when download completes, you discover the parent hash: if you already have it, your done, else you must download it. If you already know the entire chain from the repo you could trigger parallel downloads for the none-cached images(this could introduce a race condition between resolving the chain and the repo being updated.. though that is probably a really minor issue)

so I guess it is 'using a standard format' vs 'optimizations', unless I missed other advantages.

[david-caro]

What standard format?

Knowing the list of imagesnin advance allows you to make a prediction of the total size you need, allowing free soace checks and progress bars for example.