Switch to using OSBuild for most CoreOS boot image building

[dustymabe]

Since the beginning in CoreOS we have inherited or developed tooling inside COSA (CoreOS Assembler) to take the created OSTree commit and stick that into all of our different disk images that we ship (qemu, aws, openstack, ISO, etc).

Since we are the only consumers of COSA we have the desire to not maintain the bespoke image building code over the long term. The consensus is to use Image Builder/OSBuild for disk image building in the future. A good amount of work has gone into enabling OSBuild to be able to do this already.

This will also allow anyone who has created a container layer (see https://github.com/coreos/layering-examples/) to build a boot image from a derived container layer using OSBuild and not requiring COSA.

For the first round of this we will focus on building the qemu, metal and metal4k images. Most other images that we build are actually based off of the qemu image so they will also be indirectly affected (in the future we'll use OSBuild to build them directly).

Out of scope for this first round is the ISO/PXE artifacts as those are a bit more complicated.

[cgwalters]

I edited your text to s/image/disk image/ for osbuild.

[dustymabe]

We discussed this in the Community Meeting Today:

* AGREED: We will roll out disk image building using OSBuild to our `rawhide`
          mechanical stream first (this is a non-user facing stream). Then we
          will roll out OSBuild to the `next` stream and let it soak there for
          some time. We may choose to switch over `testing` before the rebase
          to F40, but we won't do it until F40 has at least reached `next` at
          F40 beta time. (@dustymabe:matrix.org, 16:52:30)

[dustymabe]

First step of enablement for this in the pipeline:

• https://github.com/coreos/fedora-coreos-pipeline/pull/956

[dustymabe]

OK last week we started building using OSBuild for rawhide in our pipeline.

In order to compare a little I decided to write a small helper utility to do size comparison. diffbuildsizes.py.txt

Here is the output for the OSBuild built artifacts:

    platform     | compression  | 40.20240131.91.0 | 40.20240201.91.0
---------------------------------------------------------------------
qemu             | compressed   | 663MiB           | 848MiB          
qemu             | uncompressed | 1577MiB          | 876MiB          
metal4k          | compressed   | 661MiB           | 670MiB          
metal4k          | uncompressed | 2458MiB          | 2458MiB         
metal            | compressed   | 663MiB           | 698MiB          
metal            | uncompressed | 2458MiB          | 2458MiB         

What I really don't understand is the sizes for the qemu image. The uncompressed size is much smaller, but the compressed size is about 185MiB larger. All of this analysis is done using the size values from the meta.json file.

[dustymabe]

OK I think I figured this out. OSBuild does the qcow2 conversion with compression on. This means the contents are internally compressed, which explains why the "uncompressed" size is much better (876MiB versus 1577MiB), but that means the outer xz compression is less effective (848MiB versus 663MiB).

[dustymabe]

ok a lot of work has happened here in the background. https://github.com/osbuild/osbuild/pull/1589 was the last big piece IIUC. I've now enabled OSBuild for the branched stream too (https://github.com/coreos/fedora-coreos-pipeline/pull/966).

We'll enable it for next-devel once we are closer to F40 beta by merging https://github.com/coreos/fedora-coreos-pipeline/pull/967

[dustymabe]

With the switch of the next stream to F40 we are now shipping OSBuild built qemu, metal, and metal4k artifacts that were directly built using OSBuild. The first release for this was 40.20240322.1.0.

Note that most other images (aws, gcp, etc) are actually derived from the qemu artifact so in a way those are also indirectly built using OSBuild too. We'll switch them over to direct building as we add the functionality to support those.

[dustymabe]

The fix for this went into next stream release 40.20240322.1.0. Please try out the new release and report issues.

[dustymabe]

We missed this when we switched the testing stream over to F40. It will be in the next testing release that goes out.

[jlebon]

Out of scope for this first round is the ISO/PXE artifacts as those are a bit more complicated.

When closing this ticket, let's file another ticket to track that follow-up work. That way also, others interested in this can subscribe to it.

Edit: actually, filed it at https://github.com/coreos/fedora-coreos-tracker/issues/1725

[dustymabe]

PR to make building using OSBuild the default in COSA: https://github.com/coreos/coreos-assembler/pull/3800

[dustymabe]

The fix for this went into testing stream release 40.20240504.2.0. Please try out the new release and report issues.

[travier]

Looks like this is now in stable:

• https://github.com/coreos/coreos-assembler/pull/3800
• https://github.com/coreos/fedora-coreos-pipeline/pull/988

[marmijo]

The fix for this went into stable stream release 40.20240504.3.0. Please try out the new release and report issues.