Boot performance vs dnf image

[ericcurtin]

Running some performance tests, we found some differences between an ostree version of Automotive Stream Distribution and a dnf version of Automotive Stream Distribution (called "regular" in the below tables). Boot time is under a microscope as part of our Automotive efforts.

These numbers were taken from 50 runs. One interesting result is the difference in the time it takes to get to "initrd-switch-root.service" which is the service that takes us to the final rootfs.

STARTUP SECTIONS	ER3 ‘regular’	ER3 ‘ostree’	% TIME Increase w/‘ostree’
kernel	1.09	1.10	0.9%
initrd	5.69	6.11	7.1%
userspace	16.79	18.95	12.1%
TOTAL	23.57	26.16	10.4%

SYSTEMD UNITS	ER3 ‘regular’	ER3 ‘ostree’	% TIME Increase w/‘ostree’
NetworkManager-wait-online.service	5.91	5.93	0.3%
initrd-switch-root.service	2.85	3.67	29%
systemd-random-seed.service	1.78	2.02	13%
NetworkManager.service	1.68	1.4	-17%
systemd-tmpfiles-setup.service	1.28	2.64	106%
initrd-cleanup.service	0.92	1.06	15%
systemd-udev-trigger.service	0.66	0.61	-7.6%
dbus-broker.service	0.63	0.61	-3.2%
systemd-logind.service	0.60	0.59	-1.7%

An admirable goal could be to get to initrd-switch-root.service as quickly as a dnf image.

[cgwalters]

called "regular"

I really don't like being implicitly "irregular" or "not normal" personally. https://github.com/openshift/machine-config-operator/pull/3580#discussion_r1124910396

NetworkManager-wait-online.service

Why is this even pulled in for you? Is NBDE in play here?

Do you have pre-built example images here? Or even just the relevant initramfs.img?

[ericcurtin]

called "regular"

I really don't like being implicitly "irregular" or "not normal" personally. openshift/machine-config-operator#3580 (comment)

I would prefer an alternate name also for the record.

NetworkManager-wait-online.service

Why is this even pulled in for you? Is NBDE in play here?

It is pulled in at present, but it may not be in play in a real life deployment.

Do you have pre-built example images here? Or even just the relevant initramfs.img?

@jharriga collected these. So he would have to confirm, but it would be something like the initramfs from:

git@gitlab.com:CentOS/automotive/sample-images.git
cd sample-images/osbuild-manifests

make cs9-ridesx4-developer-regular.aarch64.simg

and

make cs9-ridesx4-developer-ostree.aarch64.simg

but you can also build vm versions of these via:

make cs9-abootqemu-developer-regular.aarch64.qcow2

and

make cs9-abootqemu-developer-ostree.aarch64.qcow2

[antheas]

I will jump in and say that this is often a complaint I get from users and something I experienced myself when comparing the Bazzite Deck flavor to other similar distributions when booting from external media (which is slower than an internal nvme). I got a complaint about it today as well.

Bazzite seems to boot measurably slower than other distributions. This is both true in deck (SteamOS) flavors and desktop kinoite/silverblue-like flavors.

I do not have any numbers yet to back this up and I am planning to look into it in the next 2-3 weeks. For all we know it is Bazzite itself that is the issue.

We are not using composefs. I want to say that the hard link dir should theoretically be just as fast as a normal boot.

Just jotting this comment down before I forget.

[ericcurtin]

I will say at this point, these performance metrics are out of date... There are performance benefits to composefs...

composefs without fs-verity should actually be faster than ext4, xfs, btrfs (since it is erofs backed) composefs with fs-verity on though, it should be slower, checking file integrity has a cost