where is the source code for building the silverblue installer image?

[omac777]

Hi there, I have many questions about how the image for silverblue installer is created, how the installer determines what device to partition and what specific commands are used to partition the btrfs volume and subvolumes.

I think the best way to answer the question would be to find the source code for the building of silverblue installer image. Please could somebody help me to find this source code? Thank you in advance.

This is all in the hopes to help get a fedora silveblue installer image prepared for the Starfive VisionFive 2. There is a fedora 33 workstation(NOT SILVERBLUE) image I believe running on VisionFive 1, but nothing for the VF2 yet. That f33 image for vf1 doesn't use btrfs volume/subvolumes, but it was the closest reference point to fedora I could find to help prepare a vf2 fedora silverblue installer.

To be honest, I've been jumping back and forth between debian image-build/vmdb2/anaconda/kickstart/osbuild/compose and I'm getting dizzy LOL.

I have played with debian live-build, archlinux archiso tools to customize installer images in the past which is why I found the courage to attempt to create a kickstart file for vf2.

[travier]

The installer ISO is built using lorax (https://github.com/weldr/lorax) in Fedora Pungi (https://pagure.io/pungi-fedora/blob/main/f/fedora.conf#_854) with some ostree specific lorax templates (https://pagure.io/fedora-lorax-templates).

I've tried to get a mostly complete example working to build one in https://gitlab.com/fedora/ostree/ci-test/-/blob/main/justfile#L277.

[omac777]

Mr. Ravier I really appreciate your work. Thank you.

[AdamWill]

you can also refer to the openQA test that builds an ostree and then embeds it in an ostree installer image:

https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/tests/_ostree_build.pm

it's written in perl, but it's mostly just running shell commands. And a hacky Python script which does a job that Pungi does in the official process.

[omac777]

Thank you Adam Will, you are very helpful :) I appreciate it very much. Am I looking at the right logs? https://openqa.fedoraproject.org/tests/1771475/file/_ostree_build-lorax.log https://openqa.fedoraproject.org/tests/1771475/file/_ostree_build-ostree.log If I understood correctly, these are all the steps to create the image that includes the ostree stuff for fedora silverblue. Now all I have to do is understand what each step in these logs mean.

[AdamWill]

those log files are the outputs of the tools; the test uploads them for reference if we need to debug things or look into differences between the builds from different updates or anything like that. The actual commands to build an ostree and then build an installer image with that ostree embedded are found in the test source that I linked, https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/tests/_ostree_build.pm . For a quick guide to what that's doing - anywhere it calls get_var it's reading a configuration setting for the specific instance of the test that's running (e.g. finding out if it's running on a Rawhide update or a Fedora 37 update, what the identifier of the specific update it's running on is, or what arch the update it's running on is). Any time it does assert_script_run it's literally running the string that follows as a command in a shell - those are the actual commands that do the work of building the image.

The stuff about the /var/tmp scratch disk is not necessary in general, it's just there to make sure the test has enough disk space for the operation (the system disk used by these test runs is quite small and does not have enough free space). The work to include the 'advisory' and 'workarounds' repo in the operation is to ensure the packages from the update being tested actually get used in the image build process: if you purely want to recreate the official process without changing the image in any way you don't have to do that stuff, but if you do want to pull additional/replacement packages into your build, that shows you how to do it. Some of the stuff is just there to upload log files for debugging and ensure the test fails properly if a command goes wrong; again you wouldn't need to do that yourself.

So really the process boils down to: checkout workstation-ostree-config, set up additional repo files and edit them into the appropriate fedora-(flavor).yaml file if you want to do that, set up an empty ostree repo, run rpm-ostree compose with the appropriate args pointing to the empty ostree repo and the .yaml file. That gets you your ostree.

Now checkout the lorax template repo and pungi-fedora repo, then do something rather hacky to generate the appropriate args for a lorax command from the pungi-fedora config files (this part is done by pungi in the "real" process, but I don't want to set up a whole pungi config in this test process, so I wrote a hacky python script to do it instead; you can work out the args manually if you like, or you can just trust my script to do it for you, note the script does blindly trust that it can execute chunks from the pungi-fedora config files as Python code, if you run it in a production environment you should be really sure that nobody has snuck anything nasty into the config files first), then run lorax with all the appropriate args figured out by the script plus a few more, which produces the installer image with the ostree embedded into it.

Simple, right?! OK, yes, not very simple. Sorry, this is how building OSes tends to be :P It took me about a week to get this working in openQA, so it might take you a while to crack it even with the references, sorry about that.

[AdamWill]

If you want to see what all the commands for a specific run of the test work out to after all the variable logic and script running and all the rest of it, you can get the autoinst-log.txt file from any execution of the test, e.g. https://openqa.fedoraproject.org/tests/1771475/file/autoinst-log.txt , and search it for the commands. It's a very verbose log file, but you should be able to find the relevant bits. So e.g. you can search for lorax -p and find that the full lorax command for that specific execution of the test worked out as:

lorax -p Fedora -v 39 -r 39 --repo=/etc/yum.repos.d/fedora-rawhide.repo --variant=Silverblue --nomacboot --buildarch=x86_64 --volid=F-Silverblue-ostree-x86_64-oqa --logfile=./lorax.log --rootfs-size=8 --add-template=/fedora-lorax-templates/ostree-based-installer/lorax-configure-repo.tmpl --add-template=/fedora-lorax-templates/ostree-based-installer/lorax-embed-repo.tmpl --add-template=/fedora-lorax-templates/ostree-based-installer/lorax-embed-flatpaks.tmpl --add-template-var=\"ostree_install_repo=file:///var/tmp/ostree/repo\" --add-template-var=\"ostree_update_repo=https://ostree.fedoraproject.org\" --add-template-var=\"ostree_osname=fedora\" --add-template-var=\"ostree_oskey=fedora-39-primary\" --add-template-var=\"ostree_contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist\" --add-template-var=\"ostree_install_ref=fedora/rawhide/x86_64/silverblue\" --add-template-var=\"ostree_update_ref=fedora/rawhide/x86_64/silverblue\" --add-template-var=\"flatpak_remote_name=fedora\" --add-template-var=\"flatpak_remote_url=oci+https://registry-no-cdn.fedoraproject.org\" --add-template-var=\"flatpak_remote_refs=runtime/org.fedoraproject.Platform/x86_64/f37 app/org.gnome.baobab/x86_64/stable app/org.gnome.Calculator/x86_64/stable app/org.gnome.Calendar/x86_64/stable app/org.gnome.Characters/x86_64/stable app/org.gnome.Cheese/x86_64/stable app/org.gnome.clocks/x86_64/stable app/org.gnome.Connections/x86_64/stable app/org.gnome.Contacts/x86_64/stable app/org.gnome.eog/x86_64/stable app/org.gnome.Evince/x86_64/stable app/org.gnome.Extensions/x86_64/stable app/org.gnome.font-viewer/x86_64/stable app/org.gnome.TextEditor/x86_64/stable app/org.gnome.Logs/x86_64/stable app/org.gnome.Maps/x86_64/stable app/org.fedoraproject.MediaWriter/x86_64/stable app/org.gnome.NautilusPreviewer/x86_64/stable app/org.gnome.Weather/x86_64/stable\" --repo=/etc/yum.repos.d/advisory.repo --repo=/etc/yum.repos.d/workarounds.repo ./results

various things in there change depending on the arch, the Fedora release, the variant, and if the pungi-fedora config changes (e.g. to add or remove embedded flatpaks). oh, and the quotes probably wouldn't need escaping if you were running it directly.

[travier]

We are now looking at KIWI for building Live ISO for Atomic Desktops. This is now tracked in https://gitlab.com/fedora/ostree/sig/-/issues/32 thus closing this one. Thanks for the report.