Open schaefi opened 8 years ago
RPM-OSTree package for openSUSE Leap 15.0 / Tumbleweed: https://build.opensuse.org/package/show/home:Pharaoh_Atem:SUSE_RPMOSTree/rpm-ostree
Docs and examples:
@Conan-Kudo I am still wondering two things:
What actually means supporting rpm-ostree in KIWI? Probably this caused by my lack of knowledge regarding rpm-ostree. I assume this is a new package manager right? or at most an extension of the feature set of DNF and/or any other that has some layer of cooperation.
Which is the use case? According to what I understand this is not much different of building a root-tree over an ostree, thus building rootfs in some container like layered filesystem, correct? So I am wondering what is actually the result of using rpm-ostree as a package manager for an e.g. Fedora build. Mostly because if rpm-ostree is a package manager (from KIWI POV) then it should not have impact on the build procedure. It should still be possible to build isos, vms, oem, etc.
I can imagine features like usign the derived_from
attribute to import an ostree and create a new images out of it by adding a new transaction on top with rpm-ostree. Am I pointing to the right direction?
Also stuff like making each <packages>
section as an individual transaction, even I am having hard time to see the use case and potential benefits of it.
We need to refine this support to narrow what actually means and also understand the use cases we are covering with it. @Conan-Kudo would be appreciated if you can help us on that regard.
RPM-OSTree produces an OSTree, which is a special rootfs type representing a largely immutable operating system environment. The rpm-ostree
program has features for producing and managing OSTrees from RPM content. It does leverage DNF stack libraries, but it does not have the same interface as DNF.
What it would mean from a KIWI perspective is that it would be able to orchestrate creation of OSTrees and applying them to the desired medium so that they can be used in the same way traditional Linux system trees are.
Today, there aren't any tools that make it easy to produce RPM-OSTree based things other than VM disk images with cosa. As KIWI can produce way more types and is easier to leverage (think OBS and similar), that would give it a major leg up for producing immutable infrastructure OSes. As it is, I currently cannot create a live ISO of an rpm-ostree based system, nor can I easily create layered OSTree repositories for feeding update repos or derivative distros.
@cgwalters and @jlebon would have a better idea of the exact mechanics here, as they're the current developers of rpm-ostree.
I currently cannot create a live ISO of an rpm-ostree based system,
cosa buildextend-live
is used to generate the "live ISO" linked from the FCOS builds.
Another player in the field of "packages" is luet
the term package in this regard means a container+plus_metadata
the term package in the context of rpm-ostree is an extended format for rpm (iirc)
In any case kiwi has no support for the sophisticated new idioms of packages which are actually sub-systems. As of now I still don't know if there is a high demand to natively support these type of package managers in kiwi.
kiwi itself is flexible enough to use any os-tree you will give it. Therefore it's possible to combine rpm-ostree or luet or whatever else will come providing "systems" to let those tools create/manage a new root tree and let kiwi take over only for the task to create images out of it
From the perspective of a stacked solution I currently don't see the need to add work into kiwi.
However it might be a good idea to discuss a change in kiwi that makes the "installation of packages" as we see it now an optional step. Currently the following is possible with kiwi:
$ kiwi-ng system build --description ... --target-dir ... --allow-existing-root
This will re-use an eventually existing root tree no matter who created it, luet, rpm-ostree, etc... But the provided description still needs:
<packages>
section even if you don't install anything in addition to the existing treeThis tells us that kiwi itself expects some connection to repositories and packages as they are provided by the distribution vendors. The discussion we can have here is if we want to weaken this concept if customers can provide a system root tree to work with ?
Your feedback is welcome
Thanks
However it might be a good idea to discuss a change in kiwi that makes the "installation of packages" as we see it now an optional step. Currently the following is possible with kiwi:
$ kiwi-ng system build --description ... --target-dir ... --allow-existing-root
This will re-use an eventually existing root tree no matter who created it, luet, rpm-ostree, etc... But the provided description still needs:
* at least one old school package repository from the distribution * at least on `<packages>` section even if you don't install anything in addition to the existing tree
This tells us that kiwi itself expects some connection to repositories and packages as they are provided by the distribution vendors.
I have thinking a quite a lot within the last few days regarding these topics. First things regarding the demand that there might be regarding these sort of ostree/luet/container like packages do think there is no demand on adding support at package manager level. It is certainly possible (I already did a successful PoC for Luet as a hobby thing) but even I believe it might make sense in some specific use cases I also think this is not best approach to take advantatge of these sort of technologies from an appliance building PoV.
What is appealing from containers? IMHO from developers PoV, beyond the isolation and the DevOps trends, something that is nice is the stacking or layering concept. You never start from scratch and you can always add some little salt to anything and turn something that was close to what you wanted to something else that really matches your needs or expectations. With all that you probably don't really need to care much about how your base layer was created, what is actually including and all configuration details. Damn easy to get started.
What if we make use of container technologies to build the idea of derived images for any type of image. Imagine that after the prepare step the created root tree (including all KIWI artifacts) is kept in some ostree or OCI storage/registry. So we could eventually rebuild the exact same image by just pulling in this stored prepare step (it would already contain the XML description and any other artifact that KIWI requires to create an image). Alternatively, we could recreate the whole tree (re run the prepare step on top of the already provided tree so, for instance, it could upgrade packages and rebuild again the provided image). But what if we also provide a delta of a new XML description? What if in that procedure we allow to include XML description artifacts that are simply merged with the base XML, so on the fly a new full XML gets created and applied on top of the imported rootfs.
Imagine we have a Live image, with a <type>
like:
<type image="iso" flags="overlay" firmware="efi" kernelcmdline="splash" hybridpersistent_filesystem="ext4" hybridpersistent="true">
<rootfs-base keep="oci:myliveimage:latest"/>
</type>
With all the packages and so one. So the prepare step of this image could eventually push the root-tree into an OCI image myliveimage:latest
and the end of it. So a kiwi-ng system build
call, as part of the build process, stores the WORKING root-tree in some OCI storage for any potential later reuse.
Then imagine I want to tweak this image and build a new one on top of this one. Imagine we could do something like:
<image>
<derived from="myliveimage:latest"/>
<packages type="image">
<package name="my-new-branding"/>
</packages>
<packages type="uninstall">
<package name="my-old-branding">
</packages>
</image>
And a kiwi-ng system build
call on that system could eventually result into the former base XML with these additional packages sections appended and executed the build over the already provided root-tree as it happens with --allow-existing-root
. Taking it one step further this could even be used to build a different type of image or build the same image but using dmsquash instead of overlay... The obvious problem here is how to define the XML merge and some details of the build steps (create only vs prepare and create). So some questions to figure out, but I believe it should be possible to find some simple and comprehensible procedure for those.
The discussion we can have here is if we want to weaken this concept if customers can provide a system root tree to work with ?
So, what if there is no need to weaken this concept, what if we enforce the imported rootfs to be a fully qualified KIWI rootfs, including the XML, config.sh, etc.
Something like that provides 100 ways to shoot yourself, of course, but you can already shoot yourself quite easily at the moment if you start using all the schema possibilities randomly.
This is probably a dumb idea, but I believe there should be a way to build images on top of another image and at the moment this is the only "sane" procedure that came into my mind :P
Any thoughts?
If it helps, iit's somewhere on the roadmap for me to better integrate (rpm-)ostree with container images, and make the experience support something much more like what you're talking about - a Dockerfile
-style FROM fedoracoreos
then do stuff. A lot of stuff involved there though.
I like the idea to preserve a kiwi root tree in a registry. Correct me when wrong, the stored root tree would be a docker container and can also be used as such but its main use case is to preserve a root tree from a kiwi stage for further operations as you mentioned, right ?
I think this is a great concept to implement the "reproducable build feature" you had in mind earlier and allows for more use cases.
From a docker registry perspective the pushed data is probably not very useful as a container. Thus I was thinking if providing the tree as an ostree would fit in better. However I must say that using a docker registry and the handling of data as a docker container is pretty straight forward and easy compared to an rpm ostree. But this could also because of my lack of knowledge on the ostree end
But what if we also provide a delta of a new XML description?
I had played with this in the past and I must say it was very hard to achieve this on the XML level. A generic approach that allows kind of a XML merge is possible but ends in a cumbersome to use concept. The delta part will not validate the schema only the combined information will be valid. The model I came up then as an alternative was the profiled sections, but this doesn't play well on a delta concept. So the idea is nice but for a delta concept I suggest to use a meta language on top which can be composed into a complete XML description for validation. Similar to the C prepocessing
Closing the issue since we are definitely deviating from the original topic and no progress is expected. Also related ideas are discussed in #1771
I was talking to @travier today in the Fedora KDE SIG meeting, and he said he'd be interested in helping us understand how to add RPM-OSTree support to KIWI.
So, I have been playing around a teeny-tiny bit with ostree and unfortunately I am still far from really understanding it, but this is roughly how I got it:
What I have not found out yet, is how you create a disk/iso image from this.
However, an ostree commit is afaik just a tarball of the filesystem, so if we can create a VM/iso from a tarball, then we should be able to build a system with ostree enabled. But admittedly, this is all very hand-wavy at the moment.
This code from osbuild shows how they generate an rpm-ostree commit, and that might be useful for coming up with a way to do it in kiwi.
Relying on deep kernel functionality to introduce unncessary "immutable architecture" to a build environment just seems painful. Almost everything rpm-ostree provides can be done with distinct directories with hardlinks between them, a much older and more stable technology with no requirement for introducing potentially fragile new technologies. If it's necessary to make the components of such an environmentimmutable, use "chattr". But don't hold up desirable image building tools on complex technologies which don't actually help perform the task.
That is pretty much how rpm-ostree works, actually. It's heavily managed hardlink/reflink farm.
Yep, though I'd clarify rpm-ostree uses ostree, which is what does all that. ostree is really a very overgrown wrapper for the link()
system call along with the ability to fetch via http.
FYI: unfortunately, RPM-OSTree is now fundamentally incompatible with openSUSE, so I have stopped attempting to make it available for the distribution. Since openSUSE now uses /usr/etc
, it is not compatible with how rpm-ostree
uses it. Development of the RPM-OSTree backend cannot be done on openSUSE at this time.
If anyone is looking for a reference implementation, the Automotive variant is basically boiled down to a one liner:
https://sig.centos.org/automotive/building/
git clone https://gitlab.com/CentOS/automotive/sample-images.git
sudo sample-images/auto-image-builder.sh cs9-qemu-minimal-ostree.x86_64.qcow2
If you grep the output:
$ grep ^org.osbui.*ostree ~/cs9-ridesx4-minimal-ostree.aarch64.aboot.simg1711378035.txt
org.osbuild.ostree.genkey: 039b0760153259cd15a5c28ded1f190fbf6e828f76ff9925a4a205ebf7a9dfb6 {
org.osbuild.ostree.preptree: 865c268aca38255b56d4dc7726297c6db8f4f837e0cd37b176a442bd7fef4a07 {
org.osbuild.ostree.init: 8704ff9ae85760c85475f06fe26683a9d518ea103699b1ac0462b978bba28703 {
org.osbuild.ostree.commit: 3236b1465f7e3ec24618a6350532276fe055419033c9d4a9fb77b93811a91e75 {
org.osbuild.ostree.sign: bef42772d505ad28285e67fae9796a69f904e9a83f95bceb172cbe0c7aaf8bd6 {
org.osbuild.ostree.init-fs: baf0238cd955d4ebff78feba4360da63cd0a3c0e6a6c284163eaeab2db9888fd {}
org.osbuild.ostree.pull: d4c638e615651f975bbe62a3395f3aa5dda4ab099db33df7fd71c4ebfadb7a09 {
org.osbuild.ostree.os-init: 77bb8ccfc5c1959ed4b424cbb31a4f13d6253077b57254fbb9ea15ddcf8e493f {
org.osbuild.ostree.config: 90430db80ed476ca8a8c709822908dd5f78762e144b8a1db3b58505e1980d583 {
org.osbuild.experimental.ostree.config: 9a12f564fc854ed001e98aaa69eaa83579d917dcb10f08b3725442b6229645b2 {
org.osbuild.ostree.remotes: 675cff32c80db8f17a54a254f94ec317eabb814cac2ce3c06bf356011dec6d74 {
org.osbuild.ostree.deploy: db3f60947577dd6c09b7f496f2b69e97c4d599aa297a1e32cfa2edcf10e6213b {
org.osbuild.ostree.fillvar: 5f63f87ccfe85cf26e22980ad6b907092b18891fc3e02c7aa8917baf93a410d8 {
org.osbuild.ostree.selinux: 25e239b11a1d3b1e1c3b09f486a78deb7a379d8dfbb0fb9b118fd8a43b4e27e8 {
you should be able to see the general flow.
These python scripts that power this are around here basically:
/usr/lib/osbuild/stages/org.osbuild.ostree.preptree
Since openSUSE now uses /usr/etc
Link/more info?
Since openSUSE now uses /usr/etc
Link/more info?
openSUSE now uses /usr/etc
to store vendor configs of stuff that is in /etc
, and my understanding of the situation is that this is the location RPM-OSTree uses to divert and archive pristine copies of /etc
. At least, I think that's what you told me when you removed /usr/etc
from Fedora's filesystem
package years ago.
Since RPM-OSTree does not use a private hierarchy for this, it effectively has become incompatible with openSUSE (not that there weren't already other issues with its usage of %verifyscript
and a few other things, but this is the final nail in the coffin).
I've been reading on how kiwi works and I think that we should do the following:
We will add support for building ostree native containers to kiwi later. We will also need to figure out where bootupd fits in there but we can do that later as well.
Let me know what you think.
This seems like a reasonable plan. So then the end state we should be able to create live media and disk images that are RPM-OSTree backed?
I was wondering if the kiwi stackbuild plugin could already be helpful here. See the following documentation
Let's say I take this and replace the example container with an ostree native container I should be able to produce an RPM-OSTree backed image. I would be eager to try that out. Where can I find ostree containers ?
Thanks
So then the end state we should be able to create live media and disk images that are RPM-OSTree backed?
Ideally yes.
I was wondering if the kiwi stackbuild plugin could already be helpful here
I don't think this will work. Ostree needs special handling to deploy a version to a system rootfs.
Overall, we'll have to duplicate the logic from https://github.com/coreos/coreos-assembler/blob/main/src/create_disk.sh#L312..L379 (https://manpages.debian.org/testing/ostree/ostree-admin-deploy.1.en.html).
I would be eager to try that out. Where can I find ostree containers ?
https://quay.io/repository/fedora-ostree-desktops/kinoite?tab=tags
I don't think this will work. Ostree needs special handling to deploy a version to a system rootfs.
yeah I see and that matches my results. So here is what I did
vi ostree-to-live/config.kiwi
<?xml version="1.0" encoding="utf-8"?>
* build the image selecting an ostree rawhide container
sudo kiwi-ng system stackbuild --stash kinoite:rawhide --from-registry quay.io/fedora-ostree-desktops --target-dir /tmp/my-ostree-live --description ostree_to_live
This works and builds a live image, but it's huge :)
I was looking into the rootfs and I found the specials when using ostree.
total 24 dr-xr-xr-x. 2 root root 6 Mar 13 01:00 afs lrwxrwxrwx. 1 root root 7 Mar 13 01:00 bin -> usr/bin dr-xr-xr-x. 5 root root 4096 May 14 17:47 boot drwxr-xr-x 3 root root 92 May 14 17:49 dev drwxr-xr-x. 136 root root 8192 May 14 17:49 etc drwxr-xr-x. 2 root root 6 Mar 13 01:00 home drwxr-xr-x. 3 root root 38 May 14 17:49 image lrwxrwxrwx. 1 root root 7 Mar 13 01:00 lib -> usr/lib lrwxrwxrwx. 1 root root 9 Mar 13 01:00 lib64 -> usr/lib64 drwxr-xr-x. 2 root root 6 Mar 13 01:00 media drwxr-xr-x. 2 root root 6 Mar 13 01:00 mnt drwxr-xr-x. 2 root root 6 Mar 13 01:00 opt lrwxrwxrwx. 2 root root 14 Jan 1 1970 ostree -> sysroot/ostree drwxr-xr-x 2 root root 6 May 14 17:46 proc dr-xr-x---. 3 root root 18 May 14 17:47 root drwxr-xr-x. 34 root root 4096 May 14 17:47 run lrwxrwxrwx. 1 root root 8 Mar 13 01:00 sbin -> usr/sbin drwxr-xr-x. 2 root root 6 Mar 13 01:00 srv drwxr-xr-x 2 root root 6 May 14 17:46 sys drwxr-xr-x. 3 root root 20 Jan 1 1970 sysroot drwxrwxrwt. 2 root root 6 May 14 17:49 tmp drwxr-xr-x. 12 root root 144 May 14 17:47 usr drwxr-xr-x. 23 root root 4096 May 14 17:47 var
in /sysroot/ostree I could see all the different objects that makes up that ostree.
At this point we need to add the deploy functionality that you pointed me to, but thanks for sharing the containers and the link, I understood a little bit more on the concept :)
So far I can do most things using scripts and not touching kiwi code. WIP PR with the descriptions in https://pagure.io/fedora-kiwi-descriptions/pull-request/58.
This is currently failing on:
[ ERROR ]: 10:39:39 | KiwiKernelLookupError: No kernel found in /home/fedora/fedora-kiwi-descriptions/outdir/build/image-root, searched for []
as ostree does not install the kernel in /boot
but in /boot/ostree/fedora-<hash>/...
:
$ tree outdir/build/image-root/boot/ostree
[drwxr-xr-x root root 142] outdir/build/image-root/boot/ostree
└── [drwxr-xr-x root root 204] fedora-738613c8ada41ad14ff1b4bec3e06ab89c3948aed6e614844730e23de50ea337
├── [-rw-r--r-- root root 131M] initramfs-6.8.10-300.fc40.x86_64.img
└── [-rwxr-xr-x root root 15M] vmlinuz-6.8.10-300.fc40.x86_64
Does it needs to find the kernel to generate the GRUB/bootloader config? In our case, we use the BLS configs so we don't need that.
It's likely that we'll also install the bootloader using bootupd soon.
Does it needs to find the kernel to generate the GRUB/bootloader config? In our case, we use the BLS configs so we don't need that.
It does if we want to be able to build ISOs.
We need to be able to identify a kernel. In the container world the kernel is shared with the host and you simply don't care but if you want to boot a system no matter if it's an ISO or a disk image we need to be able to make a choice for the kernel and also find the kernel binary.
In kiwi the version lookup is based on the versioned directory name of the kernel in either /lib/modules
or /usr/lib/modules
and so far we trusted the sub-directories below these paths to name a kernel version e.g
/lib/modules/5.3.18-59.10-default
and the kernel version is then 5.3.18-59.10-default
In case of an ostree I don't understand how it's done. The ostree can provide a number of objects, e.g
/boot/ostree/fedora-738613c8ada41ad14ff1b4bec3e06ab89c3948aed6e614844730e23de50ea337/vmlinuz-6.8.10-300.fc40.x86_64
. To match a kernel version the name of the kernel file is the only source that gives a hint.
But how does this match the kernel version when I can have a thousand objects ?
For ostree and bootc systems, the kernel is canonically in /usr/lib/modules/$kver
- the /boot/ostree
is an implementation detail that other projects should not peek into, basically.
Alternatively of course, you can look at the bootloader entries in /boot/loader
and what those point to.
As far as I can tell, it looks like this is exactly the logic in:
but if I understand correctly, it did not look for them in the ostree deployment, but in the "sysroot", where we only have the copy in /boot/ostree
so it did not find the kernels.
But ideally kiwi does not have to find the kernel as it should let the bootloader follow the BLS config that has been written by ostree.
I'll make a temporary workaround and then we can circle back to this once I understand better how kiwi works.
With https://github.com/OSInside/kiwi/pull/2557 and https://pagure.io/fedora-kiwi-descriptions/pull-request/58, I get a booting QCOW2 image!
I made a few additional hacks and got a LiveISO working!
The next step I need help with for the Live ISO is figuring out a way to read a BLS entry from the image dir to get the ostree
kernel command line argument and "pass" them to the final kernel command line for the Live ISO GRUB config.
Another option is to read those kernel command line arguments from the image dir in the GRUB config step but so far I haven't been able to find a way to do that. Maybe I missed the right variable with the path to the root dir.
How do we pass information in general from the prepare step to the image step?
The next step I need help with for the Live ISO is figuring out a way to read a BLS entry from the image dir to get the ostree kernel command line argument and "pass" them to the final kernel command line for the Live ISO GRUB config.
There is a kiwi hook script called editbootconfig
which allows to do custom changes to the bootloader configuration. Also see: https://osinside.github.io/kiwi/image_description/elements.html. Be careful when using this script hook because it does not run chrooted
There is also the kernelcmdline
attribute in the <type>
section of the image description which allows to add custom kernel boot options to the image. However this is only useful for static data that you would like to pass to the kernel, dynamic information such as UUID's or similar doesn't make sense to be placed there
How do we pass information in general from the prepare step to the image step?
The job of the prepare step is to create the data of the image root-tree. Any information that is needed to let the create step pass is expected to be present inside of the root tree such that the create step can do the job. Sometimes people create temporary (present during build time) data and delete them later by one of the available script hooks. Usually this sort of tricks are only needed for bootloader related specialities which requires to be done late in the processing.
In this particular case for building a live ISO, kiwi uses its own grub template which you can overwrite with your own template file. See https://osinside.github.io/kiwi/image_description/elements.html and search for grub_template
Hope this helps
The next step I need help with for the Live ISO is figuring out a way to read a BLS entry from the image dir to get the
ostree
kernel command line argument and "pass" them to the final kernel command line for the Live ISO GRUB config.Another option is to read those kernel command line arguments from the image dir in the GRUB config step but so far I haven't been able to find a way to do that. Maybe I missed the right variable with the path to the root dir.
How do we pass information in general from the prepare step to the image step?
In osbuild, the scripts assume one version of the OS in installed in the rootfs, so based on that assumption they just populate the grub bls file with values that make sense from querying the filesystem. So we can make the same assumptions. If we have two kernels initially or example rpm-ostree doesn't like it anyway.
And when we do need multiple versions of the OS, it's easier just to do multiple builds.
I am thinking of the Asahi case more than the .iso case though which is likely missing the point.
However this is only useful for static data that you would like to pass to the kernel, dynamic information such as UUID's or similar doesn't make sense to be placed there
That's the case here. I don't know the ostree<hash>
command line parameter in advance and I need to read it from the BLS config that ostree generated to add it to the GRUB config used for the ISO.
So, to make this more integrated, I'm looking at creating a new "package manager" that would take a special "package" for the "image" type/phase that would be the container image URL. I still need the bootstrap package manager & packages to get the tools in the bootstrap env to install this container.
Not sure if that makes sense / how to integrate that properly?
Yes, that makes sense. You'd probably want to make the bootstrap package manager just be a stub that uses the DNF class, and then for the image section, you'd use rpm-ostree that was installed in bootstrap.
With rpm-ostree, you can create versioned filesystem trees to provide atomic, reliable upgrades. "git-like" management of appliances would be pretty sweet.