Open achilleas-k opened 9 months ago
Looking more into Unified Core and I realise now (which I should have realised sooner) that this option doesn't create different system trees or artifacts, but it does things differently, more safely and sanely. A big part of that (as I understand it) is containerising some of the actions performed when composing the ostree commit, which we already do and is the source of the issue. It makes me wonder if this is worth the effort if we're already doing things correctly in osbuild.
I understand that there's a desire to drop the old code paths from rpm-ostree and make UC the only way to compose commits, but is it worth locking out (or at least making it difficult) to run it in cases where the environment is already sane?
I'm going to keep looking into this, maybe I misunderstood some things and this is in fact not hard to get working with the way we do things in osbuild. But given the above, I don't think this should be a high priority unless the non-UC way of composing is going away very soon.
@cgwalters @runcom wdyt?
there's a now old issue somewhere about unified-core and I thought the culprit of the issue was around rpms, perhaps related https://github.com/osbuild/osbuild/issues/1134 (?) - I may completely mistaken - also, @paulwhalen while switching fedora iot to unified core experienced some selinux issues too
also cc @travier
...I don't think this should be a high priority unless the non-UC way of composing is going away very soon.
My fear is exactly that; the non-UC mode of composing ostree commits is nearly deprecated and isn't being fully tested (or at all?) as the ostree stack changes. Based on discussions with the CoreOS folks, they aren't interested in continuing to support the non-UC path and have a desire to unify the ostree-based operating systems around the UC path. (Using the UC path also allows ostree-based operating system to adopt the use of bootupd for bootloader updates, which is a large gap in Fedora IoT/R4E right now)
The right folks from the CoreOS team have been tagged here, so I'll let them weigh in with any corrections or additional context.
The unified core mode goes from RPMs to ostree commit without any steps in the middle, so you can not install the RPMs, do some postprocess, etc. like the current ostree mode does. It needs to go directly from the rpm-ostree manifest + RPMs to the ostree commit.
Ideally, we would directly support only rpm-ostree compose image
instead of doing this in multiple steps.
Looking more into Unified Core and I realise now (which I should have realised sooner) that this option doesn't create different system trees or artifacts, but it does things differently, more safely
Right, like running each %post
script in its own container. Now honestly, it's a lot of reinvention of stuff. It's had some benefits but clearly is a huge maintenance burden and code not shared with dnf or osbuild.
(Using the UC path also allows ostree-based operating system to adopt the use of bootupd for bootloader updates,
There's not some proprietary secret sauce though in rpm-ostree; I think ultimately the only load-bearing thing unified core mode does differently here (related to https://github.com/coreos/bootupd/issues/441 ) is basically that in unified core mode rpm-ostree ensures that the shim/grub RPM files that install into /boot
are moved into /usr/lib/ostree-boot
instead. Adding a bootupd stage into the existing osbuild logic would likely literally just be mkdir -p /usr/lib/ostree-boot && mv /boot/* /usr/lib/ostree-boot
or so after the RPM installs are done.
Or I guess it'd likely make even more sense to just stick that code into bootupd too.
Looking more into Unified Core and I realise now (which I should have realised sooner) that this option doesn't create different system trees or artifacts, but it does things differently, more safely
Right, like running each
%post
script in its own container. Now honestly, it's a lot of reinvention of stuff. It's had some benefits but clearly is a huge maintenance burden and code not shared with dnf or osbuild.
And that is essentially what my question/comment hinges on. It sounds like in osbuild's case there's no technical need for unified core, and it seems bootupd doesn't strictly depend on it either. So the main issue remains that the non-unified-core paths are deprecated, untested, and soon to be removed (?), which brings me to a follow-up question: Is there any potential for a more fine-grained/separated workflow that would make composing with rpm-ostree with unified core easier to work with in environments that don't need (or are incompatible with) some of the steps in the whole process?
Or maybe, a more general question: does this ability already exist when running things using the "Granular tree compose" (https://github.com/coreos/rpm-ostree/blob/main/docs/compose-server.md#granular-tree-compose-with-installpostprocesscommit) like we do in osbuild? I'd love to get an overview of what unified core does differently so we can figure out where the issue I'm seeing here is coming from, but also what's happening differently at every stage. For reference, when we're composing ostree commits in osbuild we do (very roughly):
rpm install
into a tree.rpm-ostree compose postprocess
on the tree.rpm-ostree compose commit
The postrpocess
stage fails with --unified-core
, but the commit
stage doesn't (if postprocess
is first run without it).
I feel like we're having a similar conversation to what was discussed here (since Antonio linked it earlier): https://github.com/osbuild/osbuild/issues/1134#issuecomment-1281459308
So the main issue remains that the non-unified-core paths are deprecated, untested,
We can also just choose to start testing the osbuild path more in upstream rpm-ostree, it's just work but that'd probably close off the main concerns.
and soon to be removed (?)
We can't drop it in the near future I'd say. And ultimately the "unified core" path is hard to replicate in an existing container. What seems most sustainable long term is to drive functionality we need more directly into dnf/libdnf/rpm and the scripts, and have a common "postprocess the filesystem tree" tool.
Given all that, and the current priorities, I'm understanding that we can push Unified Core way down the priority list for osbuild. And it also seems like UC really isn't a requirement for making images "properly" or making them compatible with bootupd, which was the source of all the major concerns for me.
What seems most sustainable long term is to drive functionality we need more directly into dnf/libdnf/rpm and the scripts, and have a common "postprocess the filesystem tree" tool.
Now that would be great!
While adding
--unified-core
to osbuild, there are selinux issues runningrpm-ostree compose postprocess
with--unified-core
enabled. I suspect it's related to how we run stages in containers.The relevant bits are:
/var/lib/...
being read-only is clearly an issue but what I don't know is if this is a var that's created and managed by rpm-ostree or our own stage/var
.From what I understand, one of the things that unified core does is run certain tasks in containers, so there's clearly an overlap and source of potential conflicts there.