More buildroot fallout

[cgwalters]

OK...the switch to have a "pure" buildroot continues to have fallout, https://github.com/coreos/rpm-ostree/pull/2731 is just the most recent.

I am floating around the solution space trying to satisfy a few competing constraints:

• I'd really like to avoid running yum install in CI jobs because it can be a large source of flakes and slowness. It's just all around better if our CI jobs purely pull containers and not containers and packages; we can also consistently version our containers and not have to think about package versions too
• It's good if the buildroot is small, because well we shouldn't need the world to build
• On the other hand, unit tests...are valuable and tend to have much more dependencies. Projects like ostree-ext want /usr/bin/ostree for their unit tests, and ostree itself wants e.g. gjs and pygobject.
• rpm-ostree in particular has grown other dependencies for its build process, including e.g. createrepo_c

Lately I'm thinking that it's basically rpm-ostree/ostree that are special; ignition, coreos-installer for example just need Go and Rust compilers.

Soooo...I'm thinking of just adding a quay.io/coreos-assembler/fcos-rpmostree-buildroot:testing-devel that is FROM our existing buildroot, stick it in rpm-ostree git master. And then we drain off the other random dependencies from here. For now ostree would reuse the rpm-ostree buildroot.

Conceptually though we could try to split up build and unit test dependencies but down that road is a lot more pain.

On the other hand, it would be really good to try to push towards more "installed tests"...

[jlebon]

Soooo...I'm thinking of just adding a quay.io/coreos-assembler/fcos-rpmostree-buildroot:testing-devel that is FROM our existing buildroot, stick it in rpm-ostree git master. And then we drain off the other random dependencies from here. For now ostree would reuse the rpm-ostree buildroot.

Idea makes sense to me. Though maybe we should just push this further: we have fcos-buildroot just essentially be "@buildsys-build + rustc + golang" and most (every?) projects define their own buildroot images that live in the individual repos deriving from the FCOS buildroot. So maintenance-wise, the base sharing still makes it easy to have some centralized management, but every repo can tweak things as needed. The major value of ensuring the buildroot is using the same $releasever as testing-devel is still there.

Apart from CI and bandwidth, project-specific images also makes it easier to customize for contributors.

[cgwalters]

Well...I think e.g. having say clang-analyzer and the ASAN libs in the buildroot is part of the whole idea of CI. (And Fedora isn't covering this at all, it's up to us upstream). We also of course don't have much C code other than ostree/rpm-ostree but there is a tiny bit in Ignition.

I'd hope to avoid having too many per-repo images; it feels much simpler to fall back to yum install per repo.