Support rpm deployment #170

Open Cynical-Optimist opened 4 years ago

See original issue on GitLab In GitLab by [Gitlab user @jjardon] on Dec 12, 2017, 20:12

In GitLab by [Gitlab user @LaurenceUrhegyi] on Dec 13, 2017, 09:46

[Gitlab user @jjardon] if you have thoughts on the implementation, please share these, as I know [Gitlab user @jonathanmaw] has some background here and may be looking at this soon: he may have some ideas that cross over with yours.

In GitLab by [Gitlab user @Conan_Kudo] on Jan 3, 2018, 12:21

What exactly does this mean, from a BuildStream point of view?

In GitLab by [Gitlab user @tristanvb] on Jan 3, 2018, 13:13

[Gitlab user @Conan_Kudo] It basically means creating an rpm deployment kind of plugin, similar to the one we have in the bst-external repo for debian packages.

These will basically take the output artifact from a given module build, and consume public data from the given element to fill in any blanks needed to generate the package metadata.

This could arguably be filed against the bst-external repo directly, but then we'd just have all these bug trackers to keep track of flying around :)

In GitLab by [Gitlab user @Conan_Kudo] on Jan 3, 2018, 13:21

[Gitlab user @tristanvb] So this is different from supporting RPMs as input or building RPMs, then?

In GitLab by [Gitlab user @tristanvb] on Jan 3, 2018, 13:26

[Gitlab user @Conan_Kudo] It's different from supporting RPMs as input, but it is pretty much exactly "building RPMs"

For a package deployment plugin like this, naturally one needs to have the element which builds RPMs, depend on a runtime which provides rpmbuild, usually you want this runtime to match exactly the rpm based distro that you want to deploy to (this is not much different than using an autotools plugin, and assuming that the user is depending on a runtime that has tooling such as autoconf, automake and make).

In GitLab by [Gitlab user @Conan_Kudo] on Jan 3, 2018, 13:31

Well, at least for constructing environments and getting dependencies, you can use DNF on Fedora, Mageia (6+), and openSUSE (Tumbleweed and Leap 15.0), which means you also have a Python API to work with.

As for building RPMs, that should be relatively painless (assuming you want to translate YAML to a spec). If you want to "construct" the RPMs directly, that'd require wiring into librpm itself (probably through its Python bindings).

As for making repositories, there's createrepo_c unless you want to build the metadata yourself (which you could!).

In GitLab by [Gitlab user @tristanvb] on Jan 3, 2018, 13:45

Just noting here, it is not desirable to use host rpm, so there will be no dependency that a user have rpm installed on their host - every mutation in the BuildStream pipeline needs to happen inside the controlled sandboxed environment.

Yes, we generate a spec file from the YAML, specifically a rather lax and uncontrolled part of the YAML that we call "public data", and we feed that to whichever rpm mechanics exist in the controlled runtime (the python plugin is mostly responsible for interpreting that YAML, generating a spec file and invoking rpmbuild in the isolated build sandbox).

There is an earlier, more rigid version of this approach already in https://gitlab.com/baserock/ybd, which mostly does the right thing but certainly needs some tuning (for the purpose it is used now, it is currently used in some production systems and working well, but I expect it to break on corner cases).

In GitLab by [Gitlab user @Conan_Kudo] on Jan 3, 2018, 13:53

So does that mean you need to do your own dependency resolving and whatnot? Or are you going to grab a generic preinstall image to bootstrap from?

The way you do debs is through debootstrap, which implies that needs to exist on the system somehow.

In GitLab by [Gitlab user @tristanvb] on Jan 3, 2018, 14:32

Generally yes it implies that something like debootstrap should be employed to generate a consistent image for the runtime. This is not always the case with rpm based distros, take for instance those crazy tizen derived things that keep rearing their head... those usually come with tarball SDKs from their ISVs.

It's also perfectly possible to build everything from a tiny runtime, including the bootstrap of the compiler, build RPM itself and configure it how you want your distro to be, and then deploy everything you need for your entire custom distro after everything has been built, using your own custom built runtime as a base.

As long as host tools were not used, there are many ways to fry the cat...

In GitLab by [Gitlab user @Conan_Kudo] on Jan 3, 2018, 15:18

So as long as you can bootstrap up to DNF (or even just microdnf if I could figure out what I'm doing wrong in this PR), you should be able to pull this off for any arbitrary architecture.

In GitLab by [Gitlab user @tristanvb] on Mar 31, 2018, 10:38

changed title from {-[Feature Request] -}Support rpm deployment to Support rpm deployment

buildstream-migration / bst-staging

Support rpm deployment #170