Improve guidelines for mock-based chain builds

[ncoghlan]

I'm currently aiming to create a custom download/build config that allows a fully automated build of the rh-python35 recipe in a clean mock chroot [1]. This is helping to find some previously unhandled circular dependencies in the recipe itself, but I've also run into some problems with getting the built RPMs properly installed along the way.

The key issue is that the --postinstall option to mock does a yum install not a yum reinstall, which means that RPMs later in the build chain won't actually replace the ones earlier in the build chain.

So I'm thinking that to do this robustly, I'm going to have do the following:

1. Use --resultdir to put the built RPMs inside the working directory rather than leaving them in the mock chroot
2. Use mock --shell to invoke yum reinstall rather than yum install for each of the RPMs

I'm filing this as an issue here, as even if we decide it isn't applicable to the default mock build strategy, it would still be useful to document as something to watch out for when writing custom build strategies.

[1] https://github.com/ncoghlan/pyscl-devel/blob/master/rpmlb/sclorg-distgit-download.yml

[junaruga]

Hi @ncoghlan using mock -n option in the custom file, does not work for your situation [1]? Why do you want to use mock ---postinstall option in custom file? I did not suppose that the option was used.

[1] https://github.com/sclorg/rpm-list-builder/pull/68/files L9

[ncoghlan]

The packages in the recipe depend on each other as build dependencies, so after each one is built, they need to be installed back into the mock chroot in order for later builds to succeed.

However, some of the earlier builds (which selectively disable features to break cyclic build dependencies) also need to be replaced by later ones (which rebuild with the same NEVRA, but updated build settings that enable all the features)

[ncoghlan]

I've managed to put together something that seems like it should work reliably: https://github.com/ncoghlan/pyscl-devel/blob/master/rpmlb/sclorg-distgit-download.yml

However, I'm sure it can be made more efficient than it currently is (e.g. by injecting a helper script into the chroot that does all the necessary checks and command for all of the RPMs in a single mock --shell call, rather than the current approach of making multiple such calls per RPM)

[ncoghlan]

Alas, it turns out even that is inadequate - I think the rpm -qp call is querying the package file itself, rather than extract the package name and using that to query the chroot's RPM database.

So I think I'm going to have to stop messing about with shell scripts and instead create a more reliable Python script that I can inject into the chroot and then pass it the list of RPMs that each SRPM build creates.

[junaruga]

The packages in the recipe depend on each other as build dependencies, so after each one is built, they need to be installed back into the mock chroot in order for later builds to succeed.

I can understand the situation. Using mock -n and bootstrapping logic in the recipe file might be good for your situation.

You may refer below recipe file for Ruby on Rails. https://github.com/sclorg/rhscl-rebuild-recipes/blob/master/ror.yml

rubygem-rspec-* (such as rubygem-rspec-support) packages "depend on each other as build dependencies".

However, some of the earlier builds (which selectively disable features to break cyclic build dependencies) also need to be replaced by later ones (which rebuild with the same NEVRA, but updated build settings that enable all the features)

I guess that that only --build koji (KojiBuilder) option recently supported above situation.

I've managed to put together something that seems like it should work reliably: https://github.com/ncoghlan/pyscl-devel/blob/master/rpmlb/sclorg-distgit-download.yml The above URL is useful for me to know what you are doing now. Thanks.

Maybe you are using recipe file https://github.com/sclorg/rhscl-rebuild-recipes/blob/master/python.yml . Preparing sclo-python.yml for your situation looks easier isn't it?

[ncoghlan]

While I started from the rhscl recipe, the actual one I'm building now is https://github.com/ncoghlan/pyscl-devel/blob/master/rpmlb/python-recipe.yml

I'm also learning that by focusing on local mock builds first, and deliberately working outside the backend Fedora/CentOS/RHEL build service infrastructure I'm actually doing something relatively unusual, but I figure that will be useful in the long run, since it will offer a clear starting point for any future SCL bootstrapping efforts: define a bootstrapping recipe, get it running locally in mock on Fedora, then adapt it to either COPR or Koji (or both).

[hroncok]

What I think you actually want for local mock chain building, is adding the built packages to a local repo that is available from the mock.

[ncoghlan]

@hroncok That still runs into the NEVRA re-use problem, where the bootstrapping packages won't be reliably replaced by the properly built versions that include the full feature set.

However, I think I've figured out a way of actually avoiding the NEVRA re-use problem that doesn't involve bumping the release number just to do a full bootstrap build locally: injecting a leading .0.bootstrap into the dist field for the bootstrapping builds. That way, "x.y.z-release.0.bootstrap.dist" will reliably sort as being an earlier version than "x.y.z-release.dist", and so the later build will overwrite the earlier one.

I'll be able to do this because I'm already passing --release el7 to the fedpkg srpm call in the custom backend - even without making any changes to rpm-list-builder itself, I can make that configurable by changing it to --release ${BOOTSTRAP_PREFIX}el7 and using the cmd section in the recipe to touch a file that indicates this is a bootstrap RPM build rather than a normal one. It also means that even if one of the bootstrap builds escapes the build environment, there's a significantly lower risk of it being mistaken for a properly built RPM.

None of that will be applicable to actually building in COPR or Koji (since they're more opinionated about permitted dist targets), but for those there's the option @torsava suggested of just starting with a deliberately broken bootstrapping-only SRPM as release zero, and then pushing the proper version as release 1.

[ncoghlan]

While attempting to implement the above idea, I found a better marker for identifying locally modified bootstrap builds: [ -e ${PKG}.spec.orig ]

That's generated by rpm-list-builder itself when it modifies the spec file, so using it just means that deliberately pristine builds need to avoid using any of the spec file editing options.

[hroncok]

That still runs into the NEVRA re-use problem, where the bootstrapping packages won't be reliably replaced by the properly built versions that include the full feature set.

I thought Koji has this very same problem, so it has to be solved (if isn't) anyway. What am I not getting?

[ncoghlan]

@hroncok It does, but I'm more willing to rely on "previous version" based bootstrapping in Koji and COPR, since you can assume a persistent build environment between runs. That means you only need "no existing package" style builds when rebasing to a new feature branch (and potentially not even then for sclo-python, since it's a rolling release even between feature branches).

By contrast, mock based builds start with an empty repo by default, so I've set my self the goal of getting from "git clone https://github.com/ncoghlan/pyscl-devel" -> locally built SCL in a fully automated fashion. This also has the added benefit of making the bootstrapping testable, reducing the risk of finding it unexpectedly broken when it comes time to bootstrap the next feature release in Koji & COPR.

[ncoghlan]

The custom dist-tag option for mock based builds is looking promising:

• ${PKG}.spec.orig as the filesystem marker indicating a bootstrap RPM
• for bootstrap builds, call fedpkg sources with the "normal" release name, and then call rpmbuild -bs directly, with all the options fedpkg would normally pass, but dist amended to include the bootstrapping prefix
• use mock --postinstall to install the RPMs after building them

[ncoghlan]

Ah, I found a possible opportunity for simplification - now that I'm configuring the RPM build to correctly override dist, it looks like I should be able to go back to always generating the SRPM with fedpkg.

However, the 0.bootstrap string specifically is bad, since it confuses RPM (the numeric field sorts as newer than the mixed el7 field). Fortunately, the plain bootstrap string sorts as older than both elX and fcX, so I can just use that.