Create first sclo-python release based on rh-python35

[ncoghlan]

This issue covers making an initial release of sclo-python that exactly matches the existing rh-python35 SCL, except using the sclo-python SCL name.

Learn to use RPM List builder:

• [x] Define an updated build recipe, mock configuration, and custom rpm-list-builder config
• [x] Build rh-python35 locally from scratch based on the updated recipe

Do the initial build of sclo-python:

• [x] Create EL7 development branch for metapackage (Done: https://github.com/ncoghlan/sclo-python/tree/sig-sclo7-sclo-python)
• [ ] Extract a list of all the places where "Python X.Y" and related constructs are referenced in file names or contents
• [ ] Rename the SCL to sclo-python
• [ ] Build the renamed SCL locally still using the existing sig-sclo7-rh-python35-rh branches for components other than the metapackage
• [ ] Rename the current build recipe to sclo-python-full, and define two new recipes for sclo-python-bootstrap (which only builds the partial packages needed for bootstrapping) and sclo-python (which assumes previous versions are already available to meet any bootstrapping requirements)
• [ ] Build sclo-python in https://copr.fedorainfracloud.org/coprs/ncoghlan/sclo-python-preview/

Add dedicated packaging branches for the rolling Python SCL:

• [ ] Create corresponding sig-sclo7-sclo-python EL7 branch for all the Python repos under https://github.org/sclorg-distgit
• [ ] Build the renamed SCL using the sig-sclo7-sclo-python branches

Actually release the new SCL:

• [ ] Move the metapackage repo to https://github.com/sclorg-distgit/sclo-python
• [ ] Set up CentOS Build System builds
• [ ] Set up a new softwarecollections.org page
• [ ] Build and release the first sclo-python package set

[ncoghlan]

Additional resources:

• SCL builder: https://github.com/sclorg/rpm-list-builder
• RHSCL rebuild recipe: https://github.com/sclorg/rhscl-rebuild-recipes/blob/master/python.yml

[ncoghlan]

As I look more at this, I think I'm going to have to push full event-driven release automation into the "later" bucket (and it will probably become dependent on the Fedora Modularity and Fedora CI work).

Instead, the initial release process will be dependent on my kicking it off locally for whatever I want to update.

[ncoghlan]

Filed #4 and #5 to reflect the fact that rebuilds won't initially be automated.

[ncoghlan]

Steps for the initial tool installation:

git clone https://github.com/sclorg/rpm-list-builder.git
pushd rpm-list-builder
git checkout v1.0.0
vex -m pyscl-devel
pip install -r requirements.txt
pip install .
popd
git clone https://github.com/sclorg/pyscl-devel.git
cd pyscl-devel/rpmlb
rpmlb --download custom --custom-file sclorg-distgit-download.yml \
          --build mock --mock-config epel-7-x86_64 \
          --work-directory .working \
          python-recipe.yml sclo-python

TODO: Write a custom rpm-list-builder YAML file for getting SRPMs and sources from sclorg-distgit

[ncoghlan]

Initial build attempt failed due to a few test failures when running the tests in a with_rewheel=0 build, so I've created https://github.com/ncoghlan/python-spec as a fork where I can work on spec fixes

[ncoghlan]

Turns out there is a run_selftest_suite macro I can set to zero instead, but it would also make sense to update the check macro to exclude these tests when running without rewheel:

• test_ensurepip
• test_resource
• test_venv

The spec also isn't passing -j4 to the test suite, so it's serialising all the tests rather than running any of them in parallel (which adds up, since the self-check tests both the debug and release binaries)

[ncoghlan]

I've published the rpmlb files I'm using at https://github.com/ncoghlan/pyscl-devel/tree/master/rpmlb

Running it requires CUSTOM_DIR support in rpmlb, though: https://github.com/sclorg/rpm-list-builder/issues/67

[ncoghlan]

@encukou @torsava @stratakis @irushchyshyn While my notes here are a bit stream-of-consciousness as I try to get the initial sclo-python builds working, I'm guessing they may be of interest as review commentary on the SCL build recipe and Python spec file :)

[ncoghlan]

Using "macros" in the recipe doesn't appear to be working as intended, as the %global expressions in the original spec file end up overriding the rpmlb provided values.

I'm going to switch to using replaced_macros in the recipe for now, but a more preferable long term solution might be to make those global definitions conditional, so they don't do anything if the macro has already been set.

[hroncok]

I wanted to make those bconds anyway:

https://github.com/sclorg/rhscl-rebuild-recipes/issues/25 https://github.com/sclorg/rpm-list-builder/issues/70

[ncoghlan]

@torsava Running RPM list builder as documented doesn't seem to be setting a suitable SCL prefix - is there something currently undocumented that needs to be passed to the rpmlb call to get it to be set appropriately for all the RPM builds?

[ncoghlan]

@hroncok Aye, that would be a very nice solution.

[torsava]

@torsava Running RPM list builder as documented doesn't seem to be setting a suitable SCL prefix - is there something currently undocumented that needs to be passed to the rpmlb call to get it to be set appropriately for all the RPM builds?

Did my email suggestion to add scl-utils-build and metapackage-build to the buildroot solve this issue?

[ncoghlan]

@torsava The missing piece was passing --define 'scl rh-python35' to mock as one of the entries in --rpmbuild-opts. At the moment I'm setting that directly in the custom build backend.

[ncoghlan]

@torsava https://github.com/ncoghlan/pyscl-devel/commit/69e364835900fb3310a0e066927aa00af5fc244f was another interesting discovery - the SCL config files don't actually get generated properly when installing them post-build if yum isn't installed. (It might be one of yum's dependencies rather than yum itself that's the missing piece, but it's clear there's a missing runtime dependency that isn't necessarily included in a fresh buildroot)

[ncoghlan]

With the serialised RPM builds, a full from-scratch build (including setting up the build root) took just under an hour of real time here (as measured by doing time rpmlb...). Three key speedups are included in that:

• the first bootstrapping Python build is release-only (no debug build)
• the first bootstrapping Python build skips running the Python test suite
• the Python test suite is run with the -j0 option so it uses multiple processes

There's plenty of opportunity to speed that up further by doing some of the later package builds in parallel.

I'd also be interested to know whether the instructions at https://github.com/ncoghlan/pyscl-devel/blob/master/README.md work for anyone else (since the point of the exercise is to make the local bootstrapping almost entirely automated)

[torsava]

@ncoghlan Interesting! We did not have to deal with these issues when building in Brew or Copr.

A full serialized from-scratch build of all the packages in the collection in under an hour? That sounds literally unbelievable to me... wow, good work!

I'd also be interested to know whether the instructions at https://github.com/ncoghlan/pyscl-devel/blob/master/README.md work for anyone else (since the point of the exercise is to make the bootstrapping almost entirely automated)

I can try it out if it'll help you.

[ncoghlan]

@torsava I suspect what I'm mainly finding are a few implicit dependencies on things that are pre-installed in the COPR & Koji buildroots for the SCL builds, but aren't listed in the default EPEL mock configs.

As far as the from scratch build in under an hour goes, I suspect running entirely from SSD is a big part of that. Next time I do a full build I'll try to remember to keep an eye on the disk IO bandwidth usage.

[torsava]

@ncoghlan So I did the build myself and I sent a PR to amend the README a little bit. It indeed finished inside an hour, impressive!

I did get a few of these: WARNING: Failed install built packages, see build.log.

[ncoghlan]

@torsava D'oh about the install failures - I knew about the problem with --postinstall not affecting the mock return code if it failed, but I forgot to double check my own logs for that error message. It's hard to tell the full significance just from the build log though (since that doesn't say which packages failed to install).