A collection of small scripts, adhering to the Unix philosophy, used to release The Foreman. They automate parts of the branch and release processes. It is also used for the Katello branch and release processes.
The most important environment variables are PROJECT
and VERSION
.
Procedures are kept in the procedures
directory.
Both the branch and release procedures defines two roles: an owner and an engineer. Someone can be both owner and engineer at the same time, and it is preferable for a fast process. Having both in (very) different timezones can lengthen the process. Sometimes it's just not possible and multiple people perform the procedures. Below is a listing of access used.
releases
group on DiscourseThese are the same as Foreman's release engineer.
Use procedure_branch
to display the steps. Modify PROJECT
and VERSION
as needed.
PROJECT=foreman VERSION=3.7 ./procedure_branch
It will have created releases/${PROJECT}/${VERSION}/settings
, which should be submitted as a pull request.
A more advanced invocation is to specify the parameters directly and copy the output.
This uses wl-copy
for Wayland; xclip
achieves the same for Xorg.
PROJECT=foreman VERSION=3.7 ./procedure_branch 2023-05-23 TheOwner TheEngineer | wl-copy
Now post this to Discourse Development/Releases as $PROJECT $VERSION branching process
.
Follow the process as instructed.
First make sure you have fetched, pulled in the latest commits and rebased your fork. Secondly ensure releases/${PROJECT}/${VERSION}/settings
contains the correct FULLVERSION
. Modify as needed and submit as a pull request.
Then generate the procedure, similar to the branching process:
PROJECT=foreman VERSION=3.7 ./procedure_release
Or the complete version:
PROJECT=foreman VERSION=3.7 ./procedure_release 2023-05-23 TheOwner TheEngineer | wl-copy
Now post this to Discourse Development/Releases as $PROJECT $FULLVERSION release process
.
Follow the process as instructed.
Python and python-jenkins are needed. On Fedora this can be installed:
dnf install python3-jenkins
Foreman requires GPG signed git tags. Configure git with your personal gpg key id as your signing key:
git config --global user.signingKey <YOUR_PUB_KEY_HASH>
For storing secrets, gopass is used. On Fedora it can be installed:
dnf install gopass
Then make sure you have a GPG key on your system and initialize gopass with your key:
gopass init <YOUR-PUB-KEY-HASH>
For running jobs in Jenkins, make sure you have access and add your Jenkins password or API token:
gopass edit theforeman/jenkins-token --create
Copr is where all packages are built and the stage repositories are generated from for releases. API access is needed in order to perform release activities. To ensure you have Copr access setup:
For commands on the Foreman infrastructure, add your personal sudo
password:
gopass edit theforeman/unix --create
The releases store from the shared secret storage is also needed:
gopass clone secrets.theforeman.org:/srv/secretsgit/theforeman-release.git theforeman/releases
When a GPG key has already been generated, it can be imported from the backups:
import_gpg_private
Make sure VERSION
is correct in settings
and FULLVERSION
in releases/$PROJECT/$VERSION/settings
. This assumes a GPG key is already present.
./tag_project
./release_tarballs
./download_tarballs
./inspect_tarballs
./sign_tarballs
./bump_deb_packaging
./bump_rpm_packaging
./release_packages
# These steps can happen during the build after RPMs have been built but DEBs are still running
./generate_stage_repository
./sign_stage_rpms
./upload_stage_rpms
When handling non-Foreman releases (currently supported: Katello and Client), set PROJECT
to the lowercase name of the project and VERSION
to the version of the project (if it differs from the Foreman one).
PROJECT=client ./generate_stage_repository
PROJECT=katello VERSION=3.13 ./generate_stage_repository
When starting a new release, the following scripts can be used to generate a new key:
generate_gpg
export_gpg_private
export_gpg_public
sign_gpg
upload_gpg
Our Debian repository doesn't rotate keys based on releases, but on a time basis.
To generate a new key:
export PROJECT=foreman-debian
export VERSION="$(date '+%Y')"
generate_gpg
export_gpg_private
export_gpg_public
sign_gpg
upload_gpg
Settings can be customized in settings.local
. The following settings are supported:
GIT_DIR=$HOME/dev # Projects are cloned here
GIT_REMOTE=upstream # Git remote for cloned projects
PACKAGING_PR=true # Create a PR in bump_{deb,rpm}_packaging
The build_stage_repository
script generates a stage repository locally that can then be uploaded to the staging repository server or used locally for testing. This is done for RPMs and SRPMs. The workflow for each is:
For releases (executed by release owner):
1. Uses reposync to copy all RPMs from production (yum.theforeman.org) to a local repository
2. Run repodiff comparing the local copy of production and Copr for the release
3. Download the new packages from Copr
4. Filter all packages through comps file, removing anything not in the comps file
5. Runs createrepo
6. Generates a new module metadata file based upon the local repository
7. Generate list of unsigned RPMs
8. Sign the unsigned RPMs
9. Update the repository metadata
For nightly (executed by Jenkins):
1. Copy all RPMs from Copr for the given repository to local
2. Filter the downloaded RPMs through comps file, removing anything not in the comps file
3. Runs createrepo
4. Generates a new module metadata file based upon the local repository
The local repository can then be uploaded to the server backing stagingyum.theforeman.org using rsync. Where any RPM found locally already in staging is kept, and any RPM found in staging not in the local repository is removed.
The release workflow is execute using the following actions:
generate_stage_repository
sign_stage_repository
upload_stage_repository