= Jenkins Release :toc: right
== Introduction
This repository contains everything needed to release https://github.com/jenkinsci/jenkins[Jenkins core].
This includes:
The release process implies following Git repository:
== Settings
[cols="1,3a,0,0", options="header"] |=== | Variable | Description | Release | Packaging
| GIT_EMAIL
| Defines git email used in git commits.
| x
| x
| GIT_NAME
| Defines git name used in git commits.
| x
| x
| GIT_SSH_COMMAND
| Defines options that can be passed to every git commands used in release scripts.
| x
| x
| GPG_KEYNAME
| Define the GPG key name that will be used to sign artifacts.
| x
| x
| GPG_FILE
| Define the GPG file name that will be used to import gpg key into keyring.
| x
| x
| GPG_VAULT_NAME
| Define the Azure key vault containing the gpg key.
| x
| x
JENKINS_ASC |
Defines the Jenkins signature file name. |
|---|---|
| x |
JENKINS_DOWNLOAD_URL |
Define the url used to download jenkins.war. |
|---|---|
| x |
| JENKINS_VERSION
| Define which version of Jenkins will be downloaded for packaging.
It accepts following values:
latest, fetches the latest version available in a maven repository based on the maven-metadata.xml.version, search the latest version available based on version.More information in https://github.com/jenkins-infra/release/blob/master/utils/getJenkinsVersion.py[getJenkinsVersion.py]
. Versions available depends on the maven repository defined by MAVEN_REPOSITORY_NAME
. JENKINS_VERSION is totally independent of RELEASE_PROFILE explained below
==== | | x
JENKINS_WAR |
Defines the jenkins.war file name downloaded from a maven repository using getJenkinsVersion.py. |
|---|---|
| x |
| RELEASE_GIT_REPOSITORY
| Defines the Jenkins git repository read by the build. It is also the repository where generated commits and new tags will be pushed.
| x
| x
| RELEASE_GIT_BRANCH
| Defines the Jenkins branch used to checkout and then pushes commits related to the release.
| x
| x
RELEASE_GIT_PRODUCTION_BRANCH |
Defines the branch used as the production branch during promotion. |
|---|---|
| x |
RELEASE_GIT_PRODUCTION_REPOSITORY |
Defines the git repository used as the production git repository during promotion. |
|---|---|
| x |
RELEASE_GIT_STAGING_BRANCH |
Defines the branch used as the staging git branch used to promote to the production one. |
|---|---|
| x |
RELEASE_GIT_STAGING_REPOSITORY |
Defines the git repository used as the staging git repository during promotion. |
|---|---|
| x |
RELEASE_GIT_STAGING_REPOSITORY_PATH |
Defines path where staging git repository will be be checkout. |
|---|---|
| x |
|MAVEN_REPOSITORY_URL
| Defines the maven repository URL where is pushed generated artifacts.
| x
| x
|MAVEN_REPOSITORY_NAME
| Defines the maven repository name where is pushed generated artifacts.
| x
| x
|MAVEN_REPOSITORY_PASSWORD
| Defines the maven repository password.
| x
| x
|MAVEN_REPOSITORY_USERNAME
| Defines the maven repository username.
| x
| x
MAVEN_REPOSITORY_PRODCUTION_NAME |
Defines the production maven repository name used during maven promotion. |
|---|---|
| x |
PKGSERVER |
Defines where the different packages will be published. |
|---|---|
| x |
PKGSERVER_SSH_OPTS |
Defines custom ssh options used to connect to PKGSERVER. |
|---|---|
| x |
| PROMOTE_STAGING_MAVEN_ARTIFACTS_ARGS
| Defines parameters used by promoteMavenArtifacts.py.
Default value is set to item --mode copy --source $MAVEN_REPOSITORY_NAME --destination $MAVEN_REPOSITORY_PRODUCTION_NAME --url $MAVEN_REPOSITORY_URL --username $MAVEN_REPOSITORY_USERNAME --password $MAVEN_REPOSITORY_PASSWORD --search '/org/jenkins-ci/main' $(./utils/getJenkinsVersion.py --version)}"
| | x
RELEASELINE |
Define the release line used by packaging scripts in https://github.com/jenkinsci/packaging[jenkinsci/packaging]. |
|---|---|
| x |
| RELEASE_PROFILE
| Define a file containing environment variables specific to a release, located in the profile.d directory.
| x
| x
| SIGN_ALIAS
| Define code signing certificate name.
| x
| x
| SIGN_KEYSTORE_FILENAME
| Define code signing certificate file name.
| x
| x
| SIGN_KEYSTORE
| Define signing keystore.
| x
| x
| SIGN_CERTIFICATE
| Define code signing certificate file name.
| x
| x
|===
== Process
=== Introduction
The release process is divided in two categories. The first part that we mention by using the term the release, is when we create a new java code release. It relies on the Maven Release Plugin to perform the release. It involves signing with a GPG key and a code signing certificate. At the end of this operation signed maven artifacts are pushed to a Maven repository.
The second part that we name packaging is when we retrieve from a Maven repository, the version we want to package. Then we build distribution packages, publish them, promote artifacts between staging and production environment if needed and finally we ensure that our mirrors are up to date.
==== Required In order to trigger a new release, you must fulfil following requirements:
=== Release At this stage, we are going to retrieve the Java code, release a new version using the maven release plugin and then publish artifacts on a maven repository.
It's important to notice that we do not use the maven release plugin to checkout git repositories neither to push changes. This allow us to release from a different git repository than the one defined in the pom.xml. We also need to be able to push commits to a different repository than the one defined in the pom.xml.
. link:http://maven.apache.org/maven-release/maven-release-plugin/perform-mojo.html#localCheckout[localCheckout] must be set to true . link:http://maven.apache.org/maven-release/maven-release-plugin/prepare-mojo.html#pushChanges[pushChanges] must be set to false
==== Steps
Estimated time +- 1h30
. Connect to the Jenkins private VPN (private.vpn.jenkins.io). . Open your favorite browser to https://release.ci.jenkins.io. . Trigger the release job on the master branch. https://release.ci.jenkins.io/job/core/job/release/[Link]. . Once triggered, it asks you which release line you want to do. It's important to know that the release line matches one of the profiles file defined https://github.com/jenkins-infra/release/tree/master/profile.d[here], so please carefully review the settings and be sure that it does what you are looking for. . At the end of the job, git commits and maven artifacts will be pushed to their respective locations.
==== Validate
To validate that the release went well, excepted by having a green build, you can double-check that your artifacts have been correctly pushed to Maven repository located on $MAVEN_REPOSITORY_URL/$MAVEN_REPOSITORY_NAME/org/jenkins-ci/main/jenkins-war/. You could also run jarsigner -verify <your generated artifact> is correct.
==== Stage
In order to have private maven releases, you can modify RELEASE_GIT_REPOSITORY or MAVEN_REPOSITORY_NAME, to respectively use code from a private git repository and then push artifacts to a private maven repository.
Artifact promotion is done in the next stage 'packaging'.
=== Packaging
The packaging process looks after the latest Jenkins version published on a Maven repository and then build and publish artifacts for Debian, Red Hat, SUSE, Windows. It also republishes the War file on the package server. If enabled, then it promotes git commits between git repository, promotes maven artifacts between maven repository.
NOTE: Packages are not re-published if they already exist, only package website is overridden so it's safe to re-trigger the job.
==== Steps
Estimated time +- 30min
. Connect to the Jenkins private VPN (private.vpn.jenkins.io) . Open your favorite browser to https://release.ci.jenkins.io[release.ci.jenkins.io] . Trigger the packaging job on the master branch. https://release.ci.jenkins.io/job/core/job/package/[Link] . Once triggered, it asks you which release line you want to package for. The release line matches one of the profile defines in https://github.com/jenkins-infra/release/tree/master/profile.d[profile.d], so please carefully review those settings in order to validate that's what you are looking for.
Once the job is done, every package will be published and then mirror synchronized.
==== Validate Ensure that packages are correctly published on pkg.jenkins.io and correctly signed.
==== Stage Staging packages is not yet fully supported, more information on link:https://issues.jenkins-ci.org/browse/INFRA-1363[INFRA-1363] and link:https://issues.jenkins-ci.org/browse/INFRA-2608[INFRA-2608]
== Profile The release profile is used to identify the kind of release we are going to do. They are mainly influenced by following elements:
. Do we want to releases based on different repository branch? . Do we want to release based on different git repository?
At the moment we identify three release types:
. Weekly . Stable . Security (both weekly and LTS)
=== Weekly
The weekly release is the default release. It is scheduled every Tuesday as defined by this link:https://github.com/jenkins-infra/release/blob/19685def608c641496e6e2de3d40c275ca5e913d/Jenkinsfile.d/core/weekly#L15[cron]. It uses parameters defined in this link:https://github.com/jenkins-infra/release/blob/master/profile.d/weekly[file]
It releases using the repository https://github.com/jenkinsci/jenkins[jenkinsci/jenkins] from the branch master. Artifacts are pushed to the default maven repository 'Releases'.
If for some reason the release job needs to be re-triggered, you can:
. Connect to the Jenkins private VPN (private.vpn.jenkins.io) . Open your favorite browser to link:https://release.ci.jenkins.io[release.ci.jenkins.io] . Review the weekly environment https://github.com/jenkins-infra/release/blob/master/profile.d/weekly[file]: . Trigger the weekly link:https://release.ci.jenkins.io/job/core/job/weekly/job/release/[job]
You can re-trigger individually the two downstream jobs, release and packaging.
=== Stable Release Candidate A stable release-candidate is a manually triggered release that happens around once a month. It uses parameters defined in this link:https://github.com/jenkins-infra/release/blob/master/profile.d/stable-rc[file].
Before triggering a new stable release candidatae release, some steps are required:
. Prepare jenkinsci/jenkins repository -> missing documentation link.
. Create a branch on jenkins-infra/release with a branch name that match the release branch from jenkinsci/jenkins like rc-stable-<jenkins_version>.
. Review and update the stable environment https://github.com/jenkins-infra/release/blob/master/profile.d/stable[file] with:
.. RELEASE_GIT_BRANCH set to the jenkinsci/jenkins release branch like stable-2.235
.. PACKAGING_GIT_BRANCH set to the appropriate jenkinsci/packaging branch, e.g. stable-2.235
. Trigger the stable link:https://release.ci.jenkins.io/job/core/job/stable/job/release/[job]
You can re-trigger individually the two downstream jobs, release and packaging.
=== Stable A stable release is a manually triggered release that happens around once a month. Refer to link:https://www.jenkins.io/download/lts/[LTS Release Line] for more detailed information. It uses parameters defined in this link:https://github.com/jenkins-infra/release/blob/master/profile.d/stable[file].
Before triggering a new stable release, some steps are required:
. Prepare jenkinsci/jenkins repository -> missing documentation link.
. Create a branch on jenkins-infra/release with a branch name that match the release branch from jenkinsci/jenkins like stable-<jenkins_version>.
. Review and update the stable environment https://github.com/jenkins-infra/release/blob/master/profile.d/stable[file] with:
.. RELEASE_GIT_BRANCH set to the jenkinsci/jenkins release branch like stable-2.235
.. JENKINS_VERSION set to the final release version that will be packaged. If set to 'stable' then the packaging job will try to guess the version based on what was pushed to the maven repository. cfr settings.
.. PACKAGING_GIT_BRANCH set to the appropriated jenkinsci/packaging branch
. Trigger the stable link:https://release.ci.jenkins.io/job/core/job/stable/job/release/[job]
You can re-trigger individually the two downstream jobs, release and packaging.
=== Security The security release follows the same process as the stable one except that artifacts are published in private. So we need to promote git commits from a private repository to the public one then promote maven artifacts from a private maven repository to the public one.
The following sections assume you have prepared jenkinsci-cert/jenkins with security fixes and created a Maven staging repository as documented by the security team.
All steps need to be done twice: Once for weekly, once for LTS.
==== Preparation
. Create a (origin) branch on jenkins-infra/release with a branch name corresponding to the specific release, e.g. security-2.287 or security-stable-2.303.2. Base them on the master (weekly) or stable-2.303 (LTS) branch, respectively.
. In your fork, update the security environment https://github.com/jenkins-infra/release/blob/master/profile.d/security[file] with the following (new) entries:
.. RELEASE_GIT_BRANCH set to the jenkinsci-cert/jenkins release branch like security-stable-2.303
.. MAVEN_REPOSITORY_NAME set to the maven repository name where we are going to publish staging maven artifacts, e.g. caravelli. This is also the source location used by the packaging job to build distribution packages.
.. JENKINS_VERSION set to the final release version that will be packaged.
.. RELEASELINE set to '-stable' for an LTS release, otherwise leave empty or undefined.
.. Open a PR from your fork's branch into the origin repository's branch to allow review.
==== Staging (before release day)
To stage the Maven artifacts, trigger the generic Release link:https://release.ci.jenkins.io/job/core/job/release/[job] from the appropriate branch like security-stable-2.303.2.
To do that, follow these steps:
.. Force repository scan
.. Trigger the first build to have access to job parameter and immediately abort it
.. Trigger a job with the correct parameters
... RELEASE_PROFILE set to security
... RELEASE_GIT_BRANCH set to unused as we already define it in the release profile file, which overrides the job parameter
... MAVEN_REPOSITORY_NAME set to unused as we already define it in the release profile file, which overrides the job parameter
... VALIDATION_ENABLED set to true if the validation stage should run
==== Publishing (on release day)
. To create and publish packages, trigger the generic Packaging job link:https://release.ci.jenkins.io/job/core/job/package/[job] from the appropriate branch like security-stable-2.303.2 with correct parameters
.. RELEASE_PROFILE set to security
.. RELEASE_GIT_BRANCH set to unused same reason as before
.. MAVEN_REPOSITORY_NAME set to unused same reason as before
.. MAVEN_REPOSITORY_PRODUCTION_NAME set to unused
.. MAVEN_STAGING_REPOSITORY_PROMOTION_ENABLED set to false (manually done by publishing-tool in a parallel process)
.. GIT_STAGING_REPOSITORY_PROMOTION_ENABLED set to false (manually merged by security team)
.. VALIDATION_ENABLED set to true
== Certificate
The Jenkins project uses a Digicert account provided by CDF to request code signing certificate. The release environment is designed to download a pkcs12 certificate from Azure key vault.
. Request a code signing certificate from Digicert
. Convert the code signing certificate from Digicert from p7b format to a pfx (with pkcs12) certificate which also includes the private key but not export password
. Upload the pfx certificate to Azure Key Vault
. Update the release environment credentials with appropriated password
Generate a new code signing certificate private key and a certificate signing request:
openssl req -out jenkins-release.csr -new -newkey rsa:4096 -keyout jenkins-release.key
Show csr information
openssl req -text -noout -verify -in jenkins-release.csr
Show private key information
openssl rsa -in jenkins-release.key -check
Show certificate information
openssl x509 -in jenkins-release.crt -text -noout
Convert p7b to pkcs12
openssl pkcs7 -print_certs -in digicert.p7b -out jenkins-release.crt
## Asks for the `jenkins-release.key` private key passphraseopenssl x509 -in jenkins-release.crt -text -noout
openssl pkcs12 -export -in jenkins-release.crt -inkey jenkins-release.key -out jenkins-release.pfx
Enter Export Password: # Empty!! Verifying - Enter Export Password: # Empty!!
Check PFX (pkcs12 format) attributes
openssl pkcs12 -info -in jenkins.pfx
== Core Maintainers
More information about Jenkins Core maintainers and the different roles can be found in https://github.com/jenkinsci/jenkins/blob/master/docs/MAINTAINERS.adoc[MAINTAINERS].
== FAQ
The stage release failed and we already push commits during the maven release.
The problem here, is if we re-trigger the release job, we also update the release version again.
So if the process fails on running mvn release:stage, then we can re-trigger it from inside the container as custom parameters are located in settings-release.xml.
kubectl get pods -n release # Looking for the correct jenkins agent name
kubectl exec -i -t -n release -c maven <pod_name alias jenkins agent> bash
cd /home/jenkins/agent/workspace/core_release_master/release
mvn -B -DstagingRepository=releases::https://repo.jenkins-ci.org/releases -s settings-release.xml --no-transfer-progress release:stage== Miscellaneous