EPIC: Automate & Improve Production Of Linux Package Installers

[steelhead31]

There are currently a number of issues when producing the Linux APKs, Debs & RPMs for the JDKs / JREs at the end of a release cycle:

Notably:

• Currently take ~1 week at the end of a release
• Quite painful for CPU when doing 5 major version releases
• Lots of manual updates required to numerous files, which can be prone to error.
• Lots of work to add new versions and O/S distributions
• All architectures for a version have to be completed, before installer packages can be produced
• Removing a single architecture (when required) is even more time consuming and error prone.
• Currently doing platform specific installers is quite complicated

Proposal:

Ideally, the linux packages should be produced on a per version/architecture basis, and this should be done at the same time that the linux distribution tar balls are published to Github.

The new process, might be as follows:

When the release binaries are published to github via the jenkins refactor release tool :  

• Run a subsequent process to create the package installers, for each dist/version/architecture combination.
• Once the linux installer packages are produced, push them to artifactory

This will require the following tasks:

• Create a generic script that can build an individual arch, version, distro combination
• Dynamically generate the APK build, debian control and RPM spec files.
• For the job that generates files, how do we keep track of those files (e.g. pushed into a repo?)
  • Ideally tracked in a GitHub repository

Advantages of this approach :

• The process doesn’t disrupt the existing steps to test the installers

Improvement Items That Can Be Added:

• Improve handling of src rpms to be architecture specific, rather than a single huge file, with EVERY architecture contained.
• For the past issue where pushing to JFrog has failed
  • Need a follow-on verification test for whether an artifact was pushed successfully on JFrog
  • Download the artifact from jfrog and perform a simple test.
• Also looking at using symlinks to reduce the number of uploads to JFrog, though the technical work is ongoing
• Add metrics for how long it takes artifacts to be pushed to JFrog, but when we do improve this, we will also look back at previous releases to see how big the improvement is
  • Would also be nice to get metrics for the container images
• AIX installers don’t exist at this time

[steelhead31]

@gdams feel free to expand the above :)

[jiekang]

Another area to investigate is the need for distribution and version specific repositories for RPMs. As far as I'm aware, we are not building distribution-specific nor version specific artifacts and so we should also be able to publish them to a distribution and version agnostic repository.

For example, the JDK 23 RPM for AmazonLinux 2, Fedora 40 and Fedora 39 are the same file. And other products that provide yum repositories are agnostic:

Amazon Corretto: baseurl=https://yum.corretto.aws/$basearch

Microsoft VS Code: baseurl=https://packages.microsoft.com/yumrepos/vscode

Eclipse Temurin: baseurl=https://packages.adoptium.net/artifactory/rpm/${DISTRIBUTION_NAME:-$(. /etc/os-release; echo $ID)}/\$releasever/\$basearch

We can simplify the Eclipse Temurin structure to something like: https://packages.adoptium.net/artifactory/rpm/$basearch and have significantly less uploads of the same RPM to JFrog.

[steelhead31]

Currently prototyping a jenkins process based on the following logical flow..

1) The refactor open-jdk release job is triggered as part of the release pipeline (in dry run mode) ready to be rerun when a platform is ready to be pushed to the github binary repository.

2)  When the above process completes successfully, trigger the new "modular linux package build pipeline", passing through the current platform/arch specific parameters

3) Modular package builder checks if its a release pipeline, ( ie Release is TRUE ), if not exits and does nothing Prototype job is here: https://ci.adoptium.net/job/sfr-build-linux-package-modular/

4) If it is a release pipeline, AND dry run is FALSE ( ie binaries are being pushed to github ), then build the linux installer packages.

..... WILL Expand This As I define the process more thoroughly .....