We need to tests whether Linux packages can be installed at all.
Things we have to check:
Does the package install at all? This needs to be tested on all supported distributions, on all platforms and newest and oldest supported versions. For example, there are subtle differences between Debian and Ubuntu that made packages installable on Ubuntu but not on Debian (see #105 for an example).
Does the package pull in the right dependencies? This needs to be tested at least on all distributions and platforms. RPM for example requires a ISA indicator. Without it, yum might pull 32 bit dependencies on a 64 bit platform.
Is it possible to install the package after the matching Java implementation shipped with the distribution? Inconsistencies in update-alternatives might cause installation failures. This needs to be tested on all distributions, oldest and newest supported.
Is it possible to switch between the Java implementation shipped with the distribution and the AdoptOpenJDK package using the distribution's tools?update-alternatives on RPM-based distributions and update-java-alternatives on deb-based distributions. This needs to be tested on all distributions, newest and oldest supported versions.
Additional requirements:
The test should be performed as part of the build before packages are uploaded and should fail the build if it does not succeed.
The artifact (deb, rpm) produced during the build should be directly installed. Downloading additional packages from our repository is okay, however.
Ideally, the testing should be able to handle multiple packages simultaneously because of #106.
Over the last year, I made some experiments with TestContainers with test code written in Java. Summary:
TestContainers look promising for this use case. Communication with the containers is durable, installing packages is doable. Preparing the shell commands and dealing with the distribution differences isn't fun, however. For example, with old Ubuntus, apt isn't able to install a local package with all its dependencies.
Containers for all supported distributions are available, at least for AMD64.
Open issues:
It's unclear for me how we can integrate that with Gradle or any other tool. Leveraging Java test support including JUnit 5 would be very nice and remove some pain from writing these tests.
Machine capacity is a big question. We currently support 8 distributions: Debian/Ubuntu on 5 architectures, RHEL/CentOS/SLES on 4, Amazon on 2, Fedora/OpenSUSE on 1.
We currently build the packages on AMD64 on very few machines because the RPMs have to be signed as part of the build. If we want to keep it that way, building and testing have to be decoupled.
Is it possible to support all packages/distributions/OpenJDK versions with a single test suite?
The umbrella issue is #205.
We need to tests whether Linux packages can be installed at all.
Things we have to check:
update-alternatives
on RPM-based distributions andupdate-java-alternatives
on deb-based distributions. This needs to be tested on all distributions, newest and oldest supported versions.Additional requirements: