Test Framework for Kubernetes-Openshift deployment for Adopt+OpenJ9 JDKs

[chrisc66]

As a part of the main issue for Kubernetes deployment #9443 , this issue keeps track of discussion about the automated test framework of operator and helm chart.

The goal is to create automated tests for operator that:

• contains Kubernetes or Openshift environments
• verifies operator deployment and functionality
• reuse existing Adopt testing framework and machines

More information and testing proposal will be posted below later.

[DanHeidinga]

reuse existing Adopt testing framework and machines

@smlambert does Adopt already have an OpenShift K8 cluster? Is any OpenShift testing already happening at Adopt?

We're trying to figure out where and how to test a K8 Operator and Adopt has been suggested as a possible home for this kind of testing.

[smlambert]

We do not yet have an Openshift cluster, @LongyuZhang has been working on enablement of the quarkus openshift tests testing on some IBM resources internally (on x/p/z linux). Once we have worked through most of the 'gotchas' we should be able to setup to run them at the AdoptOpenJDK project (setup required cluster and remove the disabled tag from playlist to include those tests in our suites).

We have been holding off enabling as there are some upstream issues that need to be resolved for p & z before the full quarkus openshift suite can be run. Quick summary is that io.quarkus:quarkus-maven-plugin tests fail on p & z (related: https://github.com/quarkus-qe/quarkus-openshift-test-suite/issues/82).

I have not looked at what is involved for testing this operator under discussion, but will be interested to know what is planned, so that it aligns with a low-friction test story.

[chrisc66]

There are a few existing tests that we can study from, potentially reuse or build similar tests.

• The operator and helm chart currently runs on internal travis and hosted on GitHub enterprise. These tests call a set of bash scripts from IBM Cloud Pak CICD pipeline. The current tests are more or less jitserver centric, e.g. deploys jitserver and a client JVM, then verifies remote compilation by parsing the verbose log. However we can't use these tests directly since it is internal and owned by Cloud Pak.

• Knowing above condition, @KevinBonilla and I already built an operator deployment test on internal Jenkins. This is a very basic test, only verifies the deployment is successful. We are able to create more tests for operator and helm chart if desired, and eventually move these tests to Adopt Jenkins.

• The Runtime Component Operator also provides tests as a directory within the project. Tests are implemented using go and makefile and RC runs them on travis. The RC test contains a unit test go test ..., and an end-to-end test operator-sdk test locale2e-test.

My proposal is reusing RC tests and implementing tests / pipelines on Runtimes internal and (later move to) Adopt.

For Kubernetes / OpenShift environment, we can use Jenkins machine as client, and connect it to a cluster. Currently we can use a Fyre cluster until we have some other infrastructure available. OpenShift Client CLI oc needs to be installed on Jenkins machines, which is done already.

Personally I don't have much information on how Adopt architecture and OpenShift tests look like. I am very interested in understanding how these work and potentially contribute to Adopt tests. I am going to investigate and look for more information. Please suggest if anyone has good ideas.

[LongyuZhang]

The previou io.quarkus:quarkus-maven-plugin issue on p and z linux has been fixed with the new release of quarkus 1.7.2.Final and Dekorate 0.12.8.

Now we are able to run quarkus-openshift-test on our internal Jenkins testing framework on all x,p,z platform with Openshift Cluster on the corresponding platform. Since Openshift Cluster and s2i images are not available yet, I will create PR for this test, but remain the disabled tag temporarily. I have updated more detailed information and corresponding grinder links on runtimes/backlog 353. Thanks.

[raguks]

@chrisc66 can we explore doing tests on single node OCP for which RH has steps somewhere? I think we tried that last year on a VM. If the tests work there we might be able to use an Adopt VM to test?

@andrewcraik @DanHeidinga Test discussion and plans here. FYI.

[andrewcraik]

I would note that whatever testing IBM chooses to do internally on OpenJ9 or other components is its prerogative. For a contribution to the OpenJ9 project, the open project needs to have a way of testing the contributed component without dependence on IBM's internal processes and infrastructure (or that of any other contributor). It sounds like there is the start of a plan to achieve this with help from Adopt, but it is important to remember the contribution cannot depend on non-public testing for its verification and maintenance.

[chrisc66]

As testing is a huge part of the operator project, I would like to separate testing apart from everything else in https://github.com/eclipse/openj9/pull/10409#issuecomment-689556225.

Agreeing with @DanHeidinga in his comment, our testing needs to:

1. Make sure the original RC operator is functional, i.e. pass all existing RC tests The current RC tests and pipelines are running on travis. There are a unit test and some end-to-end tests explained in my previous comment. These tests look mature and we can reuse them without additional tests. Besides, the operator code is packaged inside docker images, the pipeline also builds and pushes the image automatically.
2. Additional tests for openj9-related features added by us At the moment there isn't lots of openj9-related features added, so this part is relatively simple. We provided JITServer and client JVM templates, thus we need to create a test that verifies JITServer remote compilation by parsing JITServer verbose log.

Since this project requires both Kubernetes and OpenShift environment, testing infrastructure is quite different from OpenJ9. The main blocker is we don't have external testing clusters. The current idea is reusing Adopt testing infrastructure. There are a few options that we can consider:

1. Minikube for Kubernetes: Minikube creates Kubernetes environment on VMs, so we can use an Adopt VM and run Minikube on top.
2. Installing single node OpenShift: OpenShift provides an installer repo that could be used as automated installation. I am going to follow the instruction and try to install a single node OpenShift, then we can evaluate if this method can be achieved thu automation.
3. possibly Adopt clusters for OpenShift: I know that we have a Fyre OpenShift cluster available and Adopt has done some testing on it. @smlambert Could you please let us know if Adopt has any external testing cluster that maybe we can borrow? Thanks.