search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 719 forks source link

Change Based Testing #17441

Open llxia opened 1 year ago

llxia commented 1 year ago

Motivation:

The increasing number of tests we have is resulting in longer testing times for completing the full testing process. For instance, the sanity.functional test currently takes approximately 2 hours when run in parallel on 3 machines, while the sanity.system test takes around 3 hours under the same conditions. Additionally, there are long-running extended and special level tests to consider. Although one potential solution is to increase the number of machines in parallel to expedite testing turnover, this approach incurs additional costs in terms of machine resources and triaging efforts. Furthermore, it prompts us to question whether it is truly necessary to execute the complete set of tests, particularly for PR testing. In light of agile development practices and a fast Java release cadence, we must strive for greater effectiveness and efficiency in our testing process. We firmly believe that implementing change-based testing can enable us to test more effectively, reduce testing turnover time, and optimize machine resources and triaging efforts. In other words, we can adopt a laser-focused approach by conducting change-based testing specifically targeting a given PR. This approach will benefit both the contributor and the committer by saving valuable time.

Proposal:

Our proposal entails leveraging code coverage information to acquire and store code and test-related data. Subsequently, we will analyze each code change within a PR and suggest and execute relevant test(s) accordingly. Ideally, from a user perspective, when a PR is created in the Openj9 repository, a GitHub workflow will automatically run and analyze the associated git PRs, initiating the execution of the relevant test(s), and the result will be displayed on PR. The committers still have the choice of running more testing if needed.

Implementation:

  1. Enhance the existing native code coverage build (see https://github.com/eclipse-openj9/openj9/issues/12133) to run and collect gcov metadata per test and store the metadata in artifactory (or other temporary storage)
  2. Process metadata to analyze code and test information. That is, analyze and store the relationship of which line covers which test(s), and which test covers which line(s) in a map or DB.
  3. Display the info through a web application with API support (i.e., add API in TRSS). For example, by providing the source code file name(s) and line number(s), API is able to respond with which test(s) should run to cover the file(s) and line(s).
  4. Create a parser to analyze git PRs to figure out which file(s) and line(s) are affected
  5. Create a GitHub workflow to build SDK based on PR and run specific tests by leveraging https://github.com/eclipse-openj9/openj9/commits/master/.github/workflows
  6. Enhace the GitHub workflow to connect all pieces together. It will call the parser to get the affected file(s) and line number(s) and call API to get test name(s), then run the test(s) and display the result on PR
  7. Further prioritize the related tests by leveraging ML

Related to https://github.com/adoptium/aqa-tests/issues/4278

DanHeidinga commented 1 year ago

Lan, this sounds really cool way to speed up testing of PRs.

I expect nightly builds will continue to execute as they do today (not be changed based) and that we can tune the changed-based PR tests if we find more than the expected number of failing tests in the nightly builds?

Subsequently, we will analyze each code change within a PR and suggest and execute relevant test(s) accordingly. Ideally, from a user perspective, when a PR is created in the Openj9 repository, a GitHub workflow will automatically run and analyze the associated git PRs, initiating the execution of the relevant test(s), and the result will be displayed on PR.

"initiating the execution of the relevant test(s)".... I think the question here is "when"? Will the tests be run on every push to the PR? Only once? Manually triggered by the committer? I guess this only matters if we run into resource constraints on github runners...

llxia commented 1 year ago

The ideal application of change-based testing is to execute only the relevant test(s) for a given PR, offering prompt testing feedback to both contributors and committers. Committers can choose to run additional tests using the existing git comment-triggered test builds. It's important to note that change-based testing does not replace any Nightly/Weekly/Release testing. The Nightly/Weekly/Release testing should continue to encompass full-level testing. However, if the accuracy of change-based testing is established, it may be possible to reduce the extent of nightly testing in the future. Currently, the focus is on testing PRs exclusively.

Regarding the triggering mechanism for change-based testing, we have a couple of options. It can be automatically triggered when a PR is ready for review. Alternatively, we can customize the triggering process based on our specific requirements. For example, instead of an auto-trigger, it can be activated through git comments by committers and/or contributors. In case the GitHub runner's resources are limited, we can adapt the setup to utilize a Jenkins job instead.

DanHeidinga commented 1 year ago

Thanks for clarifying @llxia.

Regarding the triggering mechanism for change-based testing, we have a couple of options. It can be automatically triggered when a PR is ready for review. Alternatively, we can customize the triggering process based on our specific requirements. For example, instead of an auto-trigger, it can be activated through git comments by committers and/or contributors. In case the GitHub runner's resources are limited, we can adapt the setup to utilize a Jenkins job instead.

This is the part I'm still the most unclear on (and maybe that's a "me" thing). I'm not sure what flags a PR as "ready for review" apart from leaving draft status (and most of our PRs are never put in draft). Would the change based tests be re-triggered for every push to a non-draft PR? The more you can expand on the initial approach (totally fine to change it later) the easier it will be for devs to get value from it.

llxia commented 1 year ago

GitHub workflow can be triggered by the specific GitHub event(s). ready_for_review is just one example. By default, a workflow only runs when a pull_request event's activity type is opened, synchronize, or reopened. (See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request). For example, we are using ready_for_review in AQA repo: https://github.com/adoptium/aqa-tests/blob/master/.github/workflows/directoriesFilesChangePR.yml#L4).

We can configure the GitHub workflow based on Openj9 needs. If we have concerns with auto-triggered GitHub workflow, we can do comment-triggered GitHub workflow (same as the existing auto exclude test GitHub workflow) or comment-triggered Jenkins job (same as the existing PR builds). We can start with a comment-triggered workflow/Jenkins job. It will allow us to have better control.