Enable a Disabled tests pipeline or workflow

smlambert commented 4 years ago

This test pipeline could check state of the issue that the test was disabled under and then run all disabled test targets whose issues are resolved to check if they really are in a state to reinclude/reenable.

Since there are various granularities on how we can and do disable tests, this work would look at several parts:

[x] disabled test targets in playlists
[ ] disabled openjdk tests in openjdk subdirectory (in problemlist*.txt files)
[ ] disabled functional tests in openj9 repo (several types of exclusions, testNG, cmdlinetester, others?) - optional for 1st version of this feature
[ ] https://github.com/adoptium/aqa-tests/issues/3917 (check when the playlists are changed and report if there are any exclusions that should be assessed, or any parsing issues with the playlist)

Update from comment below where there are several subtasks as part of this EPIC:

2698
2699
2701
2702

We have many pieces of this EPIC completed, and now it is time to integrate the pieces into a single working pipeline.

d3r3kk commented 4 years ago

At Microsoft we have a plan to do something along this line as well, and we would love to collaborate here.

Currently our idea is to:

For each platform/OS:

Obtain the latest known good build (latest nightly)
Obtain the ProblemList.txt files (there will certainly be more than 1).
Parse the ProblemList.txt files to enumerate the list of tests.
Run each test 1 at a time against the last known good build and record the status.
Generate a report that shows the status for each test run
Collate week-over-week runs of these tests to show trends toward success (candidates for re-introduction).

...something like that.

d3r3kk commented 4 years ago

cc: @junyzheng

smlambert commented 4 years ago

Great! since AQA invokes multiple underlying test frameworks there are several ways tests can be excluded.

You've identified how one could approach the openjdk test group. It will be good to break down the components of the solution into runnable units so we can use some of the same pieces for the different types of excludes.

Generically, we

create a list of tests and the issues they are excluded under and platform
check the state of the issue (if github, API is easy to call, if JBS, not entirely straightforward... does not allow for anonymous queries, believe it may need to scrape the webpage unless we can have a userid with access, or a different site that mirrors JBS that has API)
if issue is still open, no action
if issue is closed, initiate a rerun of the test on the platform(s)/versions(s) with latest nightly
if rerun fails, log it but do not reinclude
if rerun passes, can we automatically check if it was an intermittent issue? (likely not, unless there was a tag on the issue that could be queried). if intermittent issue, rerun XX times to verify
for reincludes, generate a PR for reinclusion of test

Related to another issue, I was going to create a simple script called rerun.sh that could either be passed a rerun link or be rerun with particular env vars that we use (BUILD_LIST, TARGET, JDK_IMPL, JDK_VERSION, etc) to run a particular test case

This will give us flexibility to then run in any CI system (Jenkins/AzDo/laptop of devs).

junyuanz1 commented 4 years ago

here is what I am thinking based on @smlambert 's comment.

find all the problemlist.txt files
iterate the problemlist.txt file line by line, send request to github, or openjdk bug tracker, or azure devops to check bug status. do nothing if it is still open, run test again if it is closed.
run test based on the intermittent tag, run N times if intermittent tag exist, otherwise only run once.
report the result and create PR by removing passed entry from the problemlist.txt file.

please let me know what you think.

I am planning to write a Python script for this and it will call grinder, etc.

llxia commented 4 years ago

Thanks @junyzheng . This is a great start. I got inspired by it and have a couple of thoughts:

I would like to collaborate with you to enhance this to support playlist.xml. In this way, we are covering all tests
I would like to create auto PR for exclude tests as well. Something like, keywords in git issues (i.e., exclude test testAAA) triggers Jenkins job which git clone repo and edit playlist.xml/problemlist.txt to exclude the test, and create auto PR using git API
If an issue is closed, it does not mean it is fixed. (i.e., it can be closed as dup). Even if the issue is fixed, we may need to make sure the fixes are in the SDK we are using. There may be a delay in the nightly build. Maybe we can check SHA; only query issues that are closed 3 days ago or get other info to verify this.
We may need to create/reopen the issue if the problem still exists

llxia commented 3 years ago

PR: https://github.com/AdoptOpenJDK/openjdk-tests/pull/1978 PR: https://github.com/AdoptOpenJDK/TKG/pull/105

smlambert commented 3 years ago

We can break this issue down to smaller tasks to use for the GHC opensource day (FYI @llxia @sophia-guo).

llxia commented 3 years ago

I tried to break this down into smaller tasks:

2698
adoptium/aqa-tests#3345
2701
2702

smlambert commented 1 year ago

adoptium / aqa-tests

Enable a Disabled tests pipeline or workflow #1853

2698

2699

2701

2702

2698

2701

2702