search

adoptium / aqa-tests

Home of test infrastructure for Adoptium builds
https://adoptium.net/aqavit
Apache License 2.0
132 stars 312 forks source link

Capture TKG test result logic may not work with ITERATIONS #4092

Open llxia opened 1 year ago

llxia commented 1 year ago

Describe the bug With the capture TKG test result, we can visualize the test summary in the Jenkins job (see below). This is really handy for the users.

image

However, this feature may give false positives or false negatives when running with ITERATIONS. In the following case (Grinder link), the test has an intermittent failure. It passed 3/5. TEST TARGETS SUMMARY shows a false positive.

image

I think we should update the logic to capture all iteration results. https://github.com/adoptium/aqa-tests/blob/9166b61bddc5156f6d3182ff536a5e035293452a/buildenv/jenkins/JenkinsfileBase#L673-L710

As we enable Rerun failed tests automatically feature, this issue will show up in the nightly runs with rerun enabled. Also, the good news is that the job status is correct (i.e., UNSTABLE) in this case because we also check TAP files.

FYI @renfeiw

renfeiw commented 1 year ago

Currently we are using manager.getLogMatcher to get the match, but it only checks the first occurence. I will try parsing manager.build.logFile to see it works.