Test results show wrong test class

eclipse-platform / eclipse.platform.releng.aggregator

Aggregated repository for Eclipse Java IDE

https://www.eclipse.org/eclipse/

Eclipse Public License 2.0

36 stars 75 forks source link

Test results show wrong test class #2034

Open stephan-herrmann opened 5 months ago

stephan-herrmann commented 5 months ago

I was searching for recent test results for this test: org.eclipse.jdt.core.tests.compiler/org.eclipse.jdt.core.tests.compiler.regression.ScannerTest.test034() But in none of the recent test results like I20240519-1800 I could find any trace of the test. In fact searching for .ScannerTest shows exactly 5 matches while 86 matches should be expected.

By contrast, github-triggered builds like https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/master/222/testReport/org.eclipse.jdt.core.tests.compiler.regression/ScannerTest/ show the expected number of tests (here multiplied with 5 for 5 compliance levels).

Going back all the way to 4.29 it shows 18 tests, still far from complete.

stephan-herrmann commented 5 months ago

@iloveeclipse has an initial result in this comment

stephan-herrmann commented 5 months ago

If this is broken at all, it probably broken at the test report or creation level (org.eclipse.jdt.core.tests.junit.extension.TestCase.buildTestsList(Class, int, long) ?), I see that there are tests from ScannerTest like testBug531716_001_since_13 but they are marked as org.eclipse.jdt.core.tests.compiler.regression.TestAll.

OMG, you're right, there are 196 results for "test034", only 2 of which are assigned to a specific test class.

stephan-herrmann commented 5 months ago

At the end of org.eclipse.jdt.core.tests.compiler.regression.TestAll.suite() the structure looks like this:

all = TestSuite ("org.eclipse.jdt.core.tests.compiler.regression.TestAll")
- fTests[3] = RegressionTestSetup
  - nested fTests 158 elements, e.g.:
    - fTests[13] = TestSuite("org.eclipse.jdt.core.tests.compiler.regression.ScannerTest")
      - fTests has 86 elements as expected.

This runtime structure looks OK and I'm not aware of anything that might have changed here in years.

How does the I-build create & collect test results?

jukzi commented 5 months ago

The file is not incomplete. jenkins reads the result from an .xml which is generated when the tests run https://ci.eclipse.org/releng/job/AutomatedTests/job/ep432I-unit-cen64-gtk3-java17/lastSuccessfulBuild/artifact/workarea/I20240520-1800/eclipse-testing/results/xml/org.eclipse.jdt.core.tests.compiler_ep432I-unit-cen64-gtk3-java17_linux.gtk.x86_64_17.xml

I never figured out how exactly the values inside the .xml are generated, but both Testnames and time are reproducible wrong. The testname probably is wrongly setup in the junit test/suite itself.