search

adoptium / aqa-systemtest

Java load testing and other full system application tests
Apache License 2.0
20 stars 67 forks source link

Modularity tests for implied readability fails #200

Open Mesbah-Alam opened 5 years ago

Mesbah-Alam commented 5 years ago

Modularity tests for implied readability fails on on both hotspot and openj9 with following error. Likely the Modularity access rules have changed since the test was developed. Need investigation. Test should be disabled for the time being. 

10:58:08 T4  1) test2(net.adoptopenjdk.test.modularity.junit.TestImpliedReadability)
10:58:08 T4  java.lang.IllegalAccessError: class net.adoptopenjdk.test.modularity.junit.TestImpliedReadability (in module com.test) cannot access class org.hamcrest.CoreMatchers (in module hamcrest.core) because module com.test does not read module hamcrest.core
10:58:08 T4     at com.test/net.adoptopenjdk.test.modularity.junit.TestImpliedReadability.test2(TestImpliedReadability.java:66)
10:58:08 T4     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10:58:08 T4     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
10:58:08 T4     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
10:58:08 T4     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
10:58:08 T4     at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
10:58:08 T4     at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
10:58:08 T4     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
10:58:08 T4     at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
10:58:08 T4     at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
10:58:08 T4     at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
10:58:08 T4     at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
10:58:08 T4     at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
10:58:08 T4     at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
10:58:08 T4     at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
10:58:08 T4     at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
10:58:08 T4     at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
10:58:08 T4     at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
10:58:08 T4     at org.junit.runners.Suite.runChild(Suite.java:128)
10:58:08 T4     at org.junit.runners.Suite.runChild(Suite.java:27)
10:58:08 T4     at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
10:58:08 T4     at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
10:58:08 T4     at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
10:58:08 T4     at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
10:58:08 T4     at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
10:58:08 T4     at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
10:58:08 T4     at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
10:58:08 T4     at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
10:58:08 T4     at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
10:58:08 T4     at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
10:58:08 T4

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/479/console

lumpfish commented 3 years ago

This test has been renamed ExplMod

lumpfish commented 3 years ago

The test reported above is expected to fail - after the test case has run the test output is checked to make sure the exception was thrown.

The test fails on openj9 only because the text in the exception is not expected by the test case.

This is the output from a hotspot test (test passes):

10:46:09  STF 09:46:09.642 - +------ Step 19 - Run TestImpliedReadability
10:46:09  STF 09:46:09.643 - | Run foreground process
10:46:09  STF 09:46:09.643 - |   Program:     /home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java
10:46:09  STF 09:46:09.643 - |   Mnemonic:    T1
10:46:09  STF 09:46:09.643 - |   Echo:        ECHO_ON
10:46:09  STF 09:46:09.643 - |   Expectation: NON_ZERO_EXIT [1] within 1m
10:46:09  STF 09:46:09.643 - |
10:46:09  STF 09:46:09.643 - Running command: /home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java --add-reads com.test=junit --module-path /home/jenkins/workspace/Grinder/jvmtest/system/openjdk-systemtest/openjdk.test.modularity/bin/tests:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/modules/1.com.hello.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/modules/4.com.discreet.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/tmp/com.hola.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/tmp/com.helper.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/results/thirdpartyjar/junit.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/results/thirdpartyjar/hamcrest-core.jar --add-modules=com.test,com.hola,junit -XX:-UseCompressedOops -classpath /home/jenkins/workspace/Grinder/jvmtest/system/openjdk-systemtest/openjdk.test.modularity/bin org.junit.runner.JUnitCore net.adoptopenjdk.test.modularity.junit.TestImpliedReadability
10:46:09  STF 09:46:09.643 - Redirecting stderr to /home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/results/19.T1.stderr
10:46:09  STF 09:46:09.643 - Redirecting stdout to /home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/results/19.T1.stdout
10:46:09  STF 09:46:09.645 - Monitoring processes: T1 
10:46:10  T1  JUnit version 4.12
10:46:10  T1  ..E
10:46:10  T1  Time: 0.015
10:46:10  T1  There was 1 failure:
10:46:10  T1  1) test2(net.adoptopenjdk.test.modularity.junit.TestImpliedReadability)
10:46:10  T1  java.lang.IllegalAccessError: class net.adoptopenjdk.test.modularity.junit.TestImpliedReadability (in module com.test) cannot access class org.hamcrest.CoreMatchers (in module hamcrest.core) because module com.test does not read module hamcrest.core
10:46:10  T1    at com.test/net.adoptopenjdk.test.modularity.junit.TestImpliedReadability.test2(TestImpliedReadability.java:66)
10:46:10  T1    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10:46:10  T1    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
10:46:10  T1    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
10:46:10  T1    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
10:46:10  T1    at junit/org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
10:46:10  T1    at junit/org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
10:46:10  T1    at junit/org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
10:46:10  T1    at junit/org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
10:46:10  T1    at junit/org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
10:46:10  T1    at junit/org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
10:46:10  T1    at junit/org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
10:46:10  T1    at junit/org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
10:46:10  T1    at junit/org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
10:46:10  T1    at junit/org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
10:46:10  T1    at junit/org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
10:46:10  T1    at junit/org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
10:46:10  T1    at junit/org.junit.runners.ParentRunner.run(ParentRunner.java:363)
10:46:10  T1    at junit/org.junit.runners.Suite.runChild(Suite.java:128)
10:46:10  T1    at junit/org.junit.runners.Suite.runChild(Suite.java:27)
10:46:10  T1    at junit/org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
10:46:10  T1    at junit/org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
10:46:10  T1    at junit/org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
10:46:10  T1    at junit/org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
10:46:10  T1    at junit/org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
10:46:10  T1    at junit/org.junit.runners.ParentRunner.run(ParentRunner.java:363)
10:46:10  T1    at junit/org.junit.runner.JUnitCore.run(JUnitCore.java:137)
10:46:10  T1    at junit/org.junit.runner.JUnitCore.run(JUnitCore.java:115)
10:46:10  T1    at junit/org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
10:46:10  T1    at junit/org.junit.runner.JUnitCore.main(JUnitCore.java:36)
10:46:10  T1  
10:46:10  T1  FAILURES!!!
10:46:10  T1  Tests run: 2,  Failures: 1
10:46:10  T1  
10:46:10  STF 09:46:10.314 - Monitoring Report Summary:
10:46:10  STF 09:46:10.314 -   o Process T1  ended with the expected exit code (1)
10:46:10  STF 09:46:10.315 - 
10:46:10  STF 09:46:10.315 - +------ Step 20 - Check output
10:46:10  STF 09:46:10.315 - | Count file matches
10:46:10  STF 09:46:10.315 - |   TargetFile:         /home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/results/19.T1.stdout
10:46:10  STF 09:46:10.315 - |   ExpectedNumMatches: 1
10:46:10  STF 09:46:10.315 - |   SearchStrings:      ["java.lang.IllegalAccessError"]
10:46:10  STF 09:46:10.315 - |
10:46:10  STF 09:46:10.315 - Found 1 instances of ["java.lang.IllegalAccessError"]
10:46:10  STF 09:46:10.315 - 
10:46:10  STF 09:46:10.315 - +------ Step 21 - Check output
10:46:10  STF 09:46:10.315 - | Count file matches
10:46:10  STF 09:46:10.315 - |   TargetFile:         /home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_16170111568043/ExplMod_1/20210329-094557-ExplMod/results/19.T1.stdout
10:46:10  STF 09:46:10.315 - |   ExpectedNumMatches: 1
10:46:10  STF 09:46:10.315 - |   SearchStrings:      ["module com.test does not read module hamcrest.core"]
10:46:10  STF 09:46:10.315 - |
10:46:10  STF 09:46:10.315 - Found 1 instances of ["module com.test does not read module hamcrest.core"]```

This is the output from an openj9 test:

11:27:29  STF 10:27:28.800 - +------ Step 19 - Run TestImpliedReadability
11:27:29  STF 10:27:28.800 - | Run foreground process
11:27:29  STF 10:27:28.800 - |   Program:     /home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java
11:27:29  STF 10:27:28.800 - |   Mnemonic:    T1
11:27:29  STF 10:27:28.800 - |   Echo:        ECHO_ON
11:27:29  STF 10:27:28.800 - |   Expectation: NON_ZERO_EXIT [1] within 1m
11:27:29  STF 10:27:28.800 - |
11:27:29  STF 10:27:28.801 - Running command: /home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java --add-reads com.test=junit --module-path /home/jenkins/workspace/Grinder/jvmtest/system/openjdk-systemtest/openjdk.test.modularity/bin/tests:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/modules/1.com.hello.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/modules/4.com.discreet.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/tmp/com.hola.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/tmp/com.helper.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/results/thirdpartyjar/junit.jar:/home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/results/thirdpartyjar/hamcrest-core.jar --add-modules=com.test,com.hola,junit -XX:-UseCompressedOops -classpath /home/jenkins/workspace/Grinder/jvmtest/system/openjdk-systemtest/openjdk.test.modularity/bin org.junit.runner.JUnitCore net.adoptopenjdk.test.modularity.junit.TestImpliedReadability
11:27:29  STF 10:27:28.801 - Redirecting stderr to /home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/results/19.T1.stderr
11:27:29  STF 10:27:28.801 - Redirecting stdout to /home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/results/19.T1.stdout
11:27:29  STF 10:27:28.802 - Monitoring processes: T1 
11:27:29  T1  JUnit version 4.12
11:27:29  T1  ..E
11:27:29  T1  Time: 0.011
11:27:29  T1  There was 1 failure:
11:27:29  T1  1) test2(net.adoptopenjdk.test.modularity.junit.TestImpliedReadability)
11:27:29  T1  java.lang.IllegalAccessError: Class net/adoptopenjdk/test/modularity/junit/TestImpliedReadability(module com.test) can not access class org/hamcrest/CoreMatchers(module hamcrest.core) because module module com.test does not read module module hamcrest.core 
11:27:29  T1    at com.test/net.adoptopenjdk.test.modularity.junit.TestImpliedReadability.test2(TestImpliedReadability.java:66)
11:27:29  T1    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:27:29  T1    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:27:29  T1    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:27:29  T1    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
11:27:29  T1    at junit/org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
11:27:29  T1    at junit/org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
11:27:29  T1    at junit/org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
11:27:29  T1    at junit/org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
11:27:29  T1    at junit/org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
11:27:29  T1    at junit/org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
11:27:29  T1    at junit/org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
11:27:29  T1    at junit/org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
11:27:29  T1    at junit/org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
11:27:29  T1    at junit/org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
11:27:29  T1    at junit/org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
11:27:29  T1    at junit/org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
11:27:29  T1    at junit/org.junit.runners.ParentRunner.run(ParentRunner.java:363)
11:27:29  T1    at junit/org.junit.runners.Suite.runChild(Suite.java:128)
11:27:29  T1    at junit/org.junit.runners.Suite.runChild(Suite.java:27)
11:27:29  T1    at junit/org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
11:27:29  T1    at junit/org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
11:27:29  T1    at junit/org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
11:27:29  T1    at junit/org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
11:27:29  T1    at junit/org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
11:27:29  T1    at junit/org.junit.runners.ParentRunner.run(ParentRunner.java:363)
11:27:29  T1    at junit/org.junit.runner.JUnitCore.run(JUnitCore.java:137)
11:27:29  T1    at junit/org.junit.runner.JUnitCore.run(JUnitCore.java:115)
11:27:29  T1    at junit/org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
11:27:29  T1    at junit/org.junit.runner.JUnitCore.main(JUnitCore.java:36)
11:27:29  T1  
11:27:29  T1  FAILURES!!!
11:27:29  T1  Tests run: 2,  Failures: 1
11:27:29  T1  
11:27:29  STF 10:27:29.326 - Monitoring Report Summary:
11:27:29  STF 10:27:29.326 -   o Process T1  ended with the expected exit code (1)
11:27:29  STF 10:27:29.327 - 
11:27:29  STF 10:27:29.327 - +------ Step 20 - Check output
11:27:29  STF 10:27:29.327 - | Count file matches
11:27:29  STF 10:27:29.327 - |   TargetFile:         /home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/results/19.T1.stdout
11:27:29  STF 10:27:29.327 - |   ExpectedNumMatches: 1
11:27:29  STF 10:27:29.327 - |   SearchStrings:      ["java.lang.IllegalAccessError"]
11:27:29  STF 10:27:29.327 - |
11:27:29  STF 10:27:29.327 - Found 1 instances of ["java.lang.IllegalAccessError"]
11:27:29  STF 10:27:29.327 - 
11:27:29  STF 10:27:29.327 - +------ Step 21 - Check output
11:27:29  STF 10:27:29.327 - | Count file matches
11:27:29  STF 10:27:29.327 - |   TargetFile:         /home/jenkins/workspace/Grinder/openjdk-tests/TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/results/19.T1.stdout
11:27:29  STF 10:27:29.327 - |   ExpectedNumMatches: 1
11:27:29  STF 10:27:29.327 - |   SearchStrings:      ["module com.test does not read module hamcrest.core"]
11:27:29  STF 10:27:29.327 - |
11:27:29  **FAILED** at step 21 (Check output). Expected return value=1 Actual=0 at /home/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/output_1617013623909/ExplMod_1/20210329-102717-ExplMod/execute.pl line 745.
11:27:29  STF 10:27:29.408 - **FAILED** execute script failed. Expected return value=0 Actual=1

So the test fails because on openj9 the java.lang.IllegalAccessError text does not contain the expected text module com.test does not read module hamcrest.core, instead it contains module com.test does not read module module hamcrest.core (note the extra 'module').

The openj9 message also includes slashes rather than dots in the referenced class name.

hotspot message:

java.lang.IllegalAccessError: class net.adoptopenjdk.test.modularity.junit.TestImpliedReadability (in module com.test) cannot access class org.hamcrest.CoreMatchers (in module hamcrest.core) because module com.test does not read module hamcrest.core

openj9 message:

java.lang.IllegalAccessError: Class net/adoptopenjdk/test/modularity/junit/TestImpliedReadability(module com.test) can not access class org/hamcrest/CoreMatchers(module hamcrest.core) because module module com.test does not read module module hamcrest.core
lumpfish commented 3 years ago

Raised https://github.com/eclipse/openj9/issues/12315. Depending on the Eclipse openj9 response the test case may need to be changed to accept the exception text from either implementation.