search

adoptium / aqa-systemtest

Java load testing and other full system application tests
Apache License 2.0
21 stars 67 forks source link

Check validity of modularity service loaders test #137

Open Mesbah-Alam opened 6 years ago

Mesbah-Alam commented 6 years ago

Modularity service loaders test fails with following error on all platforms on both hotspot and openj9 jdk9 and jdk10 builds. This test was written based on old modularity implementations and likely has become invalid. The check at which the test fails may need to be updated/ removed.

https://ci.adoptopenjdk.net/view/System%20tests/job/openjdk9_hs_systemtest_s390x_linux/27/tapResults/

===============================================
Running test ServiceLoadersTest_0 ...
===============================================
ServiceLoadersTest_0 Start Time: Tue Jun 19 02:22:10 2018 Epoch Time (ms): 1529389330131
test with NoOptions
STF 02:22:10.285 - =========================   S T F   =========================
systemtest-prereqs has been processed, and set to: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/systemtest_prereqsRetrieving amount of free space on drive containing /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0
There is 43975 Mb free
STF 02:22:10.294 - ==================   G E N E R A T I O N   ==================
STF 02:22:10.295 - Checking JVM: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../
STF 02:22:10.295 - Starting process to generate scripts: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/..//bin/java  -Dlog4j.skipJansi=true -Djava.system.class.loader=net.adoptopenjdk.stf.runner.StfClassLoader -classpath /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/systemtest/systemtest_prereqs/log4j-2.3/log4j-api-2.3.jar:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/systemtest/systemtest_prereqs/log4j-2.3/log4j-core-2.3.jar:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/stf/stf.core/scripts/../bin net.adoptopenjdk.stf.runner.StfRunner -properties "/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/stf_parameters.properties, , /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/stf/stf.core/config/stf.properties" -testDir "/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest"
GEN 02:22:11.322 - Found test. Project: 'openjdk.test.modularity' class: 'ServiceLoadersTest.class' Dir: '/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin'
GEN 02:22:11.332 - Found test. Project: 'openjdk.test.modularity' class: 'net.adoptopenjdk.test.modularity.ServiceLoadersTest'
GEN Classpath directories used by project 'openjdk.test.modularity': 
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/displayService
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/com.helper
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/displayServiceImpl1
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/com.discreet
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/com.hello
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/com.hola
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/displayServiceImpl2
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/com.hi
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-override
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-override-new
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common/displayServiceImpl3
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common/displayServiceImpl4
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/tests
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/stf/stf.core/bin
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/systemtest_prereqs/log4j-2.3/log4j-api-2.3.jar
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/systemtest_prereqs/log4j-2.3/log4j-core-2.3.jar
GEN   /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/systemtest_prereqs/junit-4.12/junit-4.12.jar
GEN 02:22:11.552 - Using Mode NoOptions. Values = ''
GEN 02:22:11.619 - 
GEN 02:22:11.619 - Test command summary:
GEN 02:22:11.620 -   Step  Stage   Command           Description
GEN 02:22:11.620 -  -----+--------+-----------------+------------
GEN 02:22:11.620 -     1  execute  jar               Create modular jar displayServiceImpl1.jar
GEN 02:22:11.620 -     2  execute  jar               create project jar displayServiceImpl3.jar
GEN 02:22:11.620 -     3  execute  jar               create project jar displayServiceImpl4.jar
GEN 02:22:11.620 -     4  execute  Run java          Run service loader test by adding the use of display service in module-info.java
GEN 02:22:11.620 -     5  execute  Run java          Run serviceloader test2 with the test as an explicit module. It is same as test1, except that requires and uses are added through addReads() and addUses() through code instead of adding them into module-info.java
GEN 02:22:11.620 -     6  execute  jar               create project jar com.test.serviceloaders2.jar
GEN 02:22:11.620 -     7  execute  Run java          Run serviceloader test3 is same as test2 except that the TestServiceLoaders2 is in an automatic module
STF 02:22:11.669 - 
STF 02:22:11.669 - Script generation completed
STF 02:22:11.669 - 
STF 02:22:11.670 - 
STF 02:22:11.670 - =======================   S E T U P   =======================
STF 02:22:11.670 - Running setup: perl /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/setUp.pl
STF 02:22:11.726 - SETUP stage completed
STF 02:22:11.734 - 
STF 02:22:11.734 - ====================   E X E C U T E -   ====================
STF 02:22:11.734 - Running execute: perl /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/execute.pl
STF 02:22:11.795 - 
STF 02:22:11.795 - Java version
STF 02:22:11.795 - Running: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../bin/java -version
openjdk version "9.0.4-adoptopenjdk"
OpenJDK Runtime Environment (build 9.0.4-adoptopenjdk+11)
OpenJDK 64-Bit Server VM (build 9.0.4-adoptopenjdk+11, mixed mode)
STF 02:22:11.911 - 
STF 02:22:11.911 - +------ Step 1 - Create modular jar displayServiceImpl1.jar
STF 02:22:11.911 - | Create modular jar
STF 02:22:11.911 - |   ModuleSpec:    openjdk.test.modularity/bin/common-mods/displayServiceImpl1
STF 02:22:11.911 - |   ModuleVersion: null
STF 02:22:11.911 - |   MainClass:     null
STF 02:22:11.911 - |   HashModules:   null
STF 02:22:11.911 - |
STF 02:22:11.912 - Running command: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../bin/jar --create --file /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/modules/1.displayServiceImpl1.jar -C /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods/displayServiceImpl1 .
STF 02:22:11.912 - Redirecting stderr to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/1.CMJ.stderr
STF 02:22:11.912 - Redirecting stdout to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/1.CMJ.stdout
STF 02:22:11.921 - Monitoring processes: CMJ
STF 02:22:12.207 - Monitoring Report Summary:
STF 02:22:12.207 -   o Process CMJ ended with the expected exit code (0)
STF 02:22:12.208 - Created modular jar: 1.displayServiceImpl1.jar
STF 02:22:12.208 - 
STF 02:22:12.208 - +------ Step 2 - create project jar displayServiceImpl3.jar
STF 02:22:12.208 - | Create project jar
STF 02:22:12.208 - |   ProjectSpec: openjdk.test.modularity/bin/common/displayServiceImpl3
STF 02:22:12.208 - |   ArchiveSpec: 
STF 02:22:12.208 - |
STF 02:22:12.208 - Running command: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../bin/jar --create --file /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/tmp/displayServiceImpl3.jar -C /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common/displayServiceImpl3 .
STF 02:22:12.208 - Redirecting stderr to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/2.JAR.stderr
STF 02:22:12.208 - Redirecting stdout to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/2.JAR.stdout
STF 02:22:12.209 - Monitoring processes: JAR
STF 02:22:12.550 - Monitoring Report Summary:
STF 02:22:12.550 -   o Process JAR ended with the expected exit code (0)
STF 02:22:12.551 - Created project jar: displayServiceImpl3.jar
STF 02:22:12.551 - 
STF 02:22:12.551 - +------ Step 3 - create project jar displayServiceImpl4.jar
STF 02:22:12.551 - | Create project jar
STF 02:22:12.551 - |   ProjectSpec: openjdk.test.modularity/bin/common/displayServiceImpl4
STF 02:22:12.551 - |   ArchiveSpec: 
STF 02:22:12.551 - |
STF 02:22:12.551 - Running command: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../bin/jar --create --file /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/tmp/displayServiceImpl4.jar -C /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common/displayServiceImpl4 .
STF 02:22:12.551 - Redirecting stderr to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/3.JAR.stderr
STF 02:22:12.551 - Redirecting stdout to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/3.JAR.stdout
STF 02:22:12.553 - Monitoring processes: JAR
STF 02:22:12.930 - Monitoring Report Summary:
STF 02:22:12.930 -   o Process JAR ended with the expected exit code (0)
STF 02:22:12.930 - Created project jar: displayServiceImpl4.jar
STF 02:22:12.930 - 
STF 02:22:12.930 - +------ Step 4 - Run service loader test by adding the use of display service in module-info.java
STF 02:22:12.930 - | Run foreground process
STF 02:22:12.930 - |   Program:     /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../bin/java
STF 02:22:12.930 - |   Mnemonic:    T1
STF 02:22:12.930 - |   Echo:        ECHO_ON
STF 02:22:12.931 - |   Expectation: CLEAN_RUN within 1m
STF 02:22:12.931 - |
STF 02:22:12.931 - Running command: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../bin/java --add-reads com.test.serviceloaders=ALL-UNNAMED --module-path /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/modules/1.displayServiceImpl1.jar:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/tests:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/tmp/displayServiceImpl3.jar --add-modules=com.test.serviceloaders,displayService -classpath /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/systemtest_prereqs/junit-4.12/junit-4.12.jar:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/systemtest_prereqs/junit-4.12/hamcrest-core-1.3.jar:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/tmp/displayServiceImpl4.jar org.junit.runner.JUnitCore adoptopenjdk.test.modularity.serviceloaders.TestServiceLoaders
STF 02:22:12.931 - Redirecting stderr to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/4.T1.stderr
STF 02:22:12.931 - Redirecting stdout to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/4.T1.stdout
STF 02:22:12.933 - Monitoring processes: T1 
T1  JUnit version 4.12
T1  .DisplayServiceImpl1
T1  DisplayServiceImpl2
T1  DisplayServiceImpl3
T1  DisplayServiceImpl4
T1  
T1  Time: 0.051
T1  
T1  OK (1 test)
T1  
STF 02:22:13.407 - Monitoring Report Summary:
STF 02:22:13.408 -   o Process T1  ended with the expected exit code (0)
STF 02:22:13.408 - 
STF 02:22:13.408 - +------ Step 5 - Run serviceloader test2 with the test as an explicit module. It is same as test1, except that requires and uses are added through addReads() and addUses() through code instead of adding them into module-info.java
STF 02:22:13.408 - | Run foreground process
STF 02:22:13.408 - |   Program:     /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../bin/java
STF 02:22:13.408 - |   Mnemonic:    T2
STF 02:22:13.408 - |   Echo:        ECHO_ON
STF 02:22:13.408 - |   Expectation: CLEAN_RUN within 1m
STF 02:22:13.408 - |
STF 02:22:13.408 - Running command: /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdkbinary/j2sdk-image/bin/../bin/java --add-reads com.test.serviceloaders2=ALL-UNNAMED --module-path /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/modules/1.displayServiceImpl1.jar:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/common-mods:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/openjdk-systemtest/openjdk.test.modularity/bin/tests:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/tmp/displayServiceImpl3.jar --add-modules=com.test.serviceloaders2,displayService -classpath /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/systemtest_prereqs/junit-4.12/junit-4.12.jar:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/jvmtest/systemtest/systemtest_prereqs/junit-4.12/hamcrest-core-1.3.jar:/data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/tmp/displayServiceImpl4.jar org.junit.runner.JUnitCore adoptopenjdk.test.modularity.serviceloaders2.TestServiceLoaders2
STF 02:22:13.408 - Redirecting stderr to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/5.T2.stderr
STF 02:22:13.408 - Redirecting stdout to /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/results/5.T2.stdout
STF 02:22:13.410 - Monitoring processes: T2 
T2  JUnit version 4.12
T2  .Added readability edge via addReads() API from com.test.serviceloaders2 to displayService
T2  Added uses edge via addUses() API from com.test.serviceloaders2 to displayService
T2  DisplayServiceImpl4
T2  E
T2  Time: 0.058
T2  There was 1 failure:
T2  1) testServiceLoader(adoptopenjdk.test.modularity.serviceloaders2.TestServiceLoaders2)
T2  java.lang.AssertionError
T2      at org.junit.Assert.fail(Assert.java:86)
T2      at org.junit.Assert.assertTrue(Assert.java:41)
T2      at org.junit.Assert.assertTrue(Assert.java:52)
T2      at com.test.serviceloaders2/adoptopenjdk.test.modularity.serviceloaders2.TestServiceLoaders2.testServiceLoader(TestServiceLoaders2.java:138)
T2      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
T2      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
T2      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
T2      at java.base/java.lang.reflect.Method.invoke(Method.java:564)
T2      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
T2      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
T2      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
T2      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
T2      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
T2      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
T2      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
T2      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
T2      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
T2      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
T2      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
T2      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
T2      at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
T2      at org.junit.runners.Suite.runChild(Suite.java:128)
T2      at org.junit.runners.Suite.runChild(Suite.java:27)
T2      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
T2      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
T2      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
T2      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
T2      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
T2      at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
T2      at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
T2      at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
T2      at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
T2      at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
T2  
T2  FAILURES!!!
T2  Tests run: 1,  Failures: 1
T2  
STF 02:22:13.886 - **FAILED** Process T2  ended with exit code (1) and not the expected exit code/s (0)
STF 02:22:13.886 - Monitoring Report Summary:
STF 02:22:13.886 -   o Process T2  ended with exit code (1) and not the expected exit code/s (0)
STF 02:22:13.887 - Killing processes: T2 
STF 02:22:13.887 -   o Process T2  is not running
**FAILED** at step 5 (Run serviceloader test2 with the test as an explicit module. It is same as test1, except that requires and uses are added through addReads() and addUses() through code instead of adding them into module-info.java). Expected return value=0 Actual=1 at /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/execute.pl line 244.
STF 02:22:13.893 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 02:22:13.893 - 
STF 02:22:13.893 - ====================   T E A R D O W N   ====================
STF 02:22:13.893 - Running teardown: perl /data/jenkins/workspace/openjdk9_hs_systemtest_s390x_linux/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15293854726833/ServiceLoadersTest_0/20180619-022210-ServiceLoadersTest/tearDown.pl
STF 02:22:13.946 - TEARDOWN stage completed
STF 02:22:13.949 - 
STF 02:22:13.949 - =====================   R E S U L T S   =====================
STF 02:22:13.949 - Stage results:
STF 02:22:13.949 -   setUp:     pass
STF 02:22:13.949 -   execute:  *fail*
STF 02:22:13.949 -   teardown:  pass
STF 02:22:13.949 - 
STF 02:22:13.950 - Overall result: **FAILED**

ServiceLoadersTest_0_FAILED
Mesbah-Alam commented 6 years ago

Hi @chandrams

Since you had originally developed the use case for the modularity serviceloaders tests (if I am not mistaken), could you please shed some light on the following?

Two of the 3 Modularity Service loaders tests fail, consistently, both on hotspot and openj9 jdk9 and jdk10 sdks. I am wondering if the tests have become invalid, or if it is pointing to a legitimate issue.

Test 2

The first failure happens in Test 2, T2 (as coded in TestServiceLoaders2.java), where we are basically running the same test as T1 except here the requires and uses are added through addReads() and addUses() programmatically instead of adding them via requires and uses clauses in module-info.java.

The test fails here: https://github.com/AdoptOpenJDK/openjdk-systemtest/blob/bda252335d3eb7a5f9858393aa0ebbb51ca8076d/openjdk.test.modularity/src/tests/com.test.serviceloaders2/adoptopenjdk/test/modularity/serviceloaders2/TestServiceLoaders2.java#L138

As you can see in the error message in the description of the issue, it lists only one display service (DisplayServiceImpl4) instead of all 4, as listed in test1.

Test 3

I commented out test 2 for testing test 3 , and that too failed at the same validation as T2, but interestingly, this time we get 2 services in the 'actual' list, instead of 1 as seen in test2.

https://urv-jenkins.canlab.ibm.com:9095/view/SV_Tests/job/test.personal/256/console

     STF 16:19:18.124 - Monitoring processes: T3 
        T3  JUnit version 4.12
        T3  .Added readability edge via addReads() API from com.test.serviceloaders2 to displayService
        T3  Added uses edge via addUses() API from com.test.serviceloaders2 to displayService
        T3  DisplayServiceImpl3
        T3  DisplayServiceImpl4
        T3  E
        T3  Time: 0.033
        T3  There was 1 failure:

Have these two tests become invalid ?

Mesbah-Alam commented 6 years ago

The failing tests have been excluded from running in the build : https://github.com/AdoptOpenJDK/openjdk-systemtest/pull/138

chandrams commented 6 years ago

Hi @Mesbah-Alam

As discussed, these issues were seen when the tests were written for which an internal bug was raised. But this was during initial days of modularity, we will need to validate the test scenario and expected test behavior and raise a bug if required.

Mesbah-Alam commented 6 years ago

@chandrams , I've dug up the internal RTC defect that you had raised on this issue against the JCL team in August 2016 (forwarded to you on Slack) and noticed that we have indeed had the exact failure since during early days. What was the source (modularity jep, etc) of the original use case, based on which this test was written? Could you point me to that?

chandrams commented 6 years ago

@Mesbah-Alam

This is the modularity source link - http://openjdk.java.net/projects/jigsaw/spec/sotms/#services I had forwarded both the above link and RTC defect link during our chat on slack last month as well. Let me know if you need help understanding the use case. It is related to service loaders and automatic modules.

Mesbah-Alam commented 6 years ago

@chandrams, yes, I could use some help! Since you wrote it originally based on the spec, could you please explain the failure that has been happening? Was the test written correctly? Did it ever pass on any Oracle SDK?