OpenJDK jdk/modules/etc/DefaultModules, jdk/modules/incubator/ImageModules - IOException: module-info.class not found

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64le_linux_Nightly_testList_0/84/ https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/77/ https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_OMR_testList_0/32/ https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_s390x_linux_OMR_testList_0/36/

jdk_lang_j9_0 jdk/modules/etc/DefaultModules.java jdk/modules/incubator/ImageModules.java

20:42:07  Command line: [/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/java -cp /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/aqa-tests/TKG/output_1698446836232/jdk_lang_j9_0/work/classes/jdk/modules/etc/DefaultModules.d:/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/aqa-tests/openjdk/openjdk-jdk/test/jdk/jdk/modules/etc:/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/aqa-tests/TKG/output_1698446836232/jdk_lang_j9_0/work/classes/test/lib:/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/jvmtest/openjdk/jtreg/lib/jtreg.jar:/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/jvmtest/openjdk/jtreg/lib/junit-platform-console-standalone-1.9.2.jar:/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/jvmtest/openjdk/jtreg/lib/testng-7.3.0.jar:/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/jvmtest/openjdk/jtreg/lib/guice-5.1.0.jar:/home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/jvmtest/openjdk/jtreg/lib/jcommander-1.82.jar -ea -esa -Xmx512m -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -Xjit -Xgcpolicy:balanced -Xnocompressedrefs --add-exports java.base/jdk.internal.module=ALL-UNNAMED /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/aqa-tests/openjdk/openjdk-jdk/test/jdk/jdk/modules/etc/TestRootModules.java ]
20:42:07  [2023-10-28T00:31:59.457479Z] Gathering output for process 37028044
20:42:07  [2023-10-28T00:32:02.163463Z] Waiting for completion for process 37028044
20:42:07  [2023-10-28T00:32:02.163763Z] Waiting for completion finished for process 37028044
20:42:07  Output and diagnostic info for process 37028044 was saved into 'pid-37028044-output.log'
20:42:07  
20:42:07  [2023-10-28T00:32:02.165168Z] Waiting for completion for process 37028044
20:42:07  [2023-10-28T00:32:02.165282Z] Waiting for completion finished for process 37028044
20:42:07  Error: java.io.IOException: module-info.class not found: /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/openjdkbinary/j2sdk-image/jmods/java.base.jmod
20:42:07  java.lang.module.FindException: java.io.IOException: module-info.class not found: /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/openjdkbinary/j2sdk-image/jmods/java.base.jmod
20:42:07    at java.base/jdk.internal.module.ModulePath.scan(ModulePath.java:254)
20:42:07    at java.base/jdk.internal.module.ModulePath.scanNextEntry(ModulePath.java:191)
20:42:07    at java.base/jdk.internal.module.ModulePath.find(ModulePath.java:155)
20:42:07    at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.newModuleFinder(JlinkTask.java:455)
20:42:07    at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.initJlinkConfig(JlinkTask.java:392)
20:42:07    at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:281)
20:42:07    at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:56)
20:42:07    at jdk.jlink/jdk.tools.jlink.internal.Main$JlinkToolProvider.run(Main.java:73)
20:42:07    at java.base/java.util.spi.ToolProvider.run(ToolProvider.java:162)
20:42:07    at DefaultModules.main(DefaultModules.java:79)
20:42:07    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
20:42:07    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
20:42:07    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:333)
20:42:07    at java.base/java.lang.Thread.run(Thread.java:1595)
20:42:07  Caused by: java.io.IOException: module-info.class not found: /home/jenkins/workspace/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_Nightly_testList_0/openjdkbinary/j2sdk-image/jmods/java.base.jmod
20:42:07    at java.base/jdk.internal.jmod.JmodFile.getInputStream(JmodFile.java:212)
20:42:07    at java.base/jdk.internal.module.ModulePath.readJMod(ModulePath.java:395)
20:42:07    at java.base/jdk.internal.module.ModulePath.readModule(ModulePath.java:344)
20:42:07    at java.base/jdk.internal.module.ModulePath.scanDirectory(ModulePath.java:285)
20:42:07    at java.base/jdk.internal.module.ModulePath.scan(ModulePath.java:233)
20:42:07    ... 13 more

Changes: https://github.com/eclipse-openj9/openj9/compare/ae0b30aeeb7...3f1a6b2ca8a https://github.com/eclipse-openj9/openj9-omr/compare/071c0c25b147...386a7080ffa

[pshipton]

I think the following is related, a number of tests failing in Metronome modes.

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_extended.functional_ppc64_aix_Nightly/80 gcNotificationTest_Metronome_0 gcNotificationTest_Metronome_1 threadMXBeanTestSuite2_2 threadMXBeanTimedParkTest_1 threadMXBeanTimersTest_1 jniOnLoadExceptions_3 testvmcheck_3 threadMXBeanTestSuite1_5 threadMXBeanTestSuite1_6 threadMXBeanTestSuite2_1 threadMXBeanTimersTest_2 jniOnLoadExceptions_2

TESTING:
[TestNG] [ERROR] 
Cannot find class in classpath: org.openj9.test.gpu.SortTest
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "org.testng.internal.ExitCodeListener.hasTests()" because "this.exitCodeListener" is null
    at org.testng.TestNG.getStatus(TestNG.java:211)
    at org.testng.TestNG.main(TestNG.java:1324)

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_special.system_s390x_linux_Personal_testList_4/16/ MiniMix_3h_0 -XX:+UseCompressedOops -Xjit -Xgcpolicy:balanced

LT    Failure num.  = 1
LT    Test number   = 3231
LT    Test details  = 'Mauve[gnu.testlet.java.util.Arrays.equals]'
LT    Suite number  = 0
LT    Thread number = 6
LT  >>> Captured test output >>>
LT  Test failed:
LT  java.lang.ClassNotFoundException: gnu.testlet.java.util.Arrays.equals
LT      at java.base/java.lang.Class.forNameImpl(Native Method)
LT      at java.base/java.lang.Class.forName(Class.java:374)
LT      at java.base/java.lang.Class.forName(Class.java:350)
LT      at net.adoptopenjdk.loadTest.adaptors.MauveAdaptor.executeTest(MauveAdaptor.java:51)
LT      at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:182)
LT      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
LT      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
LT      at java.base/java.lang.Thread.run(Thread.java:1595)
LT  <<<

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_special.system_ppc64_aix_Personal_testList_1/16/ MathLoadTest_bigdecimal_special_5m_11 -Xgcpolicy:metronome -Xcompressedrefs

MLT stderr Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.logging.log4j.core.async.ThreadNameCachingStrategy$2
MLT stderr  at org.apache.logging.log4j.core.impl.ReusableLogEventFactory.<clinit>(ReusableLogEventFactory.java:38)
MLT stderr  at org.apache.logging.log4j.core.config.LoggerConfig.<clinit>(LoggerConfig.java:96)
MLT stderr  at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:133)
MLT stderr  at org.apache.logging.log4j.core.config.NullConfiguration.<init>(NullConfiguration.java:32)
MLT stderr  at org.apache.logging.log4j.core.LoggerContext.<clinit>(LoggerContext.java:85)
MLT stderr  at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:254)
MLT stderr  at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:218)
MLT stderr  at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:140)
MLT stderr  at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:123)
MLT stderr  at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:230)
MLT stderr  at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
MLT stderr  at org.apache.logging.log4j.LogManager.getContext(LogManager.java:176)
MLT stderr  at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:666)
MLT stderr  at net.adoptopenjdk.loadTest.LoadTest.<clinit>(LoadTest.java:46)
MLT stderr Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.core.async.ThreadNameCachingStrategy$2
MLT stderr  at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:827)
MLT stderr  at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
MLT stderr  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:1104)
MLT stderr  ... 14 more

[tajila]

Grinder with standard options: https://openj9-jenkins.osuosl.org/job/Grinder/3025/ Grinder with -Xint: https://openj9-jenkins.osuosl.org/job/Grinder/3026/

[pshipton]

At least for the original problem running jdk_lang_j9, it is resolved by reverting https://github.com/eclipse-openj9/openj9/pull/16662 I'll rebuild AIX without this change and check the other failures.

@jdmpapin @Spencer-Comin

[JasonFengJ9]

Just note that this failure is observed across platforms in the latest JDK21 internal weekly runs.

[pshipton]

With https://github.com/eclipse-openj9/openj9/pull/16662 reverted, https://github.com/eclipse-openj9/openj9/issues/18371#issuecomment-1785255576 appears resolved. It was easily reproducible using the "bad" build.

@TobiAjila I don't think we need the grinders in https://github.com/eclipse-openj9/openj9/issues/18371#issuecomment-1785318306, I've stopped them.

[pshipton]

With https://github.com/eclipse-openj9/openj9/pull/16662 reverted, https://github.com/eclipse-openj9/openj9/issues/18371#issuecomment-1785306896 didn't fail in a 5x grinder either. I'm not going to re-test the 3 hour https://github.com/eclipse-openj9/openj9/issues/18371#issuecomment-1785295205.

[Spencer-Comin]

I ran jdk_lang_j9_0 in a grinder internally (https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/35776) with the environment variable TR_disableInlineVectorizedMismatch set and got the same error, so I assume the offending bug in https://github.com/eclipse-openj9/openj9/pull/16662 is not the ArraysSupport.vectorizedMismatch acceleration itself but some other change in the PR that enabled the acceleration. Edit: I hardcoded vectorizedMismatch support to false and the test passed, so the environment variable must not have been passed correctly through Jenkins.

[Spencer-Comin]

This appears to be an arraylets issue, since vectorizedMismatch and arraylets don't play well together and the failures are with metronome and balanced gc policies. I've done some preliminary internal tests with the changes from https://github.com/eclipse/omr/pull/7108 and that seems to solve the problem.

[Spencer-Comin]

I've done more thorough internal testing (see [1]) with the changes from https://github.com/eclipse/omr/pull/7108, and I don't see any of the failures here showing up. Now that https://github.com/eclipse/omr/pull/7108 has been merged, hopefully this issue disappears.

[1] https://hyc-runtimes-jenkins.swg-devops.com/view/OpenJ9%20-%20Personal/job/Pipeline-Build-Test-Personal/18982/

[pshipton]

I've started an OMR acceptance build.

[hzongaro]

@Spencer-Comin, @pshipton, does any more testing of this issue need to happen, or can it be closed?

[pshipton]

The change was only in open source builds as of last night. The builds look good.