eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

OpenJDK java/lang/String/CompactString/MaxSizeUTF16String -XX:-CompactStrings failing some platforms #20021

Closed pshipton closed 2 months ago

pshipton commented 2 months ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_aarch64_mac_Nightly_testList_0/13/ jdk_lang_1 (-XX:-UseCompressedOops) -Xcomp (-Xjit:count=0) -Xmx8g java/lang/String/CompactString/MaxSizeUTF16String.java

21:30:07  STARTED    MaxSizeUTF16String::testMaxCharArray 'testMaxCharArray()'
21:30:07  Checking max UTF16 string len: 1073741824
21:30:07  Checking max UTF16 string len: 1073741823
21:30:07  org.opentest4j.AssertionFailedError: Expected OutOfMemoryError with message prefix: UTF16 String size is
21:30:07    at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
21:30:07    at org.junit.jupiter.api.Assertions.fail(Assertions.java:135)
21:30:07    at MaxSizeUTF16String.testMaxCharArray(MaxSizeUTF16String.java:112)
21:30:07    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
21:30:07    at java.base/java.lang.reflect.Method.invoke(Method.java:586)
21:30:07    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
21:30:07    at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
21:30:07    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
21:30:07    at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
21:30:07    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
21:30:07    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
21:30:07    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
21:30:07    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
21:30:07    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
21:30:07    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
21:30:07    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
21:30:07    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
21:30:07    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
21:30:07    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
21:30:07    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
21:30:07    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
21:30:07    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
21:30:07    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
21:30:07    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
21:30:07    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
21:30:07    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
21:30:07    at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
21:30:07    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
21:30:07    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
21:30:07    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
21:30:07    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
21:30:07    at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
21:30:07    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
21:30:07    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
21:30:07    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
21:30:07    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
21:30:07    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
21:30:07    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
21:30:07    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
21:30:07    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
21:30:07    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
21:30:07    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
21:30:07    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
21:30:07    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
21:30:07    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
21:30:07    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
21:30:07    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
21:30:07    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
21:30:07    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
21:30:07    at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:142)
21:30:07    at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:95)
21:30:07    at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:61)
21:30:07    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
21:30:07    at java.base/java.lang.reflect.Method.invoke(Method.java:586)
21:30:07    at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
21:30:07    at java.base/java.lang.Thread.run(Thread.java:1587)
21:30:07  FAILED     MaxSizeUTF16String::testMaxCharArray 'testMaxCharArray()'

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_linux_Nightly_testList_1/38/ jdk_lang_1

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/11 jdk_lang_0, jdk_lang_j9_0 https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/11/ jdk_lang_1

pshipton commented 2 months ago

Test case

public class MaxChar {

private final static int MAX_UTF16_STRING_LENGTH = Integer.MAX_VALUE / 2;

    private static char[] generateCharData(int size) {
        char[] nonAscii = "\u0100".toCharArray();
        char[] arr = new char[size];
        System.arraycopy(nonAscii, 0, arr, 0, nonAscii.length); // non-latin1 at start
        return arr;
    }

public static void main(String[] args) throws Throwable {
        final char[] large_char_array = generateCharData(MAX_UTF16_STRING_LENGTH + 1);
        int[] sizes = new int[] {
                MAX_UTF16_STRING_LENGTH + 1,
                MAX_UTF16_STRING_LENGTH,
                MAX_UTF16_STRING_LENGTH - 1};
        for (int size : sizes) {
                try {
                        new String(large_char_array, 0, size);
                        if (size >= MAX_UTF16_STRING_LENGTH) {
                                System.out.println("Expected OOM for " + size);
                        }
                } catch (OutOfMemoryError e) {
                        System.out.println(e);
                }
        }
}
}

When running with -Xcomp (-Xjit:count=0) on at least aarch64 or x86, the OOM which is expected for size 1073741823 doesn't occur. It didn't fail on zlinux, some platforms (AIX, Windows) didn't run.

@hzongaro pls take a look

pshipton commented 2 months ago

This does seem related to https://github.com/eclipse-openj9/openj9/issues/19309 after all.

hzongaro commented 2 months ago

In JDK 21 and earlier, the implementation of StringUTF16.newBytesFor tested whether the length was greater than 0x3fffffff. If so, it would throw an OutOfMemoryError.

In JDK 23, the implementation of StringUTF16.newBytes, by way of String.UTF16.newBytesLength, will throw an OutOfMemoryError for length values greater than or equal to 0x3fffffff.

The IL the JIT compiler is generating is consistent with the expectation that an OutOfMemoryError will not be thrown for a length of 0x3fffffff. Before I make any further changes to the JIT compiler, is the inconsistency between the different JDK versions expected, or is one of them a bug?

hzongaro commented 2 months ago

It looks like the handling of a length of 0x3fffffff was deemed to be a bug, and the change in behaviour was introduced in JDK Level 23 in commit https://github.com/ibmruntimes/openj9-openjdk-jdk23/commit/38f0d53b1aacab53193885064fa3723e26b326f8.

I will update the JIT compiler's handling of this by falling back to the out-of-line call to StringUTF16.toBytes for lengths outside the range [0,0x3ffffffe]. That way, if the bug is ever fixed in other JDK levels, the JIT-compiled code will not show any difference in behaviour to interpreted bytecode.

hzongaro commented 2 months ago

Reopening, as I will need to port the fix to the v0.47.0-release and v0.48.0-release branches

hzongaro commented 2 months ago

Opened pull requests #20077 and #20078 to port the fix to the v0.47.0-release and v0.48.0-release branches