OpenJDK CompactString/MaxSizeUTF16String testMaxCharArray() expects OOM but gets NegativeArraySizeException with JIT

[pshipton]

jdk23 testing on amac. This also occurs with jdk-21.0.2+13 or jdk-17.0.10+7 on xLinux. It does not occur with jdk-17.0.9+9 so likely there is a change between 0.41 and 0.43 causing this.

17:09:02  openjdk version "23-internal" 2024-09-17
17:09:02  OpenJDK Runtime Environment (build 23-internal-adhoc.****.BuildJDKnextaarch64macPersonal)
17:09:02  Eclipse OpenJ9 VM (build master-b2db2045aa4, JRE 23 Mac OS X aarch64-64-Bit 20240411_113 (JIT enabled, AOT enabled)
17:09:02  OpenJ9   - b2db2045aa4
17:09:02  OMR      - 0e07ad19c1f
17:09:02  JCL      - 7ac6a9b5b39 based on jdk-23+17)

From a grinder on java/lang/String/CompactString/MaxSizeUTF16String.java

16:22:49  STARTED    MaxSizeUTF16String::testMaxCharArray 'testMaxCharArray()'
16:22:49  Checking max UTF16 string len: 1073741824
16:22:49  java.lang.NegativeArraySizeException: -2147483648
16:22:49    at java.base/java.lang.String.<init>(String.java:5655)
16:22:49    at java.base/java.lang.String.<init>(String.java:489)
16:22:49    at MaxSizeUTF16String.testMaxCharArray(MaxSizeUTF16String.java:110)

This can be duplicated with the following, when running with -Xjit:count=0. When running without or with -Xint, the expected OOM error occurs (shown below).

public class MaxChar {

private final static int MAX_UTF16_STRING_LENGTH = Integer.MAX_VALUE / 2;

    private static char[] generateCharData(int size) {
        char[] nonAscii = "\u0100".toCharArray();
        char[] arr = new char[size];
        System.arraycopy(nonAscii, 0, arr, 0, nonAscii.length); // non-latin1 at start
        return arr;
    }

public static void main(String[] args) throws Throwable {
        final char[] large_char_array = generateCharData(MAX_UTF16_STRING_LENGTH + 1);
        new String(large_char_array, 0, MAX_UTF16_STRING_LENGTH + 1);
}
}

Exception in thread "main" java.lang.OutOfMemoryError: UTF16 String size is 1073741824, should be less than 1073741823
        at java.base/java.lang.StringUTF16.newBytesLength(StringUTF16.java:60)
        at java.base/java.lang.StringUTF16.newBytesFor(StringUTF16.java:50)
        at java.base/java.lang.StringUTF16.toBytes(StringUTF16.java:169)
        at java.base/java.lang.StringUTF16.compress(StringUTF16.java:218)
        at java.base/java.lang.String.<init>(String.java:5646)
        at java.base/java.lang.String.<init>(String.java:489)
        at MaxChar.main(MaxChar.java:14)

[pshipton]

@hzongaro fyi

[hzongaro]

@pshipton, I don't seem to be able to reproduce this problem with the simplified test running jdk-17.0.10+7 on xLinux. Are you able to point me to the grinder run?

[pshipton]

Here is a grinder that shows it https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/3473

I have no problem reproducing it on my laptop, using the 17.0.10 (0.43) release. jdk-17.0.10+7/bin/java -Xjit:count=0 MaxChar

Also if I run jdk-23/bin/java MaxChar.java or jdk-23/bin/java -Xjit:count=0 it reproduces, but jdk-17.0.10+7/bin/java -Xjit:count=0 MaxChar.java does not, so it's a bit weird.

[hzongaro]

It seems that I needed to adjust the heap size in order to observe the NegativeArraySizeException:

# ./jdk-17.0.10+7/bin/java -Xjit:count=0 -Xmx4G MaxChar
Exception in thread "main" java.lang.NegativeArraySizeException: -2147483648
        at java.base/java.lang.String.<init>(String.java:5278)
        at java.base/java.lang.String.<init>(String.java:478)
        at MaxChar.main(MaxChar.java:14)

[hzongaro]

I was able to reproduce this with 17.0.5+8, so it's not a recent regression:

# ./jdk-17.0.5+8/bin/java -version
openjdk version "17.0.5" 2022-10-18
IBM Semeru Runtime Open Edition 17.0.5.0 (build 17.0.5+8)
Eclipse OpenJ9 VM 17.0.5.0 (build openj9-0.35.0, JRE 17 Linux amd64-64-Bit Compressed References 20221018_325 (JIT enabled, AOT enabled)
OpenJ9   - e04a7f6c1
OMR      - 85a21674f
JCL      - 32d2c409a33 based on jdk-17.0.5+8)
# ./jdk-17.0.5+8/bin/java -Xjit:count=0 -Xmx4G MaxChar
Exception in thread "main" java.lang.NegativeArraySizeException: -2147483648
        at java.base/java.lang.String.<init>(String.java:5276)
        at java.base/java.lang.String.<init>(String.java:478)
        at MaxChar.main(MaxChar.java:14)

[hzongaro]

Or even at least as far back as 17.0.2+8, if I'm selective about what is compiled:

# ./jdk-17.0.2+8/bin/java -version
openjdk version "17.0.2" 2022-01-18
IBM Semeru Runtime Open Edition 17.0.2.0 (build 17.0.2+8)
Eclipse OpenJ9 VM 17.0.2.0 (build openj9-0.30.0, JRE 17 Linux amd64-64-Bit Compressed References 20220128_115 (JIT enabled, AOT enabled)
OpenJ9   - 9dccbe076
OMR      - dac962a28
JCL      - 64cd399ca28 based on jdk-17.0.2+8)
root@ubuntu-1-hz1:/home/henry/docker-hostdir/defects/issue19309
# ./jdk-17.0.2+8/bin/java -Xjit:count=0,limit={*String.\<init\>\(\[CIILjava/lang/Void\;*},optLevel=noOpt -Xmx4G MaxChar
Exception in thread "main" java.lang.NegativeArraySizeException: -2147483648
        at java.base/java.lang.String.<init>(String.java:5264)
        at java.base/java.lang.String.<init>(String.java:478)
        at MaxChar.main(MaxChar.java:14)

[hzongaro]

It looks like Recognized Call Transformer transforms this IL:

n44n      treetop                                                                             [0x7f354af964c0] bci=[-1,67,5264] rc=0 vc=0 vn=- li=- udi=- nc=1
n43n        acall  java/lang/StringUTF16.toBytes([CII)[B[#369  final static Method] [flags 0x20500 0x0 ] (sharedMemory )  [0x7f354af96470] bci=[-1,67,5264] rc=2 vc=0 vn=- li=- udi=- nc=3 flg=0x20
n40n          aload  value<parm 1 [C>[#356  Parm] [flags 0x40000107 0x0 ]                     [0x7f354af96380] bci=[-1,64,5264] rc=1 vc=0 vn=- li=- udi=- nc=0
n41n          iload  off<parm 2 I>[#357  Parm] [flags 0x40000103 0x0 ]                        [0x7f354af963d0] bci=[-1,65,5264] rc=1 vc=0 vn=- li=- udi=- nc=0
n42n          iload  len<parm 3 I>[#358  Parm] [flags 0x40000103 0x0 ]                        [0x7f354af96420] bci=[-1,66,5264] rc=1 vc=0 vn=- li=- udi=- nc=0

into this:

49n      treetop                                                                             [0x7f354af96650] bci=[-1,64,5264] rc=0 vc=0 vn=- li=- udi=- nc=1
n40n        aload  value<parm 1 [C>[#356  Parm] [flags 0x40000107 0x0 ]                       [0x7f354af96380] bci=[-1,64,5264] rc=2 vc=0 vn=- li=- udi=- nc=0
n50n      treetop                                                                             [0x7f354af966a0] bci=[-1,65,5264] rc=0 vc=0 vn=- li=- udi=- nc=1
n41n        iload  off<parm 2 I>[#357  Parm] [flags 0x40000103 0x0 ]                          [0x7f354af963d0] bci=[-1,65,5264] rc=2 vc=0 vn=- li=- udi=- nc=0
n51n      treetop                                                                             [0x7f354af966f0] bci=[-1,66,5264] rc=0 vc=0 vn=- li=- udi=- nc=1
n42n        iload  len<parm 3 I>[#358  Parm] [flags 0x40000103 0x0 ]                          [0x7f354af96420] bci=[-1,66,5264] rc=3 vc=0 vn=- li=- udi=- nc=0
n44n      treetop                                                                             [0x7f354af964c0] bci=[-1,67,5264] rc=0 vc=0 vn=- li=- udi=- nc=1
n43n        newarray  jitNewArray[#102  helper Method] [flags 0x400 0x0 ] (X!=0 skipZeroInit sharedMemory )  [0x7f354af96470] bci=[-1,67,5264] rc=3 vc=0 vn=- li=- udi=- nc=2 flg=0x8004
n54n          ishl                                                                            [0x7f354b0310a0] bci=[-1,66,5264] rc=1 vc=0 vn=- li=- udi=- nc=2
n42n            ==>iload
n53n            iconst 1                                                                      [0x7f354b031050] bci=[-1,0,5249] rc=1 vc=0 vn=- li=- udi=- nc=0
n52n          iconst 8   ; array type is byte                                                 [0x7f354b031000] bci=[-1,0,5249] rc=1 vc=0 vn=- li=- udi=- nc=0
n57n      treetop                                                                             [0x7f354b031190] bci=[-1,67,5264] rc=0 vc=0 vn=- li=- udi=- nc=1
n55n        call  java/lang/String.decompressedArrayCopy([CI[BII)V[#375  final static Method] [flags 0x20500 0x0 ] ()  [0x7f354b0310f0] bci=[-1,67,5264] rc=1 vc=0 vn=- li=- udi=- nc=5 flg=0x20
n40n          ==>aload
n41n          ==>iload
n43n          ==>newarray
n56n          iconst 0                                                                        [0x7f354b031140] bci=[-1,0,5249] rc=1 vc=0 vn=- li=- udi=- nc=0
n42n          ==>iload

The length value of 2^31 is left shifted by 1 at instruction n54n, resulting in a negative array size for the jitNewArray call. The implementation of StringUTF16.toBytes(char[],int,int), however, has checks on the len argument that decide whether to throw NegativeArraySizeException or OutOfMemoryError before attempting to allocate the byte array. The length itself is not negative, so the implementation of StringUTF16.toBytes would instead end up throwing an OutOfMemoryError. The Recognized Call Transformer would need to include similar tests to replicate the expected behaviour.

[pshipton]

Excluding the test via https://github.com/adoptium/aqa-tests/pull/5251 for now.

[vij-singh]

@hzongaro Is this specifically related to JDK23 (or reproducible only on JDK23)?

[hzongaro]

Is this specifically related to JDK23 (or reproducible only on JDK23)?

No - it's a long-standing problem. I was able to reproduce it as far back as JDK 17.0.2+8 by adjusting the heap size, and by being selective about what was compiled.

[vij-singh]

@hzongaro Ok thanks for the update @pshipton Should we move this to the 0.48 Milestone instead?

[pshipton]

Ideally it should be fixed in jdk23 so we can unexclude the test, but if it needs to move it can.

[pshipton]

Reopen as the fix needs to be triple delivered (0.47, 0.48), and the test unexcluded.

[pshipton]

Created https://github.com/adoptium/aqa-tests/pull/5513 to unexclude.

[pshipton]

There is still a failure after unexcluding the test, but I don't think it's the same cause, I've opened a new issue. https://github.com/eclipse-openj9/openj9/issues/20021