JCL_TEST_Java-Security Trace/BPT trap(coredump)

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_ppc64_aix_Nightly_testList_0/697 - p8-java1-ibm07 JCL_TEST_Java-Security_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_sanity.functional_ppc64_aix_Nightly_testList_0/697/functional_test_output.tar.gz

01:20:26  [DigestLoop] [INFO] Using Provider SUN
01:20:26  [DigestLoop] [INFO] Payload size: 16 bytes
01:25:47  /bin/sh[7]: 9633960 Trace/BPT trap(coredump)

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_ppc64_aix_Nightly_testList_1/657 https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_ppc64_aix_Nightly_testList_1/160/

Change https://github.com/eclipse-openj9/openj9/pull/19095

[pshipton]

@dmitripivkine can you pls take the first look.

[dmitripivkine]

Relevant c-stack:

.....
4 IPRA.$printErrors(??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, internal error: assertion failed at line 3914 in file frame.c
??, internal error: assertion failed at line 3914 in file frame.c
??, internal error: assertion failed at line 3914 in file frame.c
??, internal error: assertion failed at line 3914 in file frame.c
??, internal error: assertion failed at line 3914 in file frame.c
??, internal error: assertion failed at line 3914 in file frame.c
??), line 313 in "NativeCrypto.c"
---------
$stkp:0x00000100257a85b0   $r14:0x00000000303dac98   $r15:0x00000000301af400
 $r16:0x0000010021ecb9d8   $r17:0xffffffffffffffff   $r18:0x0000000000000000
 $r19:0x09001000a06f1288   $r20:0x0000000000000000   $r21:0x09001000a133f290
 $r22:0x0000000000000030   $r23:0x0000000000000000   $r24:0x09001000a132dc00
 $r25:0x00000007fd392e08   $r26:0x00000007fd392d60   $r27:0x00000006202d6b30
 $r28:0x00000000000ca37d   $r29:0x00000000302f6500   $r30:0x0000000000000020
 $r31:0x0000000000000004
 $iar:0x0900000006860c1c  $link:0x0000010011f59e30
        [unset $noflregs to view floating point registers]
        [unset $novregs to view vector registers]
        [unset $novsregs to view vector scalar registers]

5 Java_jdk_crypto_jniprovider_NativeCrypto_DigestCreateContext(??, ??, ??, ??), line 887 in "NativeCrypto.c"
---------
$stkp:0x00000100257a8660   $r14:0x00000000303dac98   $r15:0x00000000301af400
 $r16:0x0000010021ecb9d8   $r17:0xffffffffffffffff   $r18:0x0000000000000000
 $r19:0x09001000a06f1288   $r20:0x0000000000000000   $r21:0x09001000a133f290
 $r22:0x0000000000000030   $r23:0x0000000000000000   $r24:0x09001000a132dc00
 $r25:0x00000007fd392e08   $r26:0x00000007fd392d60   $r27:0x00000006202d6b30
 $r28:0x00000000000ca37d   $r29:0x00000000302f6500   $r30:0x0000000000000020
 $r31:0x0000000000000004
 $iar:0x0000010011f59e30  $link:0x09000000043d6f44
        [unset $noflregs to view floating point registers]
        [unset $novregs to view vector registers]
        [unset $novsregs to view vector scalar registers]
.....

[pshipton]

@jasonkatonica pls take a look

[pshipton]

Excluding the test via https://github.com/eclipse-openj9/openj9/pull/19117

[KostasTsiounis]

According to the provided c stack trace, the failure occurs in Java_jdk_crypto_jniprovider_NativeCrypto_DigestCreateContext and more specifically in the call to printErrors() after trying to do tx = (*OSSL_MD_CTX_new)() and getting NULL. Due to the rapid creation of digests, it could be running out of memory. The return value of this call, however, is checked and it moves on to printing the error. This is done by calling printErrors().

Within printErrors(), the failure occurs when calling fprintf(stderr, "An OpenSSL error occurred\n"). More specifically, we seem to be getting this error internal error: assertion failed at line 3914 in file frame.c, which pertains to a separate file, namely frame.c. There is nothing OpenSSL or context related in this call, so one would have to assume that there might be an issue with fprintf or stderr on AIX, or more specifically the particular frame.c file.

[pshipton]

I've opened an issue to update openssl on the OpenJ9 AIX machines, since it's 1.0.2. infrastructure/issues/9451

These are the tests that have been excluded.

<class name="org.openj9.test.java.security.Test_MHLoaderConstraints"/>
<class name="org.openj9.test.java.security.DigestLoop"/>
<class name="org.openj9.test.java.security.OpenSSLInvalidCBCKeySizeTest"/>
<class name="org.openj9.test.java.security.OpenSSLInvalidGCMKeySizeTest"/>

The failure is possibly related to the test changes that occurred just before https://github.com/eclipse-openj9/openj9/pull/19095