Closed pshipton closed 4 years ago
@ashbm5 can you pls take a look
@shanchao95 please try to reproduce
I tried to re-run the tests with same parameters on Grinder, and tried to run the test9a.java locally with jvm from https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK8_ppc64le_linux_Personal/655/OpenJ9-JDK8-ppc64le_linux-20200122-162742.tar.gz
. Did not see it occur again. Here is tests results on Grinder https://ci.adoptopenjdk.net/job/Grinder/2157/
.
On the open machines, grinders on ub16-ppcle-2 (1x), ub16-ppcle-3 (5x) didn't reproduce it, but grinders on ub16p8j92 and ub16p8j95 did. https://ci.eclipse.org/openj9/view/Test/job/Grinder/631/ https://ci.eclipse.org/openj9/view/Test/job/Grinder/632/
@jdekonin are there openssl updates not installed on the ub16p8j9x machines? If you do update the machines, pls record the original level and the updated level.
Updates to both sets of machines were going to occur on a scheduled "maintenance 1/2 day". But that date hadn't been set. ub16-ppcle-1 -> ub16-ppcle-3 are on 1.0.2g-1ubuntu4.15, the systems have 135 packages to be updated, 6 are security updates. ub16p8j91 -> ub16p8j96 are on 1.0.2g-1ubuntu4, the systems have 384 packages to be updated, 161 of which are security updates.
If you are in agreement that these can be rotated through tomorrow @rajdeep-singh and I can proceed, just advise if we should be doing an full system update to bring them all current, or if we should just be performing an openssl update.
We can do a full system update, but let's only do one system of each ppcle and p8j9 first and make sure they are still working fine afterward before doing the rest.
ub16p8j91 and ub16-ppcle-2 have been updated to the latest levels and brought back online. Note that ub16-ppcle-2 also had an issue with protobuf.
Confirmed the crash still occurs on the upgraded ub16p8j91 https://ci.eclipse.org/openj9/view/Test/job/Grinder/638
@jdekonin what is the openssl version after the update?
ub16p8j91 is at 1.0.2g-1ubuntu4.15, same as ub16-ppcle-2.
This is also occurring at Adopt. Last nights JDK14/j9/aix run on tests
com/sun/net/httpserver/Test9.java.Test9
java/net/httpclient/LargeHandshakeTest.java.LargeHandshakeTest
Stacktrace
Unexpected exit from test [exit code: 255]
Standard Output
Test9:
Standard Error
Unhandled exception
Type=Segmentation error vmState=0x00040000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000032
Handler1=09001000A0D13978 Handler2=09001000A0CE9640
R0=000000005A827999 R1=000001002D3CFAB0 R2=09001000A0AA11D0 R3=00000100280FCC20
R4=09001000A0AAFFE4 R5=03FFFFFFFFFFFFFD R6=0000000066044000 R7=000000017567C67A
R8=000000015CCC1A9F R9=00000000E67A4B7F R10=0000000057048A60 R11=00000000FFE54C48
R12=00000000998353EB R13=000001002D3DB800 R14=00000000305618A0 R15=0000000077044860
R16=0000000000000000 R17=0000000000000000 R18=0000000000000000 R19=0000000000000000
R20=0000000000000000 R21=0000000000000000 R22=0000000000000000 R23=0000000000000000
R24=0000000000000000 R25=0000000000000000 R26=0000000000000000 R27=0000000000000000
R28=0000000000000000 R29=0000000000000000 R30=0000000000000000 R31=0000000000000000
IAR=0900000006F2DE5C LR=0900000006F2DC74 MSR=A00000000200D032 CTR=03FFFFFFFFFFFD62
CR=24402244000100F4 FPSCR=8200200000000000 XER=000100F482002000
FPR0 4040000000000000 (f: 0.000000, d: 3.200000e+01)
FPR1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR2 3fec4ed454000000 (f: 1409286144.000000, d: 8.846227e-01)
FPR3 433000000005c0c0 (f: 377024.000000, d: 4.503600e+15)
FPR4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR5 433000000000a9ec (f: 43500.000000, d: 4.503600e+15)
FPR6 4117030000000000 (f: 0.000000, d: 3.770240e+05)
FPR7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR8 bea0145980b6318c (f: 2159424000.000000, d: -4.792062e-07)
FPR9 bf56aefc8fb3c300 (f: 2410922752.000000, d: -1.384493e-03)
FPR10 402c62a66ab53432 (f: 1790260224.000000, d: 1.419268e+01)
FPR11 3fd55555555450ef (f: 1431589120.000000, d: 3.333333e-01)
FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
FPR13 4042800000000000 (f: 0.000000, d: 3.700000e+01)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/libcrypto.a
Module_base_address=0900000006F07180
Target=2_90_20200312_33 (AIX 7.1)
CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
(0x0900000006F2DB80 [libcrypto.a+0x26a00])
(0x0900000006F4563C [libcrypto.a+0x3e4bc])
(0x0900000006F3C24C [libcrypto.a+0x350cc])
(0x0900000006F44380 [libcrypto.a+0x3d200])
(0x0900000006F43AF4 [libcrypto.a+0x3c974])
(0x0900000006F43290 [libcrypto.a+0x3c110])
(0x0900000006F47FEC [libcrypto.a+0x40e6c])
(0x0900000006F47D50 [libcrypto.a+0x40bd0])
(0x0900000006F472A8 [libcrypto.a+0x40128])
(0x0900000006F46354 [libcrypto.a+0x3f1d4])
(0x0900000006F8C1CC [libcrypto.a+0x8504c])
(0x0900000006F8D2FC [libcrypto.a+0x8617c])
(0x0900000006F465D8 [libcrypto.a+0x3f458])
Java_jdk_crypto_jniprovider_NativeCrypto_RSADP+0xe4 (0x090000000FD8ACC8 [libjncrypto.so+0x5cc8])
(0x0900000007BB66BC [libj9vm29.so+0x14f6bc])
ffi_call+0x98 (0x0900000007BB59BC [libj9vm29.so+0x14e9bc])
(0x0900000007BC0C24 [libj9vm29.so+0x159c24])
(0x0900000007ADEF7C [libj9vm29.so+0x77f7c])
runJavaThread+0x1d8 (0x0900000007AC9FBC [libj9vm29.so+0x62fbc])
javaProtectedThreadProc+0x11c (0x0900000007A699E0 [libj9vm29.so+0x29e0])
omrsig_protect+0x4a4 (0x0900000007D61DC8 [libj9prt29.so+0x58dc8])
javaThreadProc+0x68 (0x0900000007A6984C [libj9vm29.so+0x284c])
thread_wrapper+0x558 (0x0900000007CEA63C [libj9thr29.so+0x563c])
_pthread_body+0xf0 (0x0900000000570E14 [libpthreads.a+0x3e14])
---------------------------------------
EDIT: And also JDK8/J9/aix
Occurrences in jdk11 runs: https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.openjdk_ppc64le_linux_Nightly/3 ub16p8j95 com/sun/net/httpserver/Test9.java.Test9 com/sun/net/httpserver/Test9a.java.Test9a
Occurred in a JDK14-j9-aix run last night https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk14_j9_sanity.openjdk_ppc64_aix/31/
Output files: https://ibm.box.com/s/kkvbuy03z3za24ocugw8owhppynkym7w
This is an intermittent problem tracked in a Milestone. I'm going to dissolve the "Blocking sanity.openjdk" project since the less intermittent problems were resolved by reconfiguring sanity.openjdk to tier1.
The fix https://github.com/eclipse/openj9/issues/8373#issuecomment-603814836 is merged, but not yet into the 0.20.0 branch.
The updates to the 0.20.0 branches are complete.
https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.openjdk_ppc64le_linux_Personal/1 jdk_net_0 TEST: com/sun/net/httpserver/Test9a.java