search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

cmdLineTester_SCHelperCompatibilityTests_unix_1 failed in OMR-Acceptance on OSX #5153

Open fjeremic opened 5 years ago

fjeremic commented 5 years ago

Failure link

https://ci.eclipse.org/openj9/job/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/248/tapResults/

Optional info

Failure output

cmdLineTester_SCHelperCompatibilityTests_unix_1

[2019-03-19T14:10:58.497Z] Testing: HelperCompatibilityTest 3
[2019-03-19T14:10:58.497Z] Test start time: 2019/03/19 10:10:57 Eastern Standard Time
[2019-03-19T14:10:58.497Z] Running command: "/Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdkbinary/j2sdk-image/bin/java" -Xcompressedrefs -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=HelperCompatibilityTests -Xbootclasspath/a:./Utils:. URLClassPathHelperURLHelperStaleEntryCompatibilityTest -testfile "/Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/functional/cmdLineTests/shareClassTests/SCHelperCompatibilityTests/props_unix/Test3.props" -javacdir /Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdkbinary/j2sdk-image/bin/../bin
[2019-03-19T14:10:58.497Z] Time spent starting: 5 milliseconds
[2019-03-19T14:11:07.739Z] Time spent executing: 9288 milliseconds
[2019-03-19T14:11:07.739Z] Test result: FAILED
[2019-03-19T14:11:07.739Z]  [OUT] 
[2019-03-19T14:11:07.739Z]  [OUT] ** Running: /bin/sh ./batchfiles/StaleClassPathTest1.sh /Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdkbinary/j2sdk-image/bin/../bin
[2019-03-19T14:11:07.739Z]  [ERR] Unhandled exception
[2019-03-19T14:11:07.739Z]  [ERR] Type=Segmentation error vmState=0x0005ffff
[2019-03-19T14:11:07.739Z]  [ERR] J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2019-03-19T14:11:07.739Z]  [ERR] Handler1=000000000769ABF0 Handler2=000000000788FB20 InaccessibleAddress=0000000000000040
[2019-03-19T14:11:07.739Z]  [ERR] RDI=0000000008BDFADB RSI=00007000002593B4 RAX=0000000000000000 RBX=0000000008BDFADB
[2019-03-19T14:11:07.739Z]  [ERR] RCX=000000000000004A RDX=00007F87E1D92760 R8=00000000FFFFFFF3 R9=000000000000000D
[2019-03-19T14:11:07.739Z]  [ERR] R10=00007F87E1D929C8 R11=0000000000000040 R12=00000000FFFFFFF3 R13=000000001FD539B0
[2019-03-19T14:11:07.739Z]  [ERR] R14=000000000000004A R15=0000000000000054
[2019-03-19T14:11:07.739Z]  [ERR] RIP=00007FFF8638D7D4 GS=0000 FS=0000 RSP=0000700000259380
[2019-03-19T14:11:07.739Z]  [ERR] RFlags=0000000000010202 CS=002B RBP=00007000002593A0 ERR=0000004000000004
[2019-03-19T14:11:07.739Z]  [ERR] TRAPNO=000000040000000E CPU=0040000000040000 FAULTVADDR=0000000000000040
[2019-03-19T14:11:07.739Z]  [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM1 00000000ff000000 (f: 4278190080.000000, d: 2.113707e-314)
[2019-03-19T14:11:07.739Z]  [ERR] XMM2 00000000ff000000 (f: 4278190080.000000, d: 2.113707e-314)
[2019-03-19T14:11:07.739Z]  [ERR] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
[2019-03-19T14:11:07.739Z]  [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2019-03-19T14:11:07.739Z]  [ERR] Module=/usr/lib/system/libsystem_c.dylib
[2019-03-19T14:11:07.739Z]  [ERR] Module_base_address=00007FFF8632E000 Symbol=__findenv
[2019-03-19T14:11:07.739Z]  [ERR] Symbol_address=00007FFF8638D780
[2019-03-19T14:11:07.739Z]  [ERR] 
[2019-03-19T14:11:07.739Z]  [ERR] Method_being_compiled=jdk/internal/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
[2019-03-19T14:11:07.739Z]  [ERR] Target=2_90_20190319_308 (Mac OS X 10.11.6)
[2019-03-19T14:11:07.739Z]  [ERR] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
[2019-03-19T14:11:07.739Z]  [ERR] ----------- Stack Backtrace -----------
[2019-03-19T14:11:07.739Z]  [ERR] ---------------------------------------
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP039I Processing dump event "gpf", detail "" at 2019/03/19 10:10:59 - please wait.
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15530016248530/cmdLineTester_SCHelperCompatibilityTests_unix_1/core.20190319.101059.86438.0001.dmp' in response to an event
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP010I System dump written to /Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15530016248530/cmdLineTester_SCHelperCompatibilityTests_unix_1/core.20190319.101059.86438.0001.dmp
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP032I JVM requested Java dump using '/Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15530016248530/cmdLineTester_SCHelperCompatibilityTests_unix_1/javacore.20190319.101059.86438.0002.txt' in response to an event
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP010I Java dump written to /Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15530016248530/cmdLineTester_SCHelperCompatibilityTests_unix_1/javacore.20190319.101059.86438.0002.txt
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP032I JVM requested Snap dump using '/Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15530016248530/cmdLineTester_SCHelperCompatibilityTests_unix_1/Snap.20190319.101059.86438.0003.trc' in response to an event
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP010I Snap dump written to /Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15530016248530/cmdLineTester_SCHelperCompatibilityTests_unix_1/Snap.20190319.101059.86438.0003.trc
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP007I JVM Requesting JIT dump using '/Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15530016248530/cmdLineTester_SCHelperCompatibilityTests_unix_1/jitdump.20190319.101059.86438.0004.dmp'
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP010I JIT dump written to /Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15530016248530/cmdLineTester_SCHelperCompatibilityTests_unix_1/jitdump.20190319.101059.86438.0004.dmp
[2019-03-19T14:11:07.739Z]  [ERR] JVMDUMP013I Processed dump event "gpf", detail "".
[2019-03-19T14:11:07.739Z] >> Success condition was not found: [Output match: TEST PASSED]
[2019-03-19T14:11:07.739Z] >> Failure condition was not found: [Output match: Error:]
[2019-03-19T14:11:07.739Z] >> Failure condition was found: [Output match: Unhandled Exception]
[2019-03-19T14:11:07.739Z] >> Failure condition was not found: [Output match: Exception:]
fjeremic commented 5 years ago

@andrewcraik @cathyzhyi x86 OSX failure in the JIT itself. It seems intermittent as we've launched several builds against the OMR and OpenJ9 changes in #5042 and they have been passing. I don't have an OSX machine handy I can hop on to get a backtrace but the core dump is still there.

I've saved it to /scratch/5153 in case it gets deleted. I've also backed it up locally shall we need it. You can ping me to grab the dump.

0xdaryl commented 5 years ago

Possible dup of #5046 (different test though).

fjeremic commented 5 years ago

Possible dup of #5046 (different test though).

Symptoms look nearly identical with the Symbol=__findenv. I suspect it's very likely the duplicate.

mstoodle commented 5 years ago

seems most of the root cause investigation for this one has happened via #5061 . Outlook there seems to depend on the very recently created PR #5340 (which may or may not be causing this outcome).

pshipton commented 5 years ago

Another failure https://ci.eclipse.org/openj9/job/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/307

Testing: HelperCompatibilityTest 3
Test start time: 2019/04/08 11:56:17 Eastern Standard Time
Running command: "/Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdkbinary/j2sdk-image/bin/java" -Xcompressedrefs -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xshareclasses:name=HelperCompatibilityTests -Xbootclasspath/a:./Utils:. URLClassPathHelperURLHelperStaleEntryCompatibilityTest -testfile "/Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/functional/cmdLineTests/shareClassTests/SCHelperCompatibilityTests/props_unix/Test3.props" -javacdir /Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdkbinary/j2sdk-image/bin/../bin
Time spent starting: 4 milliseconds
Time spent executing: 10719 milliseconds
Test result: FAILED
 [OUT] 
 [OUT] ** Running: /bin/sh ./batchfiles/StaleClassPathTest1.sh /Users/jenkins/workspace/Test-sanity.functional-JDK11-osx_x86-64_cmprssptrs/openjdkbinary/j2sdk-image/bin/../bin
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x0005ffff
 [ERR] J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [ERR] Handler1=000000000FC9AF20 Handler2=000000000FE8FAD0 InaccessibleAddress=0000000000010000
 [ERR] RDI=00000000111DFB80 RSI=00007000002593B4 RAX=0000000000000000 RBX=00000000111DFB80
 [ERR] RCX=000000000000004A RDX=00007FF3396A2700 R8=00000000FFFFFFF3 R9=000000000000000D
 [ERR] R10=00007FF3396A2988 R11=0000000000010000 R12=00000000FFFFFFF3 R13=000000002832CC50
 [ERR] R14=000000000000004A R15=0000000000000054
 [ERR] RIP=00007FFF8638D7D4 GS=0000 FS=0000 RSP=0000700000259380
 [ERR] RFlags=0000000000010202 CS=002B RBP=00007000002593A0 ERR=0001000000000004
 [ERR] TRAPNO=000000040000000E CPU=0000000000040000 FAULTVADDR=0000000000010000
 [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
 [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/usr/lib/system/libsystem_c.dylib
 [ERR] Module_base_address=00007FFF8632E000 Symbol=__findenv
 [ERR] Symbol_address=00007FFF8638D780
 [ERR] 
 [ERR] Method_being_compiled=java/lang/Class.getMethodImpl(Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/String;)Ljava/lang/reflect/Method;
 [ERR] Target=2_90_20190408_375 (Mac OS X 10.11.6)
 [ERR] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] ---------------------------------------
pshipton commented 5 years ago

We haven't been able to track this down yet, but it's not a regression. Moving to 0.15

pshipton commented 5 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly/42

00:30:14      [javac] Compiling 21 source files to /Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly/openjdk-tests/functional/cmdLineTests/shareClassTests/TokenHelperTests/bin/FoodJar
00:30:16      [javac] Unhandled exception
00:30:16      [javac] Type=Segmentation error vmState=0x0005ffff
00:30:16      [javac] J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
00:30:16      [javac] Handler1=00000000054F5B80 Handler2=00000000056E9AD0 InaccessibleAddress=0000000000000040
00:30:16      [javac] RDI=0000000006A51F41 RSI=00007000002593A4 RAX=0000000000000000 RBX=0000000006A51F41
00:30:16      [javac] RCX=000000000000004A RDX=00007FDFFBD1C640 R8=00000000FFFFFFF3 R9=000000000000000D
00:30:16      [javac] R10=00007FDFFBD1C8C8 R11=0000000000000040 R12=00000000FFFFFFF3 R13=000000001DC24210
00:30:16      [javac] R14=000000000000004A R15=0000000000000054
00:30:16      [javac] RIP=00007FFF9620C7D4 GS=0000 FS=0000 RSP=0000700000259370
00:30:16      [javac] RFlags=0000000000010202 CS=002B RBP=0000700000259390 ERR=0000004000000004
00:30:16      [javac] TRAPNO=000000040000000E CPU=0040000000040000 FAULTVADDR=0000000000000040
00:30:16      [javac] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM2 0000025800000000 (f: 0.000000, d: 1.273197e-311)
00:30:16      [javac] XMM3 6e616c2f6176616a (f: 1635148160.000000, d: 5.038250e+223)
00:30:16      [javac] XMM4 6e69727453746361 (f: 1400136576.000000, d: 7.358764e+223)
00:30:16      [javac] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:30:16      [javac] Module=/usr/lib/system/libsystem_c.dylib
00:30:16      [javac] Module_base_address=00007FFF961AD000 Symbol=__findenv
00:30:16      [javac] Symbol_address=00007FFF9620C780
00:30:16      [javac] 
00:30:16      [javac] Method_being_compiled=java/lang/AbstractStringBuilder.isLatin1()Z
00:30:16      [javac] Target=2_90_20190525_44 (Mac OS X 10.11.6)
00:30:16      [javac] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
00:30:16      [javac] ----------- Stack Backtrace -----------
00:30:16      [javac] ---------------------------------------
pshipton commented 5 years ago

https://ci.eclipse.org/openj9/job/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK/18 J9vmTest_5

+++ j9vm.test.threadname.ControlCharacterThreadNameTest: +++
command: /Users/jenkins/workspace/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK/openjdkbinary/j2sdk-image/bin/java -Xcompressedrefs -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xdump -Xms64m -Xmx64m    -classpath /Users/jenkins/workspace/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/functional/VM_Test/VM_Test.jar:/Users/jenkins/workspace/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/TestConfig/lib/asm-all.jar  j9vm.test.threadname.ControlCharacterThreadNameTest 

Unhandled exception
Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=0000000005E9BB80 Handler2=000000000608FAD0
RDI=00000000073E09A5 RSI=00007000002593A4 RAX=0000000000000000 RBX=00000000073E09A5
RCX=000000000000004A RDX=00007F9598D279A0 R8=00000000FFFFFFF3 R9=000000000000000D
R10=00007F9598D27C38 R11=300007F9598D26A5 R12=00000000FFFFFFF3 R13=000000001BEE53B0
R14=000000000000004A R15=0000000000000054
RIP=00007FFF9620C7D4 GS=0000 FS=0000 RSP=0000700000259370
RFlags=0000000000010202 CS=002B RBP=0000700000259390 ERR=073BD69400000000
TRAPNO=000000000000000D CPU=D694000000000000 FAULTVADDR=00000000073BD694
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM2 0000025800000000 (f: 0.000000, d: 1.273197e-311)
XMM3 6e616c2f6176616a (f: 1635148160.000000, d: 5.038250e+223)
XMM4 616c432f676e616c (f: 1735287168.000000, d: 1.986723e+161)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF961AD000 Symbol=__findenv
Symbol_address=00007FFF9620C780

Method_being_compiled=java/lang/ClassLoader.isVerboseImpl()Z
Target=2_90_20190529_22 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton commented 5 years ago

See also the failures in https://github.com/eclipse/openj9/issues/6362

pshipton commented 5 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk12_j9_sanity.functional_x86-64_mac_Nightly/92 cmdLineTester_jvmtitests_8

Testing: gste001
Test start time: 2019/07/17 06:26:36 Eastern Standard Time
Running command: "/Users/jenkins/workspace/Test_openjdk12_j9_sanity.functional_x86-64_mac_Nightly/openjdkbinary/j2sdk-image/bin/java" -Xcompressedrefs -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xdump    -Xjit:count=0 -agentlib:jvmtitest=test:gste001 -cp "/Users/jenkins/workspace/Test_openjdk12_j9_sanity.functional_x86-64_mac_Nightly/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/functional/cmdLineTests/jvmtitests/jvmtitest.jar" com.ibm.jvmti.tests.util.TestRunner
Time spent starting: 6 milliseconds
Time spent executing: 9664 milliseconds
Test result: FAILED
Output from test:
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x0005ffff
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [ERR] Handler1=000000000909C0A0 Handler2=00000000092AA980 InaccessibleAddress=0000000000000040
 [ERR] RDI=000000000A612CDD RSI=00007000002593A4 RAX=0000000000000000 RBX=000000000A612CDD
 [ERR] RCX=000000000000004A RDX=00007FCB6AD08C50 R8=00000000FFFFFFF3 R9=000000000000000D
 [ERR] R10=00007FCB6AD08EE8 R11=0000000000000040 R12=00000000FFFFFFF3 R13=00000000236A5DD0
 [ERR] R14=000000000000004A R15=0000000000000054
 [ERR] RIP=00007FFF9CF787D4 GS=0000 FS=0000 RSP=0000700000259370
 [ERR] RFlags=0000000000010202 CS=002B RBP=0000700000259390 ERR=0000004000000004
 [ERR] TRAPNO=000000040000000E CPU=0040000000040000 FAULTVADDR=0000000000000040
 [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM1 000000057fffffff (f: 2147483648.000000, d: 1.167098e-313)
 [ERR] XMM2 000000000a5f2f2e (f: 174010160.000000, d: 8.597244e-316)
 [ERR] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
 [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/usr/lib/system/libsystem_c.dylib
 [ERR] Module_base_address=00007FFF9CF19000 Symbol=__findenv
 [ERR] Symbol_address=00007FFF9CF78780
 [ERR] 
 [ERR] Method_being_compiled=com/ibm/oti/vm/VM.markCurrentThreadAsSystemImpl()I
 [ERR] Target=2_90_20190716_96 (Mac OS X 10.11.6)
 [ERR] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] ---------------------------------------
pshipton commented 5 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_mac_OMR/194

Testing: Run !findallcallsites
Test start time: 2019/07/20 15:00:11 Eastern Standard Time
Running command: /Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_OMR/openjdkbinary/j2sdk-image/bin/jdmpview -core j9core.dmp
Time spent starting: 3 milliseconds
Time spent executing: 3619 milliseconds
Test result: FAILED
Output from test:
 [OUT] DTFJView version 4.29.5, using DTFJ version 1.12.29003
 [OUT] Loading image from DTFJ...
 [OUT] 
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x0005ffff
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
 [ERR] Handler1=000000000C8A2910 Handler2=000000000BF6DD90
 [ERR] RDI=000000000DD4C16F RSI=0000700000259384 RAX=0000000000000000 RBX=000000000DD4C16F
 [ERR] RCX=0000000000000000 RDX=00007F9CD1F00510 R8=00000000FFFFFFF3 R9=000000000000000D
 [ERR] R10=00007F9CD1F00510 R11=7000000000000000 R12=000000000BF04BE8 R13=000000002BB59740
 [ERR] R14=0000000000000000 R15=0000000000000000
 [ERR] RIP=00007FFF9CF787D4 GS=0000 FS=0000 RSP=0000700000259350
 [ERR] RFlags=0000000000010202 CS=002B RBP=0000700000259370 ERR=1BB7500000000000
 [ERR] TRAPNO=000000000000000D CPU=5000000000000000 FAULTVADDR=000000001BB75000
 [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM2 00000028000001f4 (f: 500.000000, d: 8.487983e-313)
 [ERR] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
 [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/usr/lib/system/libsystem_c.dylib
 [ERR] Module_base_address=00007FFF9CF19000 Symbol=__findenv
 [ERR] Symbol_address=00007FFF9CF78780
 [ERR] 
 [ERR] Method_being_compiled=com/ibm/oti/vm/VM.markCurrentThreadAsSystemImpl()I
 [ERR] Target=2_90_20190720_204 (Mac OS X 10.11.6)
 [ERR] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] ---------------------------------------
pshipton commented 5 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_sanity.functional_x86-64_mac_Nightly/15

00:48:35      [javac] Compiling 23 source files to /Users/jenkins/workspace/Test_openjdk13_j9_sanity.functional_x86-64_mac_Nightly/openjdk-tests/functional/cmdLineTests/shareClassTests/TokenHelperTests/bin/Alphabet
00:48:36      [javac] Unhandled exception
00:48:36      [javac] Type=Segmentation error vmState=0x0005ffff
00:48:36      [javac] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
00:48:36      [javac] Handler1=000000000F09CA30 Handler2=000000000F28F980 InaccessibleAddress=0000000000000040
00:48:36      [javac] RDI=00000000105E13FD RSI=00007000002593A4 RAX=0000000000000000 RBX=00000000105E13FD
00:48:36      [javac] RCX=000000000000004A RDX=00007FACA2EB0FF0 R8=00000000FFFFFFF3 R9=000000000000000D
00:48:36      [javac] R10=00007FACA2EB1258 R11=0000000000000040 R12=00000000FFFFFFF3 R13=00000000296823D0
00:48:36      [javac] R14=000000000000004A R15=0000000000000054
00:48:36      [javac] RIP=00007FFF93FCE7D4 GS=0000 FS=0000 RSP=0000700000259370
00:48:36      [javac] RFlags=0000000000010202 CS=002B RBP=0000700000259390 ERR=0000004000000004
00:48:36      [javac] TRAPNO=000000040000000E CPU=0040000000040000 FAULTVADDR=0000000000000040
00:48:36      [javac] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM2 00000028000001f4 (f: 500.000000, d: 8.487983e-313)
00:48:36      [javac] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
00:48:36      [javac] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:48:36      [javac] Module=/usr/lib/system/libsystem_c.dylib
00:48:36      [javac] Module_base_address=00007FFF93F6F000 Symbol=__findenv
00:48:36      [javac] Symbol_address=00007FFF93FCE780
00:48:36      [javac] 
00:48:36      [javac] Method_being_compiled=com/ibm/tools/attach/target/IPC.getUid()J
00:48:36      [javac] Target=2_90_20190819_36 (Mac OS X 10.11.6)
00:48:36      [javac] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
00:48:36      [javac] ----------- Stack Backtrace -----------
00:48:36      [javac] ---------------------------------------
pshipton commented 5 years ago

I don't see any indication somebody is working on this, I'll move it forward to the next release.

pshipton commented 5 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_extended.functional_x86-64_mac_Nightly/65 cmdLineTester_lockWordAlignment_Object_Standard_0

Testing: Check oji_extends_level_2_O is still aligned with -Xlockword:noLockword
Test start time: 2019/10/12 02:33:49 Eastern Standard Time
Running command: "/Users/jenkins/workspace/Test_openjdk13_j9_extended.functional_x86-64_mac_Nightly/openjdkbinary/j2sdk-image/bin/java" -Xcompressedrefs  --add-exports=java.base/com.ibm.oti.vm=ALL-UNNAMED -Xbootclasspath/a:/Users/jenkins/workspace/Test_openjdk13_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/functional/cmdLineTests/lockWordAlignment/alignment.jar   -Xlockword:noLockword=oji_extends_level_2_O main oji_extends_level_2_O
Time spent starting: 3 milliseconds
Time spent executing: 8032 milliseconds
Test result: FAILED
Output from test:
 [OUT] 12
 [OUT] 16
 [OUT] 24
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x0005ffff
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
 [ERR] Handler1=000000000FA9B9A0 Handler2=000000000FC93880
 [ERR] RDI=0000000010FA8E83 RSI=00007000002593C4 RAX=0000000000000000 RBX=0000000010FA8E83
 [ERR] RCX=000000000000004A RDX=00007F8A3AEA8C20 R8=00000000FFFFFFF3 R9=000000000000000D
 [ERR] R10=00007F8A3AEA8EA0 R11=9FD3CECBBADBAD67 R12=00000000FFFFFFF3 R13=000000002828DE90
 [ERR] R14=000000000000004A R15=0000000000000054
 [ERR] RIP=00007FFF831E87D4 GS=0000 FS=0000 RSP=0000700000259390
 [ERR] RFlags=0000000000010202 CS=002B RBP=00007000002593B0 ERR=3B8D022800000000
 [ERR] TRAPNO=000000000000000D CPU=0228000000000000 FAULTVADDR=00007F8A3B8D0228
 [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM1 000000057fffffff (f: 2147483648.000000, d: 1.167098e-313)
 [ERR] XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
 [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/usr/lib/system/libsystem_c.dylib
 [ERR] Module_base_address=00007FFF83189000 Symbol=__findenv
 [ERR] Symbol_address=00007FFF831E8780
 [ERR] 
 [ERR] Method_being_compiled=java/lang/String.substring(II)Ljava/lang/String;
 [ERR] Target=2_90_20191011_89 (Mac OS X 10.11.6)
 [ERR] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] ---------------------------------------
pshipton commented 5 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly/166

00:33:01      [javac] Compiling 20 source files to /Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly/openjdk-tests/functional/cmdLineTests/shareClassTests/TokenHelperTests/bin/AnimalsJar
00:33:01      [javac] Unhandled exception
00:33:01      [javac] Type=Segmentation error vmState=0x0005ffff
00:33:01      [javac] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
00:33:01      [javac] Handler1=000000001069B9A0 Handler2=000000000FD42880
00:33:01      [javac] RDI=0000000011B2C32E RSI=00007000002593C4 RAX=0000000000000000 RBX=0000000011B2C32E
00:33:01      [javac] RCX=000000000000004A RDX=00007FF8CB60F800 R8=00000000FFFFFFF3 R9=000000000000000D
00:33:01      [javac] R10=00007FF8CB60FA78 R11=000800000FCDCA30 R12=00000000FFFFFFF3 R13=0000000028CE1E80
00:33:01      [javac] R14=000000000000004A R15=0000000000000054
00:33:01      [javac] RIP=00007FFF831E87D4 GS=0000 FS=0000 RSP=0000700000259390
00:33:01      [javac] RFlags=0000000000010202 CS=002B RBP=00007000002593B0 ERR=2A8290F800000000
00:33:01      [javac] TRAPNO=000000000000000D CPU=90F8000000000000 FAULTVADDR=000000002A8290F8
00:33:01      [javac] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM1 0000f80100000014 (f: 20.000000, d: 1.347234e-309)
00:33:01      [javac] XMM2 000007d000000fa0 (f: 4000.000000, d: 4.243992e-311)
00:33:01      [javac] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
00:33:01      [javac] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
00:33:01      [javac] Module=/usr/lib/system/libsystem_c.dylib
00:33:01      [javac] Module_base_address=00007FFF83189000 Symbol=__findenv
00:33:01      [javac] Symbol_address=00007FFF831E8780
00:33:01      [javac] 
00:33:01      [javac] Method_being_compiled=openj9/internal/tools/attach/target/FileLock.unlockFileImpl(J)I
00:33:01      [javac] Target=2_90_20191013_173 (Mac OS X 10.11.6)
00:33:01      [javac] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
00:33:01      [javac] ----------- Stack Backtrace -----------
00:33:01      [javac] ---------------------------------------
pshipton commented 5 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.openjdk_x86-64_mac_Nightly/1

Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=000000000F28AFB0 Handler2=000000000F4B4C20
RDI=00000000107F201B RSI=00007000002DC374 RAX=0000000000000000 RBX=00000000107F201B
RCX=000000000000004A RDX=00007FCF5A502DF0 R8=00000000FFFFFFF3 R9=000000000000000D
R10=00007FCF5A502E20 R11=6E756F6620746F6E R12=00000000FFFFFFF3 R13=00007000002DC9E8
R14=000000000000004A R15=0000000000000054
RIP=00007FFF9E9BB7D4 GS=0000 FS=0000 RSP=00007000002DC340
RFlags=0000000000010202 CS=002B RBP=00007000002DC360 ERR=1EE1201000000000
TRAPNO=000000000000000D CPU=2010000000000000 FAULTVADDR=000000001EE12010
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF9E95C000 Symbol=__findenv
Symbol_address=00007FFF9E9BB780

Method_being_compiled=java/lang/Thread.setNameImpl(JLjava/lang/String;)V
Target=2_90_20191019_178 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
andrewcraik commented 4 years ago

I still have no idea what is going wrong inside __findenv as far as I can tell we aren't doing anything bad in our call to it...

DanHeidinga commented 4 years ago

@andrewcraik I can't see anything wrong in our use of findenv either. Given it crashes as frequently as it does, would it be possible to change the way the JIT uses envvars?

This would require a larger discussion of course and the idea would be to centralize the envvar calls into one place and then reuse the result, rather then hitting findenv repeatedly.

I assume the JIT doesn't expect the env vars to change at runtime...

andrewcraik commented 4 years ago

Well the JIT uses env vars as an easy way to add internal debug controls to optimizations that do not warrant full Xjit options since they make no sense to document and probably only mean something if you trying to debug etc. Centralization would make this more complex so we would need to discuss how we might try to manage this. It would also affect OMR so it would actually need to be an OMR level discussions since the env var usage is across the compiler component in both projects.

@DanHeidinga why do you think centralization would help? Do you think it is something about when the findenv is running? You are right that the JIT doesn't expect env vars to change at runtime.

DanHeidinga commented 4 years ago

why do you think centralization would help? Do you think it is something about when the findenv is running?

If something else is modifying the env when we're accessing it, I could see that resulting in a crash. Centralizing it - and checking once - would decrease the chances of concurrent modification

andrewcraik commented 4 years ago

Ok I see your thought process. The centeralization would have to be lazy to avoid a startup hit so won't be simple. We haven't had any mention of this from users - could it be the test harness? Given we are talking about changing timing it doesn't seem an ideal fix. I do appreciate where you are coming from and am not ruling it out for sure...

andrewcraik commented 4 years ago

This is a total mac noob question - is there any tool like strace/dtrace where it might be possible to trace all the calls to setenv in one of these runs so we could at least scrutinize the callers of setenv for possibly creating a data race?

If someone could run a trace like that and point at all the setenv callers on one of these runs I'd be more than happy to go and start checking the code to see if we could have a datarace leading to some kind of corruption and crash - seems to be the most common cause of the failure symptom we are seeing from what some internet searching tells me...

andrewcraik commented 4 years ago

given the BSD origins of OSX Google suggests that dtruss might be the way to go ... https://www.freebsd.org/cgi/man.cgi?query=dtruss&sektion=1&manpath=FreeBSD+8.2-RELEASE.

Anyone have a mac and willing to try and see if we can get a set of traces?

pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.system_x86-64_mac_Nightly/207 DaaLoadTest_daa3_0

DLT stderr Type=Segmentation error vmState=0x0005ffff
DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
DLT stderr Handler1=00000000024E9ED0 Handler2=00000000026E1B60 InaccessibleAddress=0000000000000040
DLT stderr RDI=0000000003A200FF RSI=00007000002DC354 RAX=0000000000000000 RBX=0000000003A200FF
DLT stderr RCX=0000000000000053 RDX=00007FE1D1604540 R8=00000000FFFFFFF3 R9=000000000000000D
DLT stderr R10=00007FE1D16047D8 R11=0000000000000040 R12=00000000FFFFFFF3 R13=00007000002DC9E8
DLT stderr R14=0000000000000053 R15=0000000000000054
DLT stderr RIP=00007FFF80A347D4 GS=0000 FS=0000 RSP=00007000002DC320
DLT stderr RFlags=0000000000010202 CS=002B RBP=00007000002DC340 ERR=0000004000000004
DLT stderr TRAPNO=000000040000000E CPU=0040000000040000 FAULTVADDR=0000000000000040
DLT stderr XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
DLT stderr XMM5 ffffffffffffffff (f: 4294967296.000000, d: nan)
DLT stderr XMM6 0000000100000001 (f: 1.000000, d: 2.121996e-314)
DLT stderr XMM7 8000001580000015 (f: 2147483648.000000, d: -4.562291e-313)
DLT stderr XMM8 8000000080000000 (f: 2147483648.000000, d: -1.060998e-314)
DLT stderr XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr Module=/usr/lib/system/libsystem_c.dylib
DLT stderr Module_base_address=00007FFF809D5000 Symbol=__findenv
DLT stderr Symbol_address=00007FFF80A34780
DLT stderr 
DLT stderr Method_being_compiled=java/lang/String.indexOf(Ljava/lang/String;I)I
DLT stderr Target=2_90_20191130_211 (Mac OS X 10.11.6)
DLT stderr CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
DLT stderr ----------- Stack Backtrace -----------
DLT stderr ---------------------------------------
pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/224 building the test material

23:02:46      [javac] Type=Segmentation error vmState=0x0005ffff
23:02:46      [javac] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
23:02:46      [javac] Handler1=000000000C8A1ED0 Handler2=000000000BF80950
23:02:46      [javac] RDI=000000000DD76C2F RSI=00007000002DC354 RAX=0000000000000000 RBX=000000000DD76C2F
23:02:46      [javac] RCX=000000000000004A RDX=00007FE620F11210 R8=00000000FFFFFFF3 R9=000000000000000D
23:02:46      [javac] R10=00007FE620F114A8 R11=9D44F3FFBADBAD21 R12=00000000FFFFFFF3 R13=00007000002DC9E8
23:02:46      [javac] R14=000000000000004A R15=0000000000000054
23:02:46      [javac] RIP=00007FFF80A347D4 GS=0000 FS=0000 RSP=00007000002DC320
23:02:46      [javac] RFlags=0000000000010202 CS=002B RBP=00007000002DC340 ERR=2202DA2800000000
23:02:46      [javac] TRAPNO=000000000000000D CPU=DA28000000000000 FAULTVADDR=00007FE62202DA28
23:02:46      [javac] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM1 0000f80100000014 (f: 20.000000, d: 1.347234e-309)
23:02:46      [javac] XMM2 000007d000000fa0 (f: 4000.000000, d: 4.243992e-311)
23:02:46      [javac] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
23:02:46      [javac] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:02:46      [javac] Module=/usr/lib/system/libsystem_c.dylib
23:02:46      [javac] Module_base_address=00007FFF809D5000 Symbol=__findenv
23:02:46      [javac] Symbol_address=00007FFF80A34780
23:02:46      [javac] 
23:02:46      [javac] Method_being_compiled=com/ibm/oti/vm/VM.markCurrentThreadAsSystemImpl()I
23:02:46      [javac] Target=2_90_20191219_231 (Mac OS X 10.11.6)
23:02:46      [javac] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
23:02:46      [javac] ----------- Stack Backtrace -----------
23:02:46      [javac] ---------------------------------------
pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/231 testSCCacheManagement_0

.Running  :testCacheCreationListingDestroying02  (Test suite : TestOptionsCacheDir)
.Running  :testCacheDir01  (Test suite : TestOptionsCacheDir)
SimpleApp running

[-Xshareclasses persistent cache enabled]
[-Xshareclasses verbose output enabled]
JVMSHRC236I Created shared classes persistent cache Foo
JVMSHRC246I Attached shared classes persistent cache Foo
JVMSHRC765I Memory page protection on runtime data, string read-write data and partially filled pages is successfully enabled
Unhandled exception
Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000000000BCA1ED0 Handler2=000000000B39A950 InaccessibleAddress=0000000000010000
RDI=000000000D176C2F RSI=00007000002DC354 RAX=0000000000000000 RBX=000000000D176C2F
RCX=000000000000004A RDX=00007FC59BC1BBB0 R8=00000000FFFFFFF3 R9=000000000000000D
R10=00007FC59BC1BE70 R11=0000000000010000 R12=00000000FFFFFFF3 R13=00007000002DC9E8
R14=000000000000004A R15=0000000000000054
RIP=00007FFF80A347D4 GS=0000 FS=0000 RSP=00007000002DC320
RFlags=0000000000010202 CS=002B RBP=00007000002DC340 ERR=0001000000000004
TRAPNO=000000040000000E CPU=0000000000040000 FAULTVADDR=0000000000010000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 00000000000000ff (f: 255.000000, d: 1.259867e-321)
XMM2 00000000000000ff (f: 255.000000, d: 1.259867e-321)
XMM3 2f6f692f6176616a (f: 1635148160.000000, d: 3.311397e-80)
XMM4 694678696e552f6f (f: 1851076480.000000, d: 1.343746e+199)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF809D5000 Symbol=__findenv
Symbol_address=00007FFF80A34780

Method_being_compiled=java/io/UnixFileSystem.createFileExclusively(Ljava/lang/String;)Z
Target=2_90_20191225_237 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2019/12/25 23:10:22 - please wait.
JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15773330254930/testSCCacheManagement_0/core.20191225.231022.69987.0001.dmp' in response to an event
JVMDUMP010I System dump written to /Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15773330254930/testSCCacheManagement_0/core.20191225.231022.69987.0001.dmp
JVMDUMP032I JVM requested Java dump using '/Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15773330254930/testSCCacheManagement_0/javacore.20191225.231022.69987.0002.txt' in response to an event
JVMDUMP010I Java dump written to /Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15773330254930/testSCCacheManagement_0/javacore.20191225.231022.69987.0002.txt
JVMDUMP032I JVM requested Snap dump using '/Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15773330254930/testSCCacheManagement_0/Snap.20191225.231022.69987.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15773330254930/testSCCacheManagement_0/Snap.20191225.231022.69987.0003.trc
JVMDUMP007I JVM Requesting JIT dump using '/Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15773330254930/testSCCacheManagement_0/jitdump.20191225.231022.69987.0004.dmp'
JVMDUMP010I JIT dump written to /Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15773330254930/testSCCacheManagement_0/jitdump.20191225.231022.69987.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".

E.Running  :testCacheDir06  (Test suite : TestOptionsCacheDir)
pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly/282 testSCCacheManagement_0

Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=0000000002A9B1D0 Handler2=00000000021A4370
RDI=0000000003FD01C2 RSI=0000700000259374 RAX=0000000000000000 RBX=0000000003FD01C2
RCX=000000000000004A RDX=00007FAC22DA9410 R8=00000000FFFFFFF3 R9=000000000000000D
R10=00007FAC22DA96D0 R11=99EB1CA2BADBAD21 R12=00000000FFFFFFF3 R13=000000002D6B7F50
R14=000000000000004A R15=0000000000000054
RIP=00007FFF80A347D4 GS=0000 FS=0000 RSP=0000700000259340
RFlags=0000000000010202 CS=002B RBP=0000700000259360 ERR=2D72D01000000000
TRAPNO=000000000000000D CPU=D010000000000000 FAULTVADDR=000000002D72D010
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 ff00000000000000 (f: 0.000000, d: -5.486124e+303)
XMM2 ff000000000000ff (f: 255.000000, d: -5.486124e+303)
XMM3 6974752f6176616a (f: 1635148160.000000, d: 9.787124e+199)
XMM4 6c6974752f617661 (f: 794916480.000000, d: 1.713873e+214)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF809D5000 Symbol=__findenv
Symbol_address=00007FFF80A34780

Method_being_compiled=java/util/HashMap.newNode(ILjava/lang/Object;Ljava/lang/Object;Ljava/util/HashMap$Node;)Ljava/util/HashMap$Node;
Target=2_90_20200220_294 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk14_j9_extended.system_x86-64_mac_Release/7 SharedClasses.SCM23.MultiThread_0

GEN stderr Unhandled exception
GEN stderr Type=Segmentation error vmState=0x0005ffff
GEN stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
GEN stderr Handler1=000000000F8E4330 Handler2=000000000FAE45B0
GEN stderr RDI=0000000010E93FA7 RSI=000070000045F374 RAX=0000000000000000 RBX=0000000010E93FA8
GEN stderr RCX=0000000000000045 RDX=00007FBC1054E7C0 R8=00000000FFFFFFF3 R9=000000000000000D
GEN stderr R10=00007FBC1054EA60 R11=B5E84046BADBAD67 R12=00000000FFFFFFF4 R13=000000002A4D12D0
GEN stderr R14=0000000000000045 R15=0000000000000052
GEN stderr RIP=00007FFF9C7777D4 GS=0000 FS=0000 RSP=000070000045F340
GEN stderr RFlags=0000000000010202 CS=002B RBP=000070000045F360 ERR=11CFC57800000000
GEN stderr TRAPNO=000000000000000D CPU=C578000000000000 FAULTVADDR=0000000011CFC578
GEN stderr XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM2 0000000000ff0000 (f: 16711680.000000, d: 8.256667e-317)
GEN stderr XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
GEN stderr XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
GEN stderr Module=/usr/lib/system/libsystem_c.dylib
GEN stderr Module_base_address=00007FFF9C718000 Symbol=__findenv
GEN stderr Symbol_address=00007FFF9C777780
GEN stderr 
GEN stderr Method_being_compiled=jdk/internal/jimage/ImageStringsReader.hashCode(Ljava/lang/String;Ljava/lang/String;I)I
GEN stderr Target=2_90_20200321_9 (Mac OS X 10.11.6)
GEN stderr CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
GEN stderr ----------- Stack Backtrace -----------
GEN stderr ---------------------------------------
pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/331 cmdLineTester_shrcdbgddrext_1 Testing: Run !shrc aotstats

pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/65 osx1011-x86-1 testSCCacheManagement_0

pshipton commented 3 years ago

fyi https://github.com/eclipse/openj9/issues/11430