search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

Mac crash on osx1011-x86-2 in array copy test #9782

Open pshipton opened 4 years ago

pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_uma_Nightly_testList_0/5 TestArrayCopy_3 osx1011-x86-2

TestArrayCopy_3 Start Time: Wed Jun  3 02:20:17 2020 Epoch Time (ms): 1591165217431
variation: Mode619
JVM_OPTIONS:  -Xcompressedrefs -Xgcpolicy:optavgpause -Xjit 
[IncludeExcludeTestAnnotationTransformer] [INFO] exclude file is /Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_uma_Nightly_testList_0/openjdk-tests/TKG/../TestConfig/resources/excludes/latest_exclude_11.txt
...
... TestNG 6.14.2 by Cédric Beust (cedric@beust.com)
...

Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000000000B69E5B0 Handler2=000000000AC98480 InaccessibleAddress=00000000000003A8
RDI=00000000B007F918 RSI=000000000D5FC218 RAX=00000000000003A8 RBX=00000000300B9014
RCX=0000000030787720 RDX=0000000000000001 R8=000000000D5FC218 R9=00000000C00DB2F2
R10=00000000B007F920 R11=00000000B007F910 R12=0000000000000000 R13=000000000D534E60
R14=00007F943B009698 R15=000000000D534B00
RIP=000000000B62FC67 GS=0000 FS=0000 RSP=00000000B007F670
RFlags=0000000000010646 CS=002B RBP=00000000B007F8F0 ERR=000003A800000004
TRAPNO=000000040000000E CPU=03A8000000040000 FAULTVADDR=00000000000003A8
XMM0 0000000000000003 (f: 3.000000, d: 1.482197e-323)
XMM1 00000000b007f2d8 (f: 2953310976.000000, d: 1.459129e-314)
XMM2 000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
XMM3 747365742f67726f (f: 795308672.000000, d: 8.887822e+252)
XMM4 4570696b532f676e (f: 1395615616.000000, d: 3.174503e+26)
XMM5 00000000df000000 (f: 3741319168.000000, d: 1.848457e-314)
XMM6 3fdd83e7258a2f3e (f: 629813056.000000, d: 4.611757e-01)
XMM7 4023687a9f1af2b1 (f: 2669343488.000000, d: 9.704061e+00)
XMM8 0000000000000003 (f: 3.000000, d: 1.482197e-323)
XMM9 63008d6a2f9976d9 (f: 798586560.000000, d: 7.808518e+168)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 ca62c1d6ca62c1d6 (f: 3395469824.000000, d: -2.193092e+50)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_uma_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/compressedrefs/libj9vm29.dylib
Module_base_address=000000000B600000 Symbol=_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread
Symbol_address=000000000B6102C0
Target=2_90_20200602_6 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
gacholio commented 4 years ago

@liqunl Please subscribe me to the item for the diagnostic work when it is created.

DanHeidinga commented 4 years ago

Moving this forward as diagnostic code has been added but the root cause hasn't yet been identified

andrewcraik commented 4 years ago

@liqunl has the code been enabled? any results to report?

liqunl commented 4 years ago

@andrewcraik No, I still need to

  1. Try to reproduce in the old web UI
  2. Look at the security manager error

Will look at it today.

andrewcraik commented 4 years ago

Unlikely to be fixed for this release, but we will get the testing mode enabled and see if we can catch it - moving forward as a result.

pshipton commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_mac_Release_testList_1/9

liqunl commented 4 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_mac_Release_testList_1/9

Rflag in javacore is 0x0000000000010646, same as the original crash. Java call stack is different

4XESTACKTRACE                at java/lang/String.concat(String.java:1601)
4XESTACKTRACE                at java/net/URLClassLoader$ClassFinder.run(URLClassLoader.java:1082)
4XESTACKTRACE                at java/security/AccessController.doPrivileged(AccessController.java:770)
4XESTACKTRACE                at java/net/URLClassLoader.findClass(URLClassLoader.java:589(Compiled Code))                                                            
4XESTACKTRACE                at java/lang/ClassLoader.loadClassHelper(ClassLoader.java:953(Compiled Code))
5XESTACKTRACE                   (entered lock: java/lang/ClassLoader$ClassNameBasedLock@0x0000000080733588, entry count: 1)
4XESTACKTRACE                at java/lang/ClassLoader.loadClass(ClassLoader.java:898)
4XESTACKTRACE                at java/lang/ClassLoader.loadClassHelper(ClassLoader.java:942(Compiled Code))
5XESTACKTRACE                   (entered lock: java/lang/ClassLoader$ClassNameBasedLock@0x0000000080733540, entry count: 1)
4XESTACKTRACE                at java/lang/ClassLoader.loadClass(ClassLoader.java:898)

@pshipton How can I get the permission to run grinders? Where can I download the build?

pshipton commented 4 years ago

You can run internal grinders. To run grinders at OpenJ9 you need to be a committer. Let me know what you want to run and I can start the grinders.

To find the build, there is a link in the test console output, or see the Started by upstream project link and follow that to find the build.

The build is https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK8_x86-64_mac_Release/44/OpenJ9-JDK8-x86-64_mac-20201005-021706.tar.gz

pshipton commented 4 years ago

Starting with 60 iterations on 3 machines (180x), and see if that works. https://ci.eclipse.org/openj9/job/Grinder/1101/ - passed

liqunl commented 3 years ago

I can't find anything useful from crash in https://github.com/eclipse/openj9/issues/9782#issuecomment-703605885. The place we crash is too far away from the place where the flag is corrupted. And we haven't seen reproduction since then. We just have to wait for a crash in the diagnostic check.

pshipton commented 3 years ago

Considering this hasn't been seen in the last couple months, removing it from the milestone plans.

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_1/196 Diagnostics https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_1/196/functional_test_output.tar.gz perhaps related to https://github.com/eclipse/openj9/issues/11130

Running test TestArrayCopy_3 ...
===============================================
TestArrayCopy_3 Start Time: Fri Dec 25 03:28:56 2020 Epoch Time (ms): 1608884936768
variation: Mode619
JVM_OPTIONS:  -Xcompressedrefs -Xgcpolicy:optavgpause -Xjit 
[IncludeExcludeTestAnnotationTransformer] [INFO] exclude file is /Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdk-tests/TKG/../TestConfig/resources/excludes/latest_exclude_11.txt
...
... TestNG 6.14.2 by Cédric Beust (cedric@beust.com)
...

Unhandled exception
Type=Unhandled trap vmState=0x00000000
J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=00000001
Handler1=000000000D282830 Handler2=000000000D4F5B90
RDI=000000000F74ED80 RSI=000000000F74ED88 RAX=00000000307566B8 RBX=000000000D467C98
RCX=25005B57DCC064D2 RDX=00000000B007F3B8 R8=00000000307566B8 R9=0000000000000015
R10=0000000000000000 R11=0000000000000000 R12=000000000D462150 R13=0000000000000000
R14=000000000F6B1900 R15=0000000028032450
RIP=00000000B0084BB5 GS=0000 FS=0000 RSP=000000000F74ED70
RFlags=0000000000000606 CS=002B RBP=000000000F670700 ERR=0E00E00000000000
TRAPNO=0000000000000003 CPU=E000000000000000 FAULTVADDR=000000000E00E000
XMM0 0000000000000003 (f: 3.000000, d: 1.482197e-323)
XMM1 000000000f74ee40 (f: 259321408.000000, d: 1.281218e-315)
XMM2 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
XMM3 000001769905a223 (f: 2567283200.000000, d: 7.948948e-312)
XMM4 000000000f74efca (f: 259321808.000000, d: 1.281220e-315)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fb253f62f0a1417 (f: 789189632.000000, d: 7.159365e-02)
XMM7 401e7f9c1e980fa9 (f: 513281952.000000, d: 7.624619e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 ca62c1d6ca62c1d6 (f: 3395469824.000000, d: -2.193092e+50)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=java/lang/Object.<init>()V
Target=2_90_20201224_597 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
gacholio commented 3 years ago

I can't imagine how the Object constructor would be doing arrayCopy. It is worth noting that the DF (0x400) is set the flags.

liqunl commented 3 years ago

The Object constructor is not doing arraycopy, the assert fires at the entry of the compiled method, so the flag was set before the constructor run.

The problem is that we are accessing invalid address during backward rep movsd, we're in the process to throw an NPE before we can reset the DF flag, thus hitting the assertion elsewhere.

std            
rep movsd dword ptr [rdi], dword ptr [rsi]   <--- NPE
cld

I suspect other failures are due to the same reason. We probably shouldn't throw NPE here (another JIT bug is causing the NPE), don't know if there's a case where NPE can be allowed. We'd better make sure the flag set and reset completes without interruption, otherwise don't do the operation requiring setting the flag.

JamesKingdon commented 3 years ago

This sounds like a great observation @liqunl !

JamesKingdon commented 3 years ago

Would it make sense to have the trap handler conservatively reset the direction flag?

liqunl commented 3 years ago

Would it make sense to have the trap handler conservatively reset the direction flag?

Good idea, I think we can.

gacholio commented 3 years ago

This would be a bad idea - we're trying to track this down, not hide it.

gacholio commented 3 years ago

Not to mention, by the time we crash, who knows what's already been corrupted. The odds of resuming successfully are zero.

JamesKingdon commented 3 years ago

So agreed we shouldn't be trying to sweep the problem under the carpet, but if I understand correctly Liqun has suggested a mechanism that can cause the flag to be left set - she's done the tracking down. The trap handler for NPE might usefully be extended to catch that case, no? Maybe it would be better to inspect the instruction causing the NPE and only clear the flag if it was a rep?

JamesKingdon commented 3 years ago

I guess the alternative is to try and ensure that the npe trap handler can never be invoked during a rep. If we did that, I'd recommend then checking for rep in the trap handler and treating it as an internal error condition.

gacholio commented 3 years ago

As I understand it, the NPE trap fires when we fail to read/write memory below a certain small address (i.e. NULL object ptr + small offset) and the crashing PC is within a compiled method. It's not clear to me how a rep could ever hit this case unless the base was invalid (i.e. 0).

The issue with the DF here really has nothing to do with NPE.

If we were to generate a rep to zero out an object which was invalidly 0, we would want the NPE trap to throw the exception (consider zeroing out a small byte array).

Also, the DF detection is not enabled by default - it generates throwaway code in certain places to try and track down this issue. I can't imagine it will ever be on by default.

JamesKingdon commented 3 years ago

Ah, perhaps I misunderstood then. I thought the trap handler fired on any gpf, and then the code in the handler made a decision on whether the access qualified as an NPE or not. We get a steady stream of false NPE pmrs due to arbitrary JIT bugs falling into this scenario. So my interpretation was that we go into the std/rep/cld sequence with a bad destination pointer (for whatever reason), take the NPE with the direction flag set, deliver the NPE and then continue having never run cld, usually to crash later. I assume that in at least one case we crashed while handling the NPE, allowing the scenario to be detected. If the above can happen then we need some way of either avoiding the trap or compensating the direction flag.

gacholio commented 3 years ago

I think you're right about everything you said.

I'm confused about @liqunl mentioning an assert. Was this test even running with the debug option on?

Your conjecture about the NPE sounds entirely plausible, though I'm not sure it's happening here. Why are we crashing in the compiled Object constructor, which is always an entirely empty method unless instrumentation has modified it, which does not appear to be the case here.

One possible mitigation for the issue might be to do an explicit NULL check before anything that's going to generate the backwards rep memcpy (this has to be pretty infrequent). This won't help if the object pointer is completely invalid, but nothing will, really.

If we want to go with the CLD in the NPE case, it might be worth checking and recording the fact that this occurred somewhere so if we crash later, we know it might have involved the DF being set.

liqunl commented 3 years ago

The assert I mentioned is a crash forced by the DF detection I added, which only turns on in our test with a JIT environment variable. The reason why we crash in Object constructor is because at the entry of this JIT method, it detected that DF flag has been set, thus crashing the program.

The reason I suspect we were in the process of throwing an NPE is because of the java call stack

4XESTACKTRACE                at java/lang/Object.<init>(Object.java)
4XESTACKTRACE                at java/lang/Throwable.<init>(Throwable.java:89)
4XESTACKTRACE                at java/lang/Throwable.<init>(Throwable.java:101)
4XESTACKTRACE                at java/lang/Exception.<init>(Exception.java:66)
4XESTACKTRACE                at java/lang/RuntimeException.<init>(RuntimeException.java:62)
4XESTACKTRACE                at java/lang/NullPointerException.<init>(NullPointerException.java:70)
4XESTACKTRACE                at j9vm/test/arrayCopy/ArrayCopyTest.testXtraLargeReferenceArrayCopyBackward(ArrayCopyTest.java:831(Compiled Code))
4XESTACKTRACE                at j9vm/test/arrayCopy/ArrayCopyTest.testXtraLargeReferenceArrayCopy(ArrayCopyTest.java:760(Compiled Code))
4XESTACKTRACE                at j9vm/test/arrayCopy/ArrayCopyTest.testXtraLargeReferenceArrayCopy(ArrayCopyTest.java:162)

The first frame Object.<init> is compiled code where we crash. The last JIT method caller is ArrayCopyTest.testXtraLargeReferenceArrayCopy, who was doing backward array copy

std            
rep movsd dword ptr [rdi], dword ptr [rsi]  <--- executed instruction before exiting the JIT code
cld   <--- return address on the stack

We shouldn't seg fault at the rep movsd instruction, as there should be safety checks before to make sure we don't crash there. But like any opt, there can be bugs, and I suspect in this case, something's wrong with rdi or rsi so that we seg fault (I'm still trying to figure out if the two pointers are bad), thus we land in the signal handler and found that we need to throw an NPE for this instruction.

I was suspecting that the corrupted DF flag is caused by this, then we should clear the flag and let the program fail in the right way to expose the real bug. However, on a second thought, we can't say that DF flag corruption can only caused by this problem, so we can't just clear the flag in handler blindly, or maybe we should add the DF flag detection in the handler and crash in the handler.

gacholio commented 3 years ago

So this test is running in the DF detection mode?

As I said, I would not object to detecting and clearing the DF when we're going to throw NPE from jitted code. Still might be worth recording the fact (I'm sure we can find a thread-local bit that can be set).

I don't think we want to crash in the handler - this case could come up legitimately if NULL input was provided to compiled code. I doubt we'd want to add the explicit checks for a case which should essentially never occur in real life.

liqunl commented 3 years ago

So this test is running in the DF detection mode?

Correct

this case could come up legitimately if NULL input was provided to compiled code

NULL input can be provided to the compiled code, but we shouldn't do the implicit null check during backward mem copy, otherwise we have to clear the flag in NPE handler. But there are also cases where the memory pointers in the rep instruction are bad due to other bugs, and we don't want the DF flag to cause other failures to hide the original one.

liqunl commented 3 years ago

Looking at the disassembly of ArrayCopyTest.testXtraLargeReferenceArrayCopy, I saw safety checks such as nullchk, bound check before the backward array copy, but without register context, I can't figure out what is wrong. The array copy is in a loop, and the method is pretty large, it's hard to tell what is wrong and who in the JIT did the wrong thing from the disassembly.

@gacholio @JamesKingdon @0xdaryl Any advice in the next step? We should fail in a way to help investigating the unexpected array copy termination. I wonder if we can add the same diagnostic check in the X86 signal handler. The check will only be enabled in our test and will crash the VM if DF flag is set.

JamesKingdon commented 3 years ago

Sounds good to me. The earlier we can catch the problem the easier it will be to get the register values.

gacholio commented 3 years ago

For debugging purposes, I would recommend that the JIT GP handler be updated so that in the case where it's going to resume at the throw NPE code, check the DF. If DF is set, don't consider this an NPE and just proceed to let the outer VM GP handler produce the crash dump.

This will provide the full register context at the point of GPF in the compiled code.

liqunl commented 3 years ago

@pshipton This issue won't be fixed in next release. A new diagnostic check will be added to help the investigation in next occurrence.

pshipton commented 3 years ago

Thanks for the update. This Issue isn't in any Milestone plan atm, so there is nothing to update.

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_mixed_Nightly_testList_0/8/ TestArrayCopy_openj9_none_SCC_1

Unhandled exception
Type=Unhandled trap vmState=0x00000000
J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=00000001
Handler1=0000000005087290 Handler2=000000000535F800
RDI=00007F8EB2092880 RSI=00007F8EB2092888 RAX=000000000805F788 RBX=00000000052CC628
RCX=D50008203E499A31 RDX=00000000B007F3A0 R8=000000000805F788 R9=0000000000000015
R10=0000000000000000 R11=00000000B007F688 R12=00000000052C51C0 R13=0000000000000000
R14=00007F8EB20B9C00 R15=00007F8EB2003450
RIP=00000000923A2AF5 GS=0000 FS=0000 RSP=00007F8EB2092870
RFlags=0000000000000606 CS=002B RBP=00007F8EB105F100 ERR=05E5800000000000
TRAPNO=0000000000000003 CPU=8000000000000000 FAULTVADDR=0000000005E58000
XMM0 0000000000000001 (f: 1.000000, d: 4.940656e-324)
XMM1 00007f8eb2092940 (f: 2986944768.000000, d: 6.929313e-310)
XMM2 0000000000000006 (f: 6.000000, d: 2.964394e-323)
XMM3 0000017776846106 (f: 1988387072.000000, d: 7.967308e-312)
XMM4 00007f8eb2092a9a (f: 2986945280.000000, d: 6.929313e-310)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fd347dd9a987d55 (f: 2593684736.000000, d: 3.012613e-01)
XMM7 4020a2b23f3bab73 (f: 1060875136.000000, d: 8.317766e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 ca62c1d6ca62c1d6 (f: 3395469824.000000, d: -2.193092e+50)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=java/lang/Object.<init>()V
Target=2_90_20210205_10 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------

@liqunl does this latest crash have the diagnostic you need? https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk11_j9_sanity.functional_x86-64_mac_mixed_Nightly_testList_0/8/functional_test_output.tar.gz

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/230/ J9vmTest_5

+++ j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest: +++
command: /Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/java  -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xdump -Xms64m -Xmx64m    -classpath /Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdk-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar:/Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdk-tests/TKG/../TKG/lib/asm-all.jar  j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest 

Unhandled exception
Type=Unhandled trap vmState=0x00000000
J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=00000001
Handler1=0000000002287290 Handler2=000000000255DA40
RDI=000000000487FB70 RSI=000000000487FB78 RAX=00000000FF962058 RBX=00000000024CB628
RCX=4300EAF315696F76 RDX=0000700000867810 R8=00000000FF962058 R9=0000000000000015
R10=0000000000000000 R11=0000000000000000 R12=00000000024C41C0 R13=0000000000000000
R14=0000000004724400 R15=00007FB4FD0DB3D8
RIP=000000001998F635 GS=0000 FS=0000 RSP=000000000487FB60
RFlags=0000000000000606 CS=002B RBP=000000000487FF00 ERR=0306D00000000000
TRAPNO=0000000000000003 CPU=D000000000000000 FAULTVADDR=000000000306D000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 000000000487fc30 (f: 76020784.000000, d: 3.755926e-316)
XMM2 00000000df000000 (f: 3741319168.000000, d: 1.848457e-314)
XMM3 0000000000000010 (f: 16.000000, d: 7.905050e-323)
XMM4 00000000ff961fc0 (f: 4288028672.000000, d: 2.118568e-314)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fe011fab125ff8a (f: 2972057600.000000, d: 5.021947e-01)
XMM7 402e7f9c1e980fa9 (f: 513281952.000000, d: 1.524924e+01)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=java/lang/Object.<init>()V
Target=2_90_20210224_633 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------

https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/230/functional_test_output.tar.gz

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk16_j9_sanity.functional_x86-64_mac_Release_testList_0/2/ TestArrayCopy_1

0xdaryl commented 3 years ago

There hasn't been much progress on this to resolve it for 0.26. Moving out to 0.27.

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/255

+++ j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest: +++
command: /Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/java  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs  -Xdump -Xms64m -Xmx64m   -Xbootclasspath:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/default/jclSC180/vm.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/se-service.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/rt.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/resources.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/jsse.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/charsets.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/jce.jar -classpath /Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdk-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdk-tests/TKG/../TKG/lib/asm-all.jar  j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest 

Unhandled exception
Type=Unhandled trap vmState=0x00000000
J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=00000001
Handler1=000000000A236240 Handler2=000000000994FAA0
RDI=00007FB278945BA0 RSI=00007FB278945BA8 RAX=000000000FF2E268 RBX=0000000000000001
RCX=44002927489B2461 RDX=00007000007DE810 R8=000000000FF2E268 R9=0000000000000015
R10=0000000000000000 R11=0000000000000000 R12=000000000A4721C0 R13=0000000000000000
R14=00007FB27908EF00 R15=00007FFF872C7EE5
RIP=0000000010895255 GS=0000 FS=0000 RSP=00007FB278945B90
RFlags=0000000000000606 CS=002B RBP=00007FB279123300 ERR=0AF9800000000000
TRAPNO=0000000000000003 CPU=8000000000000000 FAULTVADDR=000000000AF98000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 00007fb278945c60 (f: 2022988928.000000, d: 6.936904e-310)
XMM2 000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM4 000000000ff2e000 (f: 267575296.000000, d: 1.321998e-315)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fe109f39e2d4c97 (f: 2653768960.000000, d: 5.324648e-01)
XMM7 402e7f9c1e980fa9 (f: 513281952.000000, d: 1.524924e+01)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=java/lang/Object.<init>()V
Target=2_90_20210309_655 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_OMR_testList_1/217

+++ j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest: +++
command: /Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_OMR_testList_1/openjdkbinary/j2sdk-image/bin/java  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs  -Xdump -Xms64m -Xmx64m    -classpath /Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_OMR_testList_1/openjdk-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar:/Users/jenkins/workspace/Test_openjdk11_j9_sanity.functional_x86-64_mac_OMR_testList_1/openjdk-tests/TKG/../TKG/lib/asm-all.jar  j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest 

Unhandled exception
Type=Unhandled trap vmState=0x00000000
J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=00000001
Handler1=0000000003036290 Handler2=00000000026C7BC0
RDI=00007FCB491B0BF0 RSI=00007FCB491B0BF8 RAX=0000000008D1E0F8 RBX=0000700000824B20
RCX=FD007A43D7EC06B3 RDX=0000700000824810 R8=0000000008D1E0F8 R9=0000000000000015
R10=0000000000000000 R11=0000700000824AF8 R12=00000000032731C0 R13=0000000000000000
R14=00007FCB490B5600 R15=00007FCB49196450
RIP=0000000011DA9635 GS=0000 FS=0000 RSP=00007FCB491B0BE0
RFlags=0000000000000606 CS=002B RBP=00007FCB491B1100 ERR=03D0000000000000
TRAPNO=0000000000000003 CPU=0000000000000000 FAULTVADDR=0000000003D00000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 00007fcb491b0cb0 (f: 1226509440.000000, d: 6.942170e-310)
XMM2 00000000df000000 (f: 3741319168.000000, d: 1.848457e-314)
XMM3 3dd67be8e4a77571 (f: 3836179712.000000, d: 8.179640e-11)
XMM4 3e8a4dbf571ce9ab (f: 1461512576.000000, d: 1.959778e-07)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fdd32fe7e00ebd5 (f: 2113989632.000000, d: 4.562374e-01)
XMM7 402e7f9c1e980fa9 (f: 513281952.000000, d: 1.524924e+01)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=java/lang/Object.<init>()V
Target=2_90_20210310_844 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk16_j9_sanity.functional_x86-64_mac_Nightly_testList_1/34 J9vmTest_0 j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest - as in previous comment

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_OMR_testList_1/224/ - osx1011-x86-2 J9vmTest_0 j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK_testList_1/154/ - osx1011-x86-2 J9vmTest_0 j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/271 - osx1011-x86-2 J9vmTest_0 MultiThreadedBackwardArrayCopyTest

https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_1/271/functional_test_output.tar.gz

EFlags 10602, EFlags & 0x400 = 400, DF flag is set
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=000000000E436250 Handler2=000000000DAB2AA0
RDI=0000000013FCCAA8 RSI=0000000013FCCAA0 RAX=0000000013FCCA10 RBX=000000001401C828
RCX=000000000000000F RDX=0000000000000020 R8=0000000000000000 R9=0000000000000625
R10=0000000000000022 R11=0000000014072A58 R12=0000000013FCCA10 R13=000000001401C828
R14=0000000013FCCA10 R15=0000000014081000
RIP=00000000149F2A30 GS=0000 FS=0000 RSP=00007FCAAC8480D0
RFlags=0000000000010602 CS=002B RBP=00007FCAAC813F00 ERR=1142700000000000
TRAPNO=000000000000000D CPU=7000000000000000 FAULTVADDR=0000000111427000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 00007fcaac848160 (f: 2894364928.000000, d: 6.942040e-310)
XMM2 00000000df000000 (f: 3741319168.000000, d: 1.848457e-314)
XMM3 000000000000001d (f: 29.000000, d: 1.432790e-322)
XMM4 0000000014081000 (f: 336072704.000000, d: 1.660420e-315)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fe0f7241c9b497d (f: 479938944.000000, d: 5.301686e-01)
XMM7 402e7f9c1e980fa9 (f: 513281952.000000, d: 1.524924e+01)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=j9vm/test/arraycopy/MultiThreadedBackwardArrayCopyTest.rotateSource()V
Target=2_90_20210325_671 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------

@liqunl @gacholio fyi

JamesKingdon commented 3 years ago

Oh cool, is that the new diagnostic kicking in? Update: I was hoping I might be able to open the corefile on Linux but no luck. There's a jitdump that captures the method which got me excited, but I'd need to see the version in the core file to know where it crashed.

pshipton commented 3 years ago

open the corefile on Linux but no luck

In what way? I can go on the machine and run jextract if that helps.

pshipton commented 3 years ago

Oh wait, we don't yet support jextract on Mac https://github.com/eclipse/openj9/issues/8809. Plus I realized you are trying to open a Mac core file on Linux. We can get you access to the machine, although not sure if additional tools need to be installed to look at core files.

Also I noticed for all the recent failures where we still have logs, and the original failure are all on osx1011-x86-2, which may or may not mean something.

liqunl commented 3 years ago

I've downloaded the core dump to my Mac, will look

pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/267/ osx1011-x86-2 TestArrayCopy_openj9_none_SCC_1

https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/267/functional_test_output.tar.gz

EFlags 10606, EFlags & 0x400 = 400, DF flag is set
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=00000000048362A0 Handler2=0000000003EEFA40
RDI=000000000753DE70 RSI=000000000753DE68 RAX=0000000000000001 RBX=0000000000000000
RCX=0000000000000006 RDX=00007FEA3A88B500 R8=0000000000800020 R9=0000000000100001
R10=0000000006D3DE20 R11=0000000000000000 R12=00000000000000B8 R13=0000000000000017
R14=00000000000000B8 R15=000000000000000E
RIP=0000000091A5407B GS=0000 FS=0000 RSP=00007FEA3A07E3B0
RFlags=0000000000010606 CS=002B RBP=00007FEA3B04AB00 ERR=8525C96000000000
TRAPNO=000000000000000D CPU=C960000000000000 FAULTVADDR=000000018525C960
XMM0 00000000ffffffff (f: 4294967296.000000, d: 2.121996e-314)
XMM1 00007fea3a07e378 (f: 973595520.000000, d: 6.948736e-310)
XMM2 0000000000000006 (f: 6.000000, d: 2.964394e-323)
XMM3 00000178b17f380d (f: 2977904640.000000, d: 7.993417e-312)
XMM4 00007fea3a07e49a (f: 973595776.000000, d: 6.948736e-310)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fde4c71a8687704 (f: 2825418496.000000, d: 4.734158e-01)
XMM7 402791272ee9dd8e (f: 787078528.000000, d: 1.178350e+01)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 ca62c1d6ca62c1d6 (f: 3395469824.000000, d: -2.193092e+50)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=j9vm/test/arrayCopy/ArrayCopyTest.testXtraLargeReferenceArrayCopyBackward(I)V
Target=2_90_20210407_670 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK_testList_1/169/ osx1011-x86-2 J9vmTest_0 j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest

https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK_testList_1/169/functional_test_output.tar.gz

EFlags 10602, EFlags & 0x400 = 400, DF flag is set
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=0000000002836890 Handler2=0000000001F2E560
RDI=00000000084871F0 RSI=00000000084871E8 RAX=0000000000000001 RBX=000000000851A5E0
RCX=000000000000000F RDX=0000000008530FB8 R8=0000000008487158 R9=0000000000000022
R10=0000000000000000 R11=0000000008533000 R12=0000000000000022 R13=0000000000000110
R14=0000000008530FB8 R15=0000000008533018
RIP=000000001172E79E GS=0000 FS=0000 RSP=00007FD7D21786C0
RFlags=0000000000010602 CS=002B RBP=00007FD7D212CD00 ERR=1231300000000000
TRAPNO=000000000000000D CPU=3000000000000000 FAULTVADDR=0000000112313000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 00007fd7d2178770 (f: 3524757248.000000, d: 6.944830e-310)
XMM2 00007fd7d2178770 (f: 3524757248.000000, d: 6.944830e-310)
XMM3 000000000000001f (f: 31.000000, d: 1.531604e-322)
XMM4 0000000008533000 (f: 139669504.000000, d: 6.900590e-316)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fdeeb20c640ddf4 (f: 3326139904.000000, d: 4.831011e-01)
XMM7 402e7f9c1e980fa9 (f: 513281952.000000, d: 1.524924e+01)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=j9vm/test/arraycopy/MultiThreadedBackwardArrayCopyTest.run()V
Target=2_90_20210410_437 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk16_j9_sanity.functional_x86-64_mac_Nightly_testList_1/64 J9vmTest_0

https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk16_j9_sanity.functional_x86-64_mac_Nightly_testList_1/64/functional_test_output.tar.gz

+++ j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest: +++
command: /Users/jenkins/workspace/Test_openjdk16_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/java  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs  -Xdump -Xms64m -Xmx64m    -classpath /Users/jenkins/workspace/Test_openjdk16_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdk-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar:/Users/jenkins/workspace/Test_openjdk16_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdk-tests/TKG/../TKG/lib/asm-all.jar  j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest 

EFlags 10602, EFlags & 0x400 = 400, DF flag is set
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=000000000F636810 Handler2=000000000EC45A40
RDI=00000000151D6630 RSI=00000000151D6628 RAX=0000000000000001 RBX=0000000015265BD8
RCX=000000000000000F RDX=000000001527E8F0 R8=00000000151D6598 R9=0000000000000022
R10=0000000000000000 R11=0000000015298000 R12=0000000000000022 R13=0000000000000110
R14=000000001527E8F0 R15=0000000015298018
RIP=000000001E3CED94 GS=0000 FS=0000 RSP=00007FFAB31916C0
RFlags=0000000000010602 CS=002B RBP=00007FFAB3191B00 ERR=1142700000000000
TRAPNO=000000000000000D CPU=7000000000000000 FAULTVADDR=0000000111427000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 00007ffab3191770 (f: 3004765952.000000, d: 6.952231e-310)
XMM2 00007ffab3191770 (f: 3004765952.000000, d: 6.952231e-310)
XMM3 0000000000000016 (f: 22.000000, d: 1.086944e-322)
XMM4 0000000015298000 (f: 355041280.000000, d: 1.754137e-315)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fddfc859906d5b5 (f: 2567362048.000000, d: 4.685377e-01)
XMM7 402e7f9c1e980fa9 (f: 513281952.000000, d: 1.524924e+01)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=j9vm/test/arraycopy/MultiThreadedBackwardArrayCopyTest.run()V
Target=2_90_20210413_70 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/274 osx1011-x86-2 TestArrayCopy_openj9_none_SCC_3

https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk11_j9_sanity.functional_x86-64_mac_Nightly_testList_0/274/functional_test_output.tar.gz

EFlags 10602, EFlags & 0x400 = 400, DF flag is set
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=000000000EA362A0 Handler2=000000000E128A40
RDI=000000003069E0EE RSI=000000003069E0EC RAX=0000000000000013 RBX=0000000000000029
RCX=0000000000000002 RDX=0000000000000002 R8=0000000000000062 R9=0000000000000026
R10=0000000000000013 R11=0000000000000003 R12=00000000FFFFBEEF R13=0000000000000006
R14=0000000000000064 R15=0000000011081B48
RIP=00000000B02C5B81 GS=0000 FS=0000 RSP=0000000010DF6B00
RFlags=0000000000010602 CS=002B RBP=0000000010D0FF00 ERR=1142700000000000
TRAPNO=000000000000000D CPU=7000000000000000 FAULTVADDR=0000000111427000
XMM0 00000000ffffffff (f: 4294967296.000000, d: 2.121996e-314)
XMM1 0000000010df6a80 (f: 283077248.000000, d: 1.398587e-315)
XMM2 000000000000001b (f: 27.000000, d: 1.333977e-322)
XMM3 00000178da37bf01 (f: 3661086464.000000, d: 7.996792e-312)
XMM4 0000000010df6b9a (f: 283077536.000000, d: 1.398589e-315)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fe2c0e9ed448e8c (f: 3980693248.000000, d: 5.860490e-01)
XMM7 4018f40b5ed9812d (f: 1591312640.000000, d: 6.238325e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 ca62c1d6ca62c1d6 (f: 3395469824.000000, d: -2.193092e+50)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=j9vm/test/arrayCopy/ArrayCopyTest.testArrayOfShortsCopyBackward(I)V
Target=2_90_20210415_678 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton commented 3 years ago

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/291 osx1011-x86-2 J9vmTest_0 variation: Mode110 JVM_OPTIONS: -Xjit -Xgcpolicy:gencon -Xnocompressedrefs

https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/291/functional_test_output.tar.gz

+++ j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest: +++
command: /Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/java  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs  -Xdump -Xms64m -Xmx64m   -Xbootclasspath:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/jre/lib/default/jclSC180/vm.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/jre/lib/se-service.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/jre/lib/rt.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/jre/lib/resources.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/jre/lib/jsse.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/jre/lib/charsets.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/jre/lib/jce.jar -classpath /Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdk-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar:/Users/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdk-tests/TKG/../TKG/lib/asm-all.jar  j9vm.test.arraycopy.MultiThreadedBackwardArrayCopyTest 

EFlags 10602, EFlags & 0x400 = 400, DF flag is set
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=000000000CA36250 Handler2=000000000C176AA0
RDI=000000001269CAA8 RSI=000000001269CAA0 RAX=000000001269CA10 RBX=0000000012701828
RCX=000000000000000F RDX=0000000000000020 R8=0000000000000000 R9=000000000000063E
R10=0000000012742B18 R11=0000000000000110 R12=000000001269CA10 R13=0000000012701828
R14=000000001269CA10 R15=0000000012751000
RIP=00000000130C4650 GS=0000 FS=0000 RSP=00007FC51302D6D0
RFlags=0000000000010602 CS=002B RBP=00007FC513029700 ERR=EBC6D00400000000
TRAPNO=000000000000000D CPU=D004000000000000 FAULTVADDR=FFFFFF80EBC6D004
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 00007fc51302d760 (f: 318953312.000000, d: 6.940852e-310)
XMM2 00000000df000000 (f: 3741319168.000000, d: 1.848457e-314)
XMM3 0000000000000009 (f: 9.000000, d: 4.446591e-323)
XMM4 0000000012751000 (f: 309661696.000000, d: 1.529932e-315)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 3fe0f7241c9b497d (f: 479938944.000000, d: 5.301686e-01)
XMM7 402e7f9c1e980fa9 (f: 513281952.000000, d: 1.524924e+01)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=j9vm/test/arraycopy/MultiThreadedBackwardArrayCopyTest.rotateSource()V
Target=2_90_20210419_691 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------