Closed JasonFengJ9 closed 4 years ago
@mpirvu fyi
I was able to reproduce it locally, so will look into this.
Hm, not sure if what I reproduced is the same bug. When I run the test locally, I can't get it to segfault unless I run the client in gdb. But when ran gdb, it segfaults even in non-JITServer mode. When I run it on grinder, non-JITServer didn't segfault over 20 iterations, while JITServer has 1/10 failure rate. In gdb, both segfault every time. Not sure if running it on gdb catches the same bug, or even if it's a bug at all. Will continue investigating.
I narrowed down the segfault to a single method: "java/lang/invoke/AsTypeHandle.invokeExact_thunkArchetype_X(Lnet/adoptopenjdk/test/lang/invoke/AsTypeTest;Ljava/lang/Object;)I
.
This is the part of the generated code that segfaults: At instruction selection:
n276n ( 0) NULLCHK on n26n [#32] [0x7fffdb0d4600] bci=[6,1,477] rc=0 vc=819 vn=- li=27 udi=- nc=1
n275n ( 0) PassThrough [0x7fffdb0d45b0] bci=[6,1,477] rc=0 vc=819 vn=- li=27 udi=24976 nc=1
n26n ( 2) aload <parm 2 Ljava/lang/Object;>[#359 Parm] [flags 0xc0000107 0x0 ] (in &GPR_0080) (X!=0 sharedMemory ) [0x7fffdb004d00] bci=[-1,28,49] rc=2 vc=819 vn=- li=27 udi=24976 nc=0 flg=0x
4
------------------------------
[0x7fffdb226290] mov &GPR_0080, qword ptr [vfp+0x10] # L8RegMem, SymRef <parm 2 Ljava/lang/Object;>[#461 Parm] [flags 0xc0000107 0x0 ]
[0x7fffdb2263e0] test byte ptr [&GPR_0080], 0x00 # TEST1MemImm1
Segfault happens when executing test
instruction.
So, mov
moves the address of the second parameter (java/lang/Object
) into a register,
and then test
checks whether the value at the address inside the register is 0.
However, shouldn't test
be checking whether the address of the class inside the register is 0, instead of the value? In other words, shouldn't it be a reg-immediate test, instead of mem-immediate one? What appears to be happening is that when class we are NULL checking is actually NULL, test
instruction tries to dereference that address, resulting in a segfault.
This is the part of the code generating above instructions.
@mpirvu could I get your opinion on this? I'm not very familiar with codegen so not sure if my assessment of what's happening is correct.
@dmitry-ten Do you have the entire compilation log? If so could you please send it to me?
What appears to be happening is that when class we are NULL checking is actually NULL, test instruction tries to dereference that address, resulting in a segfault.
I am not entirely sure, but this may be how we implement NULCHK. We have an object and instead of testing the object itself we dereference it. If a segfault is triggered we catch it and throw the null pointer exception we are supposed to. This is also why you are not seeing any problem when running without gdb. This feature can be disabled with the following option
{"noResumableTrapHandler", "C\tdo not generate traps for exception detections",
SET_OPTION_BIT(TR_NoResumableTrapHandler), "F" },
Oh, I see. Thanks fro the explanation. Hm, then I'll need to find some way to reproduce the actual failure.
Can't seem to reproduce the crash with the latest nightly build anymore. Ran the test 30 times, no crashes.
A similar Segmentation error vmState=0x0005ffff at Test_openjdk8_j9_sanity.openjdk_x86-64_linux_jit_Nightly/41
22:39:48 openjdk version "1.8.0_252-internal"
22:39:48 OpenJDK Runtime Environment (build 1.8.0_252-internal-jenkins_2020_04_06_18_43-b00)
22:39:48 Eclipse OpenJ9 VM (build ibm_sdk-934063f45c, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20200406_42 (JIT enabled, AOT enabled)
22:39:48 OpenJ9 - 934063f45c
22:39:48 OMR - b7eb64f76
22:39:48 JCL - 61d22079a86 based on jdk8u252-b08)
22:50:46 ===============================================
22:50:46 Running test jdk_util_j9_0 ...
22:50:46 ===============================================
22:52:15 Unhandled exception
22:52:15 Type=Segmentation error vmState=0x0005ffff
22:52:15 J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
22:52:15 Handler1=00007F140F0F51C0 Handler2=00007F140E9E69F0 InaccessibleAddress=00007FB3DD3E8D68
22:52:15 RDI=00007FB3DD3E8D28 RSI=0000000000000002 RAX=00007FB3DD3E8D28 RBX=00007FB3DD3E8D28
22:52:15 RCX=00007FB3DD3E8D68 RDX=0000000000000001 R8=00007F1408070A58 R9=0000000000000000
22:52:15 R10=0000000000000000 R11=00000000FFFFFFF7 R12=00007F13EBFFB9C8 R13=00007F140D8A48EF
22:52:15 R14=00007F14011F84A0 R15=00007F140D8A60DB
22:52:15 RIP=00007F140D1A0D40 GS=0000 FS=0000 RSP=00007F13EBFFB1E0
22:52:15 EFlags=0000000000010206 CS=0033 RBP=00007F13E8946B50 ERR=0000000000000004
22:52:15 TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=00007FB3DD3E8D68
22:52:15 xmm0 0000003500000035 (f: 53.000000, d: 1.124658e-312)
22:52:15 xmm1 2525252525252525 (f: 623191360.000000, d: 9.532824e-130)
22:52:15 xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:52:15 xmm3 000000ff00000000 (f: 0.000000, d: 5.411089e-312)
22:52:15 xmm4 4040404040404040 (f: 1077952512.000000, d: 3.250196e+01)
22:52:15 xmm5 5b5b5b5b5b5b5b5b (f: 1532713856.000000, d: 1.213625e+132)
22:52:15 xmm6 2020202020202020 (f: 538976256.000000, d: 6.013470e-154)
22:52:15 xmm7 0000002000000000 (f: 0.000000, d: 6.790387e-313)
22:52:15 xmm8 000000ff00000000 (f: 0.000000, d: 5.411089e-312)
22:52:15 xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:52:15 xmm10 ff000000000000ff (f: 255.000000, d: -5.486124e+303)
22:52:15 xmm11 0000000049d70a38 (f: 1238829568.000000, d: 6.120632e-315)
22:52:15 xmm12 000000004689a022 (f: 1183424512.000000, d: 5.846894e-315)
22:52:15 xmm13 0000000047ac082f (f: 1202456576.000000, d: 5.940925e-315)
22:52:15 xmm14 0000000048650dc0 (f: 1214582272.000000, d: 6.000833e-315)
22:52:15 xmm15 0000000046b73e38 (f: 1186414080.000000, d: 5.861665e-315)
22:52:15 Module=/home/jenkins/workspace/Test_openjdk8_j9_sanity.openjdk_x86-64_linux_jit_Nightly/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so
22:52:15 Module_base_address=00007F140CF3C000
22:52:15
22:52:15 Method_being_compiled=java/util/regex/Pattern.RemoveQEQuoting()V
22:52:15 Target=2_90_20200406_42 (Linux 2.6.32-754.27.1.el6.x86_64)
22:52:15 CPU=amd64 (8 logical CPUs) (0x1f409e000 RAM)
22:52:15 ----------- Stack Backtrace -----------
22:52:15 (0x00007F140D1A0D40 [libj9jit29.so+0x264d40])
22:52:15 (0x00007F140D2A828C [libj9jit29.so+0x36c28c])
22:52:15 (0x00007F140D2A8311 [libj9jit29.so+0x36c311])
22:52:15 (0x00007F140D089177 [libj9jit29.so+0x14d177])
22:52:15 (0x00007F140D08C582 [libj9jit29.so+0x150582])
22:52:15 (0x00007F140D08D161 [libj9jit29.so+0x151161])
22:52:15 (0x00007F140E9E7753 [libj9prt29.so+0x1b753])
22:52:15 (0x00007F140D08EE55 [libj9jit29.so+0x152e55])
22:52:15 (0x00007F140D08F408 [libj9jit29.so+0x153408])
22:52:15 (0x00007F140D08ADEB [libj9jit29.so+0x14edeb])
22:52:15 (0x00007F140D08B2E2 [libj9jit29.so+0x14f2e2])
22:52:15 (0x00007F140D08B38A [libj9jit29.so+0x14f38a])
22:52:15 (0x00007F140E9E7753 [libj9prt29.so+0x1b753])
22:52:15 (0x00007F140D08B7E4 [libj9jit29.so+0x14f7e4])
22:52:15 (0x00007F140EE55326 [libj9thr29.so+0xe326])
22:52:15 (0x00007F1410E80AA1 [libpthread.so.0+0x7aa1])
22:52:15 clone+0x6d (0x00007F14107B1C4D [libc.so.6+0xe8c4d])
22:52:15 ---------------------------------------
For example, to rebuild the failed tests in
I was able to reproduce the failure jdk_util_j9_0
with a debug build.
The segfault occurs in J9Profiler code @mpirvu do we support JProfiler for JITServer?
If we need to support it, then we should be sending some additional method recompilation info to the server.
No, we don't support JProfiler just yet. That work is in progress. Until then we should make it explicit by disabling JProfiler at the client with an option.
So, disabling JProfiler wasn't enough to make jdk_util_j9_0
pass. I also had to prevent jit profiling information from being collected, since it's stored in TR_PersistentProfileInfo
records inside TR_PersistentMethodInfo
but they are not currently transferred from client to server and vice versa.
However, I'm still getting segfaults, but the current ones come from GC with the following stacktrace:
#11 0x00007f0fedbc649a in masterSynchSignalHandler ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9prt29.so
#12 <signal handler called>
#13 0x00007f0fec929d4d in GC_ObjectModelDelegate::calculateObjectDetailsForCopy(MM_EnvironmentBase*, MM_ForwardedHeader*, unsigned long*, unsigned long*, unsigned long*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#14 0x00007f0fec8afb49 in MM_Scavenger::incrementalScanCacheBySlot(MM_EnvironmentStandard*, MM_CopyScanCacheStandard*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#15 0x00007f0fec8b272d in MM_Scavenger::completeScan(MM_EnvironmentStandard*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#16 0x00007f0fec8b4933 in MM_Scavenger::workThreadGarbageCollect(MM_EnvironmentStandard*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#17 0x00007f0fec850cf3 in MM_Dispatcher::run(MM_EnvironmentBase*, MM_Task*, unsigned long) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#18 0x00007f0fec88a429 in MM_Scavenger::scavenge(MM_EnvironmentBase*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#19 0x00007f0fec8b96c5 in MM_Scavenger::masterThreadGarbageCollect(MM_EnvironmentBase*, MM_AllocateDescription*, bool, bool) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#20 0x00007f0fec8bb5f1 in MM_Scavenger::internalGarbageCollect(MM_EnvironmentBase*, MM_MemorySubSpace*, MM_AllocateDescription*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#21 0x00007f0fec827cb7 in MM_Collector::garbageCollect(MM_EnvironmentBase*, MM_MemorySubSpace*, MM_AllocateDescription*, unsigned int, MM_ObjectAllocationInterface*, MM_MemorySubSpace*, MM_AllocationContext*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#22 0x00007f0fec90464e in MM_MemorySubSpaceSemiSpace::allocationRequestFailed(MM_EnvironmentBase*, MM_AllocateDescription*, MM_MemorySubSpace::AllocationType, MM_ObjectAllocationInterface*, MM_MemorySubSpace*, MM_MemorySubSpace*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#23 0x00007f0fec91f81c in MM_MemorySubSpaceGeneric::allocateObject(MM_EnvironmentBase*, MM_AllocateDescription*, MM_MemorySubSpace*, MM_MemorySubSpace*, bool) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#24 0x00007f0fec860637 in MM_TLHAllocationInterface::allocateObject(MM_EnvironmentBase*, MM_AllocateDescription*, MM_MemorySpace*, bool)
()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#25 0x00007f0fec87d40d in OMR_GC_AllocateObject(OMR_VMThread*, MM_AllocateInitialization*) ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#26 0x00007f0fec74e3df in J9AllocateIndexableObject ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9gc29.so
#27 0x00007f0fee7f7464 in VM_BytecodeInterpreter::run(J9VMThread*)
()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9vm29.so
#28 0x00007f0fee7e3930 in bytecodeLoop ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9vm29.so
#29 0x00007f0fee8a5d72 in c_cInterpreter ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9vm29.so
#30 0x00007f0fee8574ce in sidecarInvokeReflectMethodImpl ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9vm29.so
#31 0x00007f0fee8582c5 in sidecarInvokeReflectMethod ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9vm29.so
#32 0x00007f0fe5bc5153 in JVM_InvokeMethod_Impl ()
from /root/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libjclse29.so
#33 0x00007f0fd2e499ef in ?? ()
#34 0x0000000000139000 in ?? ()
#35 0x0000000000139000 in ?? ()
#36 0x0000000000139000 in ?? ()
#37 0x00007f0fd2426420 in ?? ()
#38 0x00000000ff9993b8 in ?? ()
#39 0x00000000e0085920 in ?? ()
#40 0x00000000e002d5e0 in ?? ()
#41 0x0000000000000005 in ?? ()
#42 0x00000000ff9993b8 in ?? ()
#43 0x00007f0fd2426260 in ?? ()
#44 0x00007f0fd2426340 in ?? ()
Not sure if it's related to the same problem, or if it's an entirely different bug.
Could that be an instance of #9221?
Did 100 runs with the current code, everything passed. Seems that one of the fixes to other bugs fixed this one as well. @mpirvu This can probably be closed now.
Closing based on the above message
Failure link
From an internal build
Test_openjdk8_j9_extended.system_x86-64_linux_jit_Nightly/32
Optional info
Failure output (captured from console output)
To rebuild the failed tests in=https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder, use the following links: https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/parambuild/?JDK_VERSION=8&JDK_IMPL=openj9&BUILD_LIST=system&PLATFORM=x86-64_linux&TARGET=ObjectTreeLoadTest_0