Segmentation Error When compiling with debug enabled on PPC

[AlenBadel]

This only occurs when building on PPCLE, and with java versions 10,11. Java 8 builds perfectly.

This occurs during the DDR stage, when it uses the built jdk.

Passes with -Xint.

To reproduce,

I checked out the OpenJDK 11 with OpenJ9 project per https://www.eclipse.org/openj9/oj9_build.html

Enabled debugging per https://blog.openj9.org/2018/06/05/debugging-openj9-in-docker-with-gdb/

export UMA_DO_NOT_OPTIMIZE_CCODE="1"
export VMDEBUG="-g3 -fno-inline -O0"
export VMLINK="-g -O0"
export enable_debug=yes
export enable_optimized=no
export enable_optimize=no
export CXXFLAGS="-O0 -g3"
export CFLAGS="-O0 -g3"
export BUILD_CONFIG=debug

To build:

cd /projects/openj9/openj9-openjdk-jdk11/
bash get_source.sh 
bash ./configure --with-freemarker-jar=/projects/freemarker.jar 
make all

We get the following error

Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000071479F7829A0 Handler2=000071479F417384
R0=0000714768CC06EC R1=00007147A01CAD30 R2=00007147A0760E28 R3=0000000000078200
R4=000000000007A100 R5=0000000000000001 R6=000000000007A100 R7=0000000000000000
R8=0000000000000001 R9=0000000000078200 R10=0000000000000000 R11=000071479EA25658
R12=00007147A0770E28 R13=00007147A01D68E0 R14=0000000000044220 R15=0000000000044500
R16=0000714768250038 R17=FFFFFFFFFFFFFFFF R18=000071479826FD5C R19=000071479800DCD0
R20=000071479826FD5C R21=0000714768032754 R22=00007147680304C0 R23=0000000000000000
R24=00007147A01CAFA0 R25=0000000000074E00 R26=00007147980B93A8 R27=0000714798006BD0
R28=0000000000000000 R29=00007147A01CAFA0 R30=00007147A01CB1B0 R31=00007147A01CAD30
NIP=000071479DCD7B10 MSR=900000010280F033 ORIG_GPR3=000071479EA256B0 CTR=000071479EA25658
LINK=000071479EA256B4 XER=0000000000000000 CCR=0000000042002844 SOFTE=0000000000000001
TRAP=0000000000000300 DAR=00007147A0776D48 dsisr=0000000040000000 RESULT=0000000000000000
FPR0 405564fe0be28b4b (f: 199396176.000000, d: 8.557801e+01)
FPR1 402f34977fe9b99d (f: 2146023808.000000, d: 1.560272e+01)
FPR2 402fe2804e87b080 (f: 1317515392.000000, d: 1.594239e+01)
FPR3 3bbcc86800000000 (f: 0.000000, d: 6.095003e-21)
FPR4 3c70400000000000 (f: 0.000000, d: 1.409463e-17)
FPR5 402f3498394dc17c (f: 961397120.000000, d: 1.560272e+01)
FPR6 3cd0000000000000 (f: 0.000000, d: 8.881784e-16)
FPR7 3d2ef35793c76730 (f: 2479318784.000000, d: 5.497923e-14)
FPR8 3d76430d991a1bfe (f: 2568625152.000000, d: 1.265444e-12)
FPR9 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
FPR10 3ff678b7679459b8 (f: 1737775488.000000, d: 1.404472e+00)
FPR11 bed0800000000000 (f: 0.000000, d: -3.933907e-06)
FPR12 402f34977fe9b99d (f: 2146023808.000000, d: 1.560272e+01)
FPR13 4037000000000000 (f: 0.000000, d: 2.300000e+01)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/home/abadel/break/jdk/lib/compressedrefs/libj9jit29.so
Module_base_address=000071479D760000
Target=2_90_20180912_000000 (Linux 4.15.0-33-generic)
CPU=ppc64le (160 logical CPUs) (0x7fb6230000 RAM)
----------- Stack Backtrace -----------
(0x000071479DCD7B10 [libj9jit29.so+0x577b10])
(0x0000714768CC06EC [<unknown>+0x0])
(0x000071479F762A04 [libj9vm29.so+0x112a04])
(0x000071479F6C1244 [libj9vm29.so+0x71244])
(0x000071479F6C387C [libj9vm29.so+0x7387c])
(0x000071479F6C237C [libj9vm29.so+0x7237c])
(0x000071479F7CBA24 [libj9vm29.so+0x17ba24])
(0x000071479F7CBBAC [libj9vm29.so+0x17bbac])
(0x000071479F6793C8 [libj9vm29.so+0x293c8])
(0x000071479F693874 [libj9vm29.so+0x43874])
(0x000071479F666C78 [libj9vm29.so+0x16c78])
(0x000071479F7C12F4 [libj9vm29.so+0x1712f4])
(0x000071479F762A04 [libj9vm29.so+0x112a04])
(0x000071479F6C1244 [libj9vm29.so+0x71244])
(0x000071479F6C387C [libj9vm29.so+0x7387c])
(0x000071479F6C237C [libj9vm29.so+0x7237c])
(0x000071479CCE1D6C [libjclse11_29.so+0x31d6c])
(0x000071479CD1AC10 [libjclse11_29.so+0x6ac10])
(0x000071479CD2E5B4 [libjclse11_29.so+0x7e5b4])
J9VMDllMain+0x288 (0x000071479CD2E8A8 [libjclse11_29.so+0x7e8a8])
(0x000071479F7AB5DC [libj9vm29.so+0x15b5dc])
(0x000071479F8B0DD8 [libj9vm29.so+0x260dd8])
(0x000071479F7AB2B0 [libj9vm29.so+0x15b2b0])
(0x000071479F7AF964 [libj9vm29.so+0x15f964])
(0x000071479F415488 [libj9prt29.so+0x35488])
(0x000071479F7A2EE0 [libj9vm29.so+0x152ee0])
J9_CreateJavaVM+0xe0 (0x000071479F796B20 [libj9vm29.so+0x146b20])
JNI_CreateJavaVM+0xc08 (0x000071479F96613C [libjvm.so+0x2613c])
JNI_CreateJavaVM+0x2ec (0x00007147A01E57CC [libjvm.so+0x157cc])
(0x00007147A0529098 [libjli.so+0x9098])
(0x00007147A0524AF8 [libjli.so+0x4af8])
(0x00007147A021885C [libpthread.so.0+0x885c])
clone+0x98 (0x00007147A0439028 [libc.so.6+0x159028])
---------------------------------------

[AlenBadel]

There a number of methods that this is reproducible with.

To name a few:

java/util/HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object
java/util/HashMap.resize()[Ljava/util/HashMap$Node
java/lang/Object.<init>()V

It can be reproduced with simply

java -version

This fails with the following options:

optLevel=noOpt
lastOptIndex=0
disableInlining
count=0

Running through this in gdb, during the transition from execution within the jit compiled method, and fast_jitInstanceOf the TOC register r2 is attempted to be populated by register r12.

0x7ffff64376e4 <fast_jitInstanceOf(J9VMThread*, J9Class*, j9object_t)>          addis   r2,r12,60                                                      │*     
   │0x7ffff64376e8 <fast_jitInstanceOf(J9VMThread*, J9Class*, j9object_t)+4>        addi    r2,r2,-19940                                                   │*

The address in r12 before these two instructions is not obvious to me, but it is a valid address. After the two instructions above, an invalid (assumed TOC) address is fed into register r2.

As execution eventually reaches inlineCheckCast we get the exception when the system tries loading from an invalid address inside of register r2.

Backtrace

#0  0x00007ffff56e869c in 00000017.plt_call._ZN12VM_VMHelpers15inlineCheckCastEP7J9ClassS1_b () from /home/abadel/break/new/jdk/lib/compressedrefs/libj9jit29.so
#1  0x00007ffff6437740 in fast_jitInstanceOf (currentThread=0x44500,
    castClass=0xa5600, object=0x7ffe03870) at cnathelp.cpp:2676
#2  0x00007fffc03607c4 in ?? ()
#3  0x00007ffff7182e24 in sendClinit (
    currentThread=<error reading variable: Cannot access memory at address 0x88>,
    clazz=<error reading variable: Cannot access memory at address 0x80>,
    reserved1=<error reading variable: Cannot access memory at address 0x78>,
    reserved2=<error reading variable: Cannot access memory at address 0x70>,
    reserved3=<error reading variable: Cannot access memory at address 0x68>)
    at callin.cpp:419

**There's a fair bit of missing debug data in this debug build.

[gita-omr]

@AlenBadel will add an update shortly, so I am jumping slightly ahead. I wanted to ask @gacholio : what is the convention between JITted code and the helpers in cnathelp.cpp in terms of setting the TOC?

[gacholio]

The C code follows the C calling convention of the platform - there are no exceptions. The wrapper helpers handle this. If the C code is called directly from the compiled code, you will need to follow the appropriate convention there.

[gacholio]

See:

https://github.com/eclipse/openj9/blob/master/runtime/oti/phelpers.m4#L269

[gita-omr]

@gacholio @zl-wang, apparently with -O0 (@AlenBadel is verifying) gcc++ creates this at the beginning of external methods:

0x7ffff64376e4 <fast_jitInstanceOf(J9VMThread, J9Class, j9object_t)> addis r2,r12,60 │
│0x7ffff64376e8 <fast_jitInstanceOf(J9VMThread, J9Class*, j9object_t)+4> addi r2,r2,-19940

JIT definitely does not set R12 to be the entry point. How should the methods in cnathelp.cpp get the right TOC then?

[gita-omr]

My feeling is we are the first ones who are trying to build DEBUG VM on PPC LE :)

[zl-wang]

No mystery here. JIT-ed code call-able helpers: the TOC should be gotten by loading from J9VMThread->jitTOC. That has been the case for: AIX32/64, BE Linux64, and LE Linux. Suppose there is such a load ... I don't see why the global-entry pre-prologue (addis/addi based off r12) can be harmful, i.e. even if r2 is set to garbage by the pre-prologue, it will be corrected by the load mentioned above (before TOC is used in that routine).

[zl-wang]

So, is this issue due to: in non-debug build, JIT-ed code calls the wrapper containing the load; in debug build, JIT-ed code calls the C routine directly? JIT-ed code doesn't set up either r12 or r2 definitely. I just cannot see how this comes by.

[gita-omr]

See:

https://github.com/eclipse/openj9/blob/master/runtime/oti/phelpers.m4#L269

Sorry, I missed where that CALL_INDIRECT macro is used. From the earlier GAC's comment it sounded like it's JIT responsibility to set R2 if calling directly from JITtecd code. But then global entry would override that.

[zl-wang]

GAC comment to me: JIT-ed code should call the wrapper as we did always. If you call the C code directly, you need to set up r12.

[zl-wang]

So, it is not an issue of production build vs. debug build. It is an issue which helper target you call. As it is, JIT-ed code doesn't set up either r12 or r2 (for helper calls... we do set up r12 or r2 for JNI calls). If the target is wrong, it will not work for both production and debug builds.

[gita-omr]

Sure, debug build just exposed some confusion in the calling conventions for helper calls. We need to figure out what they actually should be and what is the correct target.

[AlenBadel]

I can verify that the inclusion of the -O0 option directly causes this issue. Either when export UMA_DO_NOT_OPTIMIZE_CCODE="1" is set or -O0 is included in VMDEBUG.

Given that UMA_DO_NOT_OPTIMIZE_CCODE expands to

 UMA_OPTIMIZATION_CFLAGS += -O0
 UMA_OPTIMIZATION_CXXFLAGS += -O0

Earlier I thought that this was only seen in Java10, and Java11 builds however I can confirm it spans to all debug builds on Linux PPCLE including Java 8. With Java8 it's not obvious during the build, but is reproducible with setting -Xjit=count=0 and running java -version

I can also confirm that the register values of r12, and r2 don't correspond to values of the entry point address, nor do they correspond to the address of the jitTOC respectively. I checked before the direct call to getInstanceOf, at the beginning of getInstanceOf execution and right before the segmentation.

As Gita has mentioned, r12 and r2 aren't used much in production builds and I wasn't able to find any instances where they would used in a way as they contained an expected value, or they're being used as a TOC.

Even in the method that is called inside getInstanceOf, VMInlineCast assumes the data while there's no trace of such instructions on production.

0x7ffff56e8694 <00000017.plt_call._ZN12VM_VMHelpers15inlineCheckCastEP7J9ClassS1_b>             std     r2,24(r1)                           │
   │0x7ffff56e8698 <00000017.plt_call._ZN12VM_VMHelpers15inlineCheckCastEP7J9ClassS1_b+4>           addis   r12,r2,1                            │
   │0x7ffff56e869c <00000017.plt_call._ZN12VM_VMHelpers15inlineCheckCastEP7J9ClassS1_b+8>           ld      r12,24312(r12)                      │

[gacholio]

Is the compiled code loading r2 before calling the helper? The regular helpers load it before every call to the C code (INIT_GOT macro).

[gita-omr]

Last night Alen stepped through all the way from JITted code to fast_jitInstanceIOff in cnathelp.cpp and he did not see r2 set anywhere.

When we call other helpers via PicBuilder.m4 we do set r2.

Nobody seems to ever set r12.

[gacholio]

So perhaps the fast path from the compiled code is always incorrect, and we get lucky in production builds. r2 is set in pnathelp.m4 for the normal helpers.

[gita-omr]

Yes, looks like that from the Alen’s Investigation above.

What about setting r12? Or we should somehow jump to the local entry?

[gacholio]

In the macros, r12 is used as the function pointer for indirect calls, nothing else. I don't recall if that was necessary, or simply conveinient.

define({FUNC_PTR},{r12})

define({CALL_INDIRECT},{
    ifelse($1,,,{mr FUNC_PTR,$1})
    std r2,TOC_SAVE_OFFSET(r1)
    mtctr r12
    bctrl
    ld r2,TOC_SAVE_OFFSET(r1)
})

This assumes that r2 is already loaded with the jitTOC.

[zl-wang]

The other way around, GAC. as long as it is the right target of helper for JIT-ed code, it is the helper's responsibility to restore the TOC if it is needed. That has been the reason we initialized vm->jitTOC (and consequentially J9VMThread->jitTOC).

[gacholio]

@zl-wang I think you're missing the point. We're talking about calling the C code directly from the compiled code, not going via the helper (which does all the TOC work).

[gacholio]

In general, the helpers don't use the indirect call - on LE the functions are called directly.

define({CALL_DIRECT},{
    ori r0,r0,0
    bl BRANCH_SYMBOL($1)
    ori r0,r0,0
})

BRANCH_SYMBOL does no decoration. I don't recall why the call is surrounded by the NOPs.

[zl-wang]

For LE, that (r12 setting) is the requirement of the ABI definition. (outside module) caller sets r12 to the callee global entry ... (shared lib) loader relocates TOC base out of it.

[zl-wang]

If JIT-ed code calls C code directly (JNI code in essence), we do set r12 (on LE) to the global entry.

[zl-wang]

where does JIT-ed code call C code directly except JNI targets? that is the question to be answered.

[zl-wang]

no matter the actual call is made through bl or bctrl, the ABI definition needs to be fulfilled (AIX glink or LE plink or something code for bl, for example). i guessed we are mixing up the meaning of direct/indirect call here. I referred to call-directly as meaning JIT-ed code seeing it as a target (instead of meaning it through bl).

[zl-wang]

i can at least see why you have the second nop: to restore TOC after return. maybe the first nop as well: to save the TOC before call. Typically, the save is done in the glink code (AIX or BE Linux64) or plt_link code (LE).

[zl-wang]

for example, you call read() in your code. the compiler generates bl to read()'s glink or plt_link inside your module.

[gacholio]

Some of the C helpers (in particular instanceof) are called directly.

[gacholio]

Sounds like the JIT will not need to load r12 for helper calls, if r2 is loaded - we're not using the linker/loader, so the target C code will simply expect r2 to be loaded. If you're going to generate a bctrl instead of a bl, r12 may as well be used to load CTR for complete safety.

[gita-omr]

Well, looks like if compiled with -O0 target code expects r12 to be set, from which it derives r2. Copying from above:

0x7ffff64376e4 <fast_jitInstanceOf(J9VMThread, J9Class, j9object_t)> addis r2,r12,60 │
│0x7ffff64376e8 <fast_jitInstanceOf(J9VMThread, J9Class*, j9object_t)+4> addi r2,r2,-199

[zl-wang]

I guessed this issue existed on AIX and BE Linux64 as well (TOC requirement). We have been just lucky so far switching to call the C target(s) directly (instead of the helper wrapper target(s)). Need to identify the set of C helpers called directly.

[zl-wang]

Yes, for LE: ABI requires r12 to point at the callee global entry. At the global entry, two instructions are used to calculate the TOC-base off r12. 60/-199 above are filled in resulting from loader relocations.

[gacholio]

I've actually been thinking for a while that I'd like to switch the direct calls in the helper wrappers to indirect calls, which would have the side-effect of loading r12. I expect that means I can stop loading r2, and we could likely eliminate the jitTOC entirely.

[zl-wang]

there are many uses of jitTOC in JIT runtime assembly (picBuilder, Crypto, arrayCopy, and recompilation etc etc).

[zl-wang]

AIX and BE Linux64 ABI(s): they did require caller set up r2 before call. But we didn't do that for general helper calls. GAC/You set up r2 instead in helper with loading from jitTOC.

LE Linux ABI: it requires caller to set up r12 for callee global-entry address. r2 is calculated from r12. We didn't do that either. It is assumed you set up r2 in helper with loading from jitTOC as well, if TOC is needed.

[ymanton]

You'll probably want to make a change in this method, which is where most of the work for calls is done:

https://github.com/eclipse/openj9/blob/450399bbd37af823f4360f033ac0e48dbdcd1d6c/runtime/compiler/p/codegen/PPCPrivateLinkage.cpp#L1540

In that method we kill r12; you'll need to change that to put the entry point in r12 instead. The linkage parameter tells you what you're generating code for (one of TR_Private, TR_Helper, TR_CHelper) so you can check for TR_CHelper. Alternatively you can put the TOC in r2 and call the local entry point if you want I guess. Not sure which is preferable, but doing it the global entry way probably makes the code easier to understand for observers.

https://github.com/eclipse/openj9/blob/450399bbd37af823f4360f033ac0e48dbdcd1d6c/runtime/compiler/p/codegen/PPCPrivateLinkage.cpp#L1977-L1980

[gacholio]

Doing the same thing as JNI is probably the right answer (loading r12 and using it to call indirect). This is what I'm attempting in the helpers.

[ymanton]

Now that I think about it this is going to end up being slightly inefficient because later on in the compilation process we'll be able to decide if the helper is reachable via a direct call or not, and if not we'll redirect the call through a trampoline which will correctly load r12. By also loading r12 during argument setup we'll end up doing it twice if we end up going through a trampoline. A bit of a shame.

That's likely why this bug hasn't been seen until now, the helper needs to end up being reachable via a direct call and has to make use of the TOC and the C compiler has to not optimize the usage away.

[zl-wang]

How about r2 on AIX and BE Linux64 for TR_CHelper?

[ymanton]

You are right, r2 is not set up either and needs to be.

Maybe we can remove r2/r12 setup from the trampolines for C helpers and assume jitted code will always do it, that way we don't pay the penalty twice.

[zl-wang]

No, we didn't do anything about r2 in trampoline. If it is decided to use bctrl for the call, then it will not go through trampoline anymore.

[zl-wang]

The drawback of course is target address prediction. bctrl is predicted by the countCache; while bl doesn't need prediction.

[gacholio]

The reason the C calls from the helper wrappers work without loading r12 today is that the linker is assuming that r2 is already loaded for calls within the same shared library (note the +8 which the linker added below, skipping the r2 load from r12):

   0x3fffb6671ad8 <jitInstanceOf+136>:  nop
   0x3fffb6671adc <jitInstanceOf+140>:  bl      0x3fffb6659858 <old_fast_jitInstanceOf+8>
   0x3fffb6671ae0 <jitInstanceOf+144>:  nop

In pure C code, the compiler also seems to be assuming that any calls using bl (i.e. within the same shared library) will return with r2 pointing to the TOC (it does not save/restore r2 across the call, and generates code assuming r2 is still the caller TOC, which is necessarily the same for all functions in the shared library).

   0x3fffb70e604c <sendClinit+524>: mr      r3,r29
   0x3fffb70e6050 <sendClinit+528>: bl      0x3fffb713caf8 <c_cInterpreter>
   0x3fffb70e6054 <sendClinit+532>: nop

I'm not sure why the C compiler isn't doing the +8 trick with the above call (update: it's because I need a .localentry directive). I'll update the code to load the TOC from r12 on LE and add the directive to allow the optimized call.

The pre-NOP I'm generating seems unnecessary, but the C compiler is generating the post-NOP (perhaps to reload r2 if necessary after the call in some situations?).

So, for direct C helpers calls from the compiled code, either r12 must be loaded with the function address, or r2 must be loaded with the TOC and the call must be to helper+8.

So, the call sequences need to be:

AIX32/64 / Linux64 BE

load r2 with jitTOC
bl helper

Linux64 LE

load r2 with jitTOC
bl helper+8

OR

load r12 with helper
bl helper

Linux32

bl helper

This means the trampolines for C helper calls need not contain any loading of r2 or r12, though I suppose as they're doing an indirect branch anyway, they may as well use r12 on LE.

Given that the TOC setup in the C code is technically variable-sized, it's probably safer to load r12 rather than r2 on LE, and it leaves open the possibility of removing the inline load r12 if a trampoline is going to be used.

[gacholio]

@zl-wang While investigating all this, I came across:

https://github.com/eclipse/omr/blob/master/util/omrutil/unix/linux/ppc/64le/gettoc.s#L31

Should that not be addi rather than addis?

I'm basing this on what I see in the compiled code, and the doc here:

https://www.ibm.com/developerworks/library/l-improve-performance-openpower-abi/index.html

I bet this works because of the localentry directive, so those first two instructions are never executed, since this code is statically-linked with every module that uses it.

Fixing this here: https://github.com/eclipse/omr/pull/3014

[gacholio]

As part of the TOC optimization work I'm doing in the VM, the jitTOC will be stored in the TOC slot in the current C stack frame, so the JIT will have the option of encoding the r2 load on the old ABI platforms as either potentially multiple instructions to load the compile-time constant, or a single instruction to fetch it from r1 (the offset will be offsetof(J9CInterpreterStackFrame, currentTOC)).

[gacholio]

Ah, seems this is unnecessary as the jitTOC is already propagated into a slot in the J9VMThread which is also a single instruction fetch.

[zl-wang]

@gacholio Yes, that should be an addi.

[DanHeidinga]

FYi @AlenBadel @gita-omr I've tagged this for the 0.11.0 release as we're not calling the c code correctly and that may result in issues.

[DanHeidinga]

@AlenBadel @gita-omr Is this still an open issue? Will it close in time for the 0.11.0 release?

[gita-omr]

@AlenBadel is actively working on it. There are a few places that need to be checked. We are hoping for 0.11.0 release.