Verify that Java8/Java11 cross compilation works

[mpirvu]

For the remote/out-of-process compilation case we must ensure that a server based on OpenJDK11 can compile methods for a client running OpenJDK8 and viceversa.

[harryyu1994]

Investigating what's wrong with jitaas on java11. The client crashed.

[harryyu1994]

Limited to this one method that caused the crash on client: (when running java -version)

+ (warm) java/lang/ThreadLocal.setInitialValue()Ljava/lang/Object; @ 00007FC5A25DD340-00007FC5A25DD60E OrdinaryMethod - Q_SZ=0 Q_SZI=0 QW=6 j9m=0000000000EB2BB0 bcsz=50 sync remote compThread=0 CpuLoad=95%(11%avg) JvmCpu=25%

The java11 client didnt like something java11 server compiled.

crashing address points to a j2i virtual thunk

0x7efc8d311338 00000204                      b8 10 31 8d fc 7e 00 00        DQ  00007EFC8D3110B8                # j2i virtual thunk

[harryyu1994]

Server log file and client log file seems to have the same j2i thunk address 00007EFC8D3110B8. It probably means we didn't relocate the thunk properly.

[harryyu1994]

if (!useAotCompilation || TR::CompilationInfo::canRelocateMethod(compiler))
            {
            TR::compInfoPT->relocateThunks();
            }

looking at the relocateThunks() function

[harryyu1994]

1. Java8 client and Java11 server, Java8 client crashes in jitted code
2. Java8 server and Java11 client, Java8 server crashes, for this particular instance, it crashed when server tries to get declName from romCPBase().

   (gdb) frame 13
   #13 0x00007ff4793b30b1 in TR_ResolvedJ9JITaaSServerMethod::fieldOrStaticName (this=0x7ff476e18560, cpIndex=9, len=@0x7ff476e14f4c: 32756, trMemory=0x7ff476e18310, 
   kind=heapAlloc) at /root/Harry_JITaaS_Full_Build/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/vm/compiler/../compiler/env/j9methodServer.cpp:1082
   warning: Source file is more recent than executable.
   1082          J9UTF8 * declName = J9ROMCLASSREF_NAME((J9ROMClassRef *) (&romCPBase()[ref->classRefCPIndex]));
   (gdb) l
   1077       J9ROMFieldRef * ref = (J9ROMFieldRef *) (&romCPBase()[cpIndex]);
   1078       J9ROMNameAndSignature * nameAndSignature = J9ROMFIELDREF_NAMEANDSIGNATURE(ref);
   1079    
   1080       if (inROMClass(nameAndSignature))
   1081          {
   1082          J9UTF8 * declName = J9ROMCLASSREF_NAME((J9ROMClassRef *) (&romCPBase()[ref->classRefCPIndex]));
   1083          J9UTF8 * name = J9ROMNAMEANDSIGNATURE_NAME(nameAndSignature);
   1084          J9UTF8 * signature = J9ROMNAMEANDSIGNATURE_SIGNATURE(nameAndSignature);
   1085    
   1086          if (inROMClass(declName) && inROMClass(name) && inROMClass(signature))

[harryyu1994]

Java8 client and Java11 server:

n18n        icalli  java/lang/Object.hashCode()I[#358  virtual Method -32] [flags 0x500 0x0 ]  [0x7f4b4f0048c0] bci=[-1,9,339] rc=3 vc=0 vn=- li=- udi=- nc=2

Looks like we have an incorrect offset (-32), when tried with a java8 server, the offset is (-40)

n18n        icalli  java/lang/Object.hashCode()I[#358  virtual Method -40] [flags 0x500 0x0 ]  [0x7f471ecac8c0] bci=[-1,9,339] rc=3 vc=0 vn=- li=- udi=- nc=2

[harryyu1994]

(gdb) n
5444       return -(int32_t)(vTableSlot(cpIndex) - J9JIT_INTERP_VTABLE_OFFSET);
(gdb) p vTableSlot(cpIndex)
$1 = 392
(gdb) p J9JIT_INTERP_VTABLE_OFFSET
$2 = 352

on Java8, J9JIT_INTERP_VTABLE_OFFSET == 352 on Java11, J9JIT_INTERP_VTABLE_OFFSET == 360

define J9JIT_INTERP_VTABLE_OFFSET sizeof(J9Class)

[harryyu1994]

In summary, there are about 4 places we need to change if we don't change the J9JIT_INTERP_VTABLE_OFFSET definition.

1. 
int32_t TR_ResolvedJ9Method::virtualCallSelector(U_32 cpIndex)
   {
   return -(int32_t)(vTableSlot(cpIndex) - J9JIT_INTERP_VTABLE_OFFSET);
   }
^ need a JITaaS Server version of this method

2. 
TR_J9ByteCodeIlGenerator::runFEMacro(TR::SymbolReference *symRef)
^ needs to watch out, called in 2 places. 

3. 
U_32 TR_J9VMBase::virtualCallOffsetToVTableSlot(U_32 offset)

4. 
int32_t TR_J9VMBase::getVTableSlot(TR_OpaqueMethodBlock * mBlock, TR_OpaqueClassBlock * clazz)

5.
U_32 TR_ResolvedJ9Method::getResolvedInterfaceMethodOffset(TR_OpaqueClassBlock * classObject, I_32 cpIndex)
^ non issue, JITaaS Server doesn’t call this method

6.
TR_OpaqueMethodBlock * TR_RelocationRecordConstantPoolWithIndex::getAbstractMethodFromCP(TR_RelocationRuntime *reloRuntime, void *void_cp, int32_t cpIndex, TR_OpaqueMethodBlock *callerMethod)
^ non issue, Not called on server

[mpirvu]

I am favoring replacing the J9JIT_INTERP_VTABLE_OFFSET macro with a J9::VMEnv::getInterpreterVTableOffset() query that does

if (auto stream = TR::CompilationInfo::getStream())
   return TR::compInfoPT->getClientData()->getOrCacheVMInfo(stream)-> _interpreterVTableOffset;
else
   return sizeof(J9Class);

This way, we are protected against future uses of J9JIT_INTERP_VTABLE_OFFSET in the code.

[harryyu1994]

Above issue resolved by #6580 . New issue has come up:

Method_being_compiled=java/util/HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object;
Target=2_90_20190724_000000 (Linux 4.4.0-133-generic)
CPU=amd64 (8 logical CPUs) (0x1f2ec6000 RAM)
----------- Stack Backtrace -----------
(0x00007FED96CD830F [libj9jit29.so+0xd0f30f])
_Z19handleServerMessagePN9JITServer12ClientStreamEP7TR_J9VM+0x6d66 (0x00007FED96CDFC6E [libj9jit29.so+0xd16c6e])
_Z13remoteCompileP10J9VMThreadPN2TR11CompilationEP17TR_ResolvedMethodP8J9MethodRNS1_24IlGeneratorMethodDetailsEPNS1_28CompilationInfoPerThreadBaseE+0x91b (0x00007FED96CE6FED [libj9jit29.so+0xd1dfed])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0xe8e (0x00007FED96A926F0 [libj9jit29.so+0xac96f0])
_ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x273f (0x00007FED96A90933 [libj9jit29.so+0xac7933])
(0x00007FED9D1741E3 [libj9prt29.so+0x201e3])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x70d (0x00007FED96A8DD7F [libj9jit29.so+0xac4d7f])
_ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x4fd (0x00007FED96A84533 [libj9jit29.so+0xabb533])
_ZN2TR24CompilationInfoPerThread14processEntriesEv+0x247 (0x00007FED96A83729 [libj9jit29.so+0xaba729])
_ZN2TR24CompilationInfoPerThread3runEv+0x60 (0x00007FED96A82FCA [libj9jit29.so+0xab9fca])
_Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x1e6 (0x00007FED96A82D86 [libj9jit29.so+0xab9d86])
(0x00007FED9D1741E3 [libj9prt29.so+0x201e3])
_Z21compilationThreadProcPv+0x48d (0x00007FED96A82B44 [libj9jit29.so+0xab9b44])
(0x00007FED9DADF285 [libj9thr29.so+0xe285])
(0x00007FED9EF0A6BA [libpthread.so.0+0x76ba])
clone+0x6d (0x00007FED9F42B41D [libc.so.6+0x10741d])

investigating

crashed at char * data = utf8Data((J9UTF8*) ptr, len);

case MessageType::ResolvedMethod_getRemoteROMString:
         {
         auto recv = client->getRecvData<TR_ResolvedJ9Method *, size_t, std::string>();
         TR_ResolvedJ9Method *method = std::get<0>(recv);
         size_t offsetFromROMClass = std::get<1>(recv);
         std::string offsetsStr = std::get<2>(recv);
         size_t numOffsets = offsetsStr.size() / sizeof(size_t);
         size_t *offsets = (size_t*) &offsetsStr[0];
         uint8_t *ptr = (uint8_t*) method->romClassPtr() + offsetFromROMClass;
         for (size_t i = 0; i < numOffsets; i++)
            {
            size_t offset = offsets[i];
            ptr = ptr + offset + *(J9SRP*)(ptr + offset);
            }
         int32_t len;
         char * data = utf8Data((J9UTF8*) ptr, len);
         client->write(response, std::string(data, len));
         }