search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.27k stars 721 forks source link

Reduce JIT CPU consumption at the JITaaS client #3200

Open mpirvu opened 5 years ago

mpirvu commented 5 years ago

Ideally, the JIT CPU consumption at the JITaaS client should be very small because compilations are performed in another process. The reality is that the communication overhead is very high in terms of CPU cycles. Here's a perf profile of the lone compilation thread during an AcmeAir benchmark run obtained with: perf record -e cycles -c 200000 -o profile_client.txt --tid 4823 -- sleep 120

Breakdown per module

  49.92%  [kernel.kallsyms]         
  17.44%  libj9jit29.so
   9.62%  libc-2.23.so
   5.11%  libprotobuf.so.15.0.0
   4.00%  [nf_conntrack]

As shown above 50% of the CPU goes into Linux kernel to handle socket communication. The ticks in libc are mostly used for malloc/free used during communication (many of them used in protobuf layer)

libc-2.23.so
   2.81%  JIT Compilation  [.] malloc
   2.38%  JIT Compilation  [.] _int_free
   2.08%  JIT Compilation  [.] _int_malloc
   0.83%  JIT Compilation  [.] __memcpy_avx_unaligned
   0.78%  JIT Compilation  [.] free
   0.26%  JIT Compilation  [.] malloc_consolidate
   0.21%  JIT Compilation  [.] __strncmp_sse42
   0.06%  JIT Compilation  [.] __memset_avx2
   0.03%  JIT Compilation  [.] vfprintf
   0.01%  JIT Compilation  [.] __GI___inet_pton

Even the 17.44% in libj9jit29.so is dominated by routines used for communication

libj9jit29.so
   1.62%  JIT Compilation  [.] FlatPersistentClassInfo::classSize(TR_PersistentClassInfo*)  ==> Probably traversing the linked list of subclasses to get its length is the expensive part
   1.27%  JIT Compilation  [.] handleServerMessage(JITaaS::J9ClientStream*, TR_J9VM*)
   0.84%  JIT Compilation  [.] FlatPersistentClassInfo::serializeClass(TR_PersistentClassInfo*, FlatPersistentClassInfo*)
   0.68%  JIT Compilation  [.] _ZN6JITaaS3Any27MergePartialFromCodedStreamEPN6google8protobuf2io16CodedInputStreamE
   0.51%  JIT Compilation  [.] JITaaS::AnyData::~AnyData()
   0.48%  JIT Compilation  [.] JITaaS::Any::clear_type()
   0.42%  JIT Compilation  [.] _ZN6JITaaS8J9Stream12readBlockingINS_15J9ServerMessageEEEvRT_
   0.38%  JIT Compilation  [.] _ZN6JITaaS7AnyData27MergePartialFromCodedStreamEPN6google8protobuf2io16CodedInputStreamE
   0.30%  JIT Compilation  [.] JITaaS::J9ServerMessage::Clear()
   0.30%  JIT Compilation  [.] _ZN6google8protobuf8internal20RepeatedPtrFieldBase5ClearINS0_16RepeatedPtrFieldIN6JITaaS3AnyEE11TypeHandlerEEEvv
   0.27%  JIT Compilation  [.] JITaaS::Any::Any(JITaaS::Any const&)
   0.25%  JIT Compilation  [.] void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag).isra.323
   0.22%  JIT Compilation  [.] JITaaS::Any::ByteSizeLong() const
   0.21%  JIT Compilation  [.] J9::MonitorTable::removeAndDestroy(TR::Monitor*)
   0.21%  JIT Compilation  [.] TR_ResolvedJ9Method::construct()
   0.18%  JIT Compilation  [.] JITaaS::Any::InternalSerializeWithCachedSizesToArray(bool, unsigned char*) const
   0.18%  JIT Compilation  [.] JITaaS::J9ServerMessage::MergePartialFromCodedStream(google::protobuf::io::CodedInputStream*)
   0.17%  JIT Compilation  [.] JITaaS::AnyData::ByteSizeLong() const
   0.17%  JIT Compilation  [.] operator new(unsigned long)
   0.16%  JIT Compilation  [.] TR_IProfiler::findOrCreateEntry(int, unsigned long, bool)
   0.16%  JIT Compilation  [.] jitParseSignature
   0.15%  JIT Compilation  [.] JITaaS::J9ClientMessage::ByteSizeLong() const
   0.15%  JIT Compilation  [.] TR_ResolvedJ9Method::isMethodInValidLibrary()
   0.14%  JIT Compilation  [.] JITaaS::AnyData::InternalSerializeWithCachedSizesToArray(bool, unsigned char*) const
   0.14%  JIT Compilation  [.] JITaaS::J9ServerMessage::IsInitialized constv
   0.14%  JIT Compilation  [.] J9::Compilation::fej9vm()
   0.13%  JIT Compilation  [.] _ZNK6google8protobuf8internal20RepeatedPtrFieldBase3GetINS0_16RepeatedPtrFieldIN6JITaaS3AnyEE11TypeHandlerEEERKNT_4TypeEi
   0.12%  JIT Compilation  [.] J9::VMEnv::acquireVMAccessIfNeeded(TR_J9VMBase*)
   0.12%  JIT Compilation  [.] TR::CompilationInfo::getCompInfoForThread(J9VMThread*)
   0.12%  JIT Compilation  [.] TR_ResolvedJ9Method::TR_ResolvedJ9Method(TR_OpaqueMethodBlock*, TR_FrontEnd*, TR_Memory*, TR_ResolvedMethod*, unsigned int)
   0.11%  JIT Compilation  [.] acquireVMaccessIfNeeded(J9VMThread*, TR_YesNoMaybe)
   0.10%  JIT Compilation  [.] JITaaS::GetArgs<int>::getArgs(JITaaS::AnyData const*, unsigned long)
   0.10%  JIT Compilation  [.] protobuf_compile_2eproto::InitDefaultsAny()
   0.10%  JIT Compilation  [.] TR::CompilationInfoPerThread::getStream()
   0.10%  JIT Compilation  [.] _ZN6JITaaS12AnyPrimitiveIKSt5tupleIJ41TR_ResolvedJ9JITaaSServerMethodInfoStructNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_EEvE10write_implIS9_JLm0ELm1ELm
   0.10%  JIT Compilation  [.] remoteCompile(J9VMThread*, TR::Compilation*, TR_ResolvedMethod*, J9Method*, TR::IlGeneratorMethodDetails&, TR::CompilationInfoPerThreadBase*)
   0.10%  JIT Compilation  [.] void JITaaS::J9Stream::writeBlocking<JITaaS::J9ClientMessage>(JITaaS::J9ClientMessage const&)
   0.09%  JIT Compilation  [.] JITaaS::Any::Any()
   0.09%  JIT Compilation  [.] getOriginalROMMethodUnchecked
   0.09%  JIT Compilation  [.] JITaaS::Any::Clear()
   0.09%  JIT Compilation  [.] TR_ContiguousIPMethodHashTableEntry::serialize(TR_IPMethodHashTableEntry*)
   0.09%  JIT Compilation  [.] JITaaS::J9ClientMessage::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const
   0.09%  JIT Compilation  [.] OMR::Options::getCmdLineOptions()
   0.09%  JIT Compilation  [.] JITaaS::AnyData::AnyData()
   0.08%  JIT Compilation  [.] std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)
   0.08%  JIT Compilation  [.] TR::Region::allocate(unsigned long, void*)
   0.08%  JIT Compilation  [.] _ZN31TR_ResolvedJ9JITaaSServerMethod28createResolvedJ9MethodMirrorERSt5tupleIJ41TR_ResolvedJ9JITaaSServerMethodInfoStructNSt7__cxx1112basic_stringIcSt11char_traitsIc
   0.08%  JIT Compilation  [.] TR_ResolvedJ9Method::cp().localalias.201

It follows that we must reduce the number of messages exchanged per compilation between server and client

mpirvu commented 5 years ago

Here's the number and type of messages sent between server and client:

#0000    5475 compilationCode
#0001   51674 mirrorResolvedJ9Method
#0002   30209 get_params_to_construct_TR_j9method
#0105   43941 ResolvedMethod_staticAttributes
#0106   55400 ResolvedMethod_getClassFromConstantPool
#0107   10081 ResolvedMethod_getDeclaringClassFromFieldOrStatic
#0108  161798 ResolvedMethod_classOfStatic
#0109     746 ResolvedMethod_startAddressForJNIMethod
#0110  117222 ResolvedMethod_fieldAttributes
#0111   85483 ResolvedMethod_getResolvedStaticMethodAndMirror
#0112  127961 ResolvedMethod_getResolvedSpecialMethodAndMirror
#0113     704 ResolvedMethod_classCPIndexOfMethod
#0116  288175 ResolvedMethod_getResolvedPossiblyPrivateVirtualMethodAndMirror
#0118   20436 ResolvedMethod_getResolvedInterfaceMethod_2
#0119   75153 ResolvedMethod_getResolvedInterfaceMethodAndMirror_3
#0120    3611 ResolvedMethod_getResolvedInterfaceMethodOffset
#0122     259 ResolvedMethod_isSubjectToPhaseChange
#0125    8251 ResolvedMethod_getRemoteROMString
#0126    2090 ResolvedMethod_fieldOrStaticName
#0127    2685 ResolvedMethod_getRemoteROMClassAndMethods
#0141     633 ResolvedMethod_setWarmCallGraphTooBig
#0147   28800 ResolvedMethod_getResolvedImproperInterfaceMethodAndMirror
#0148   12523 ResolvedMethod_isUnresolvedString
#0149   12524 ResolvedMethod_stringConstant
#0202    5644 VM_getSuperClass     
#0208       7 VM_getSystemClassFromClassName
#0211       1 VM_getClassClassPointer
#0212   24861 VM_setJ2IThunk
#0213    2739 VM_getClassOfMethod 
#0214     199 VM_getBaseComponentClass
#0215   55781 VM_getLeafComponentClassFromArrayClass
#0216    9477 VM_isClassLoadedBySystemClassLoader
#0217   77700 VM_getClassFromSignature 
#0218  154652 VM_jitFieldsAreSame
#0219   26663 VM_jitStaticsAreSame
#0220   10918 VM_getComponentClassFromArrayClass
#0221   59615 VM_classHasBeenReplaced
#0222   25731 VM_classHasBeenExtended
#0229   10387 VM_isClassInitialized
#0231    6992 VM_getByteOffsetToLockword
#0232    1557 VM_isString1      
#0233     363 VM_getMethods
#0236    5617 VM_isPrimitiveArray 
#0237   33873 VM_getAllocationSize 
#0239   27345 VM_stackWalkerMaySkipFrames 
#0240    1712 VM_hasFinalFieldsInClass
#0242   19632 VM_getHostClass
#0245    7215 VM_getClassFromNewArrayType
#0246      98 VM_isCloneable
#0247   33880 VM_canAllocateInlineClass
#0248   18261 VM_getArrayClassFromComponentClass
#0258     196 VM_getOffsetOfClassFromJavaLangClassField
#0266   44447 VM_getResolvedVirtualMethod
#0267   14201 VM_sameClassLoaders  
#0268    9454 VM_isUnloadAssumptionRequired
#0270     336 VM_getInstanceFieldOffset
#0272     498 VM_hasFinalizer
#0273    3080 VM_getClassDepthAndFlagsValue
#0274    1019 VM_getMethodFromName
#0278     788 VM_getReferenceSlotsInClass
#0282      48 VM_getInterpreterVTableSlot
#0285     482 VM_isOwnableSyncClass
#0291       2 VM_getObjectNewInstanceImplMethod
#0297     125 VM_instanceOfOrCheckCast
#0300     323 VM_getResolvedMethodsAndMirror
#0301       1 VM_getVMInfo
#0401       2 CompInfo_getInvocationCount
#0404   10939 CompInfo_isJNINative
#0405    5470 CompInfo_isJSR292
#0406    5470 CompInfo_getMethodBytecodeSize
#0408       2 CompInfo_setInvocationCountAtomic
#0409   11515 CompInfo_isClassSpecial
#0500   16039 ClassEnv_classFlagsValue
#0501   18239 ClassEnv_classDepthOf
#0502   22619 ClassEnv_classInstanceSize
#0800       1 CHTable_getAllClassInfo
#0801    5476 CHTable_getClassInfoUpdates
#0900  230244 IProfiler_profilingSample        
#0901   85885 IProfiler_searchForMethodSample
#0903   15350 IProfiler_setCallCount
#1000    5297 Recompilation_getExistingMethodInfo
mpirvu commented 5 years ago

A new distribution of messages with a build from 2018/12/04. Warm compilations only with OSR enabled.

Type# #called TypeName
#0000    5415 compilationCode
#0001   35126 mirrorResolvedJ9Method
#0002   24516 get_params_to_construct_TR_j9method
#0105   20364 ResolvedMethod_staticAttributes
#0106   39236 ResolvedMethod_getClassFromConstantPool
#0107    7934 ResolvedMethod_getDeclaringClassFromFieldOrStatic
#0108   38156 ResolvedMethod_classOfStatic
#0109     633 ResolvedMethod_startAddressForJNIMethod
#0110   64041 ResolvedMethod_fieldAttributes
#0111   18924 ResolvedMethod_getResolvedStaticMethodAndMirror
#0112   31741 ResolvedMethod_getResolvedSpecialMethodAndMirror
#0113     426 ResolvedMethod_classCPIndexOfMethod
#0116   46748 ResolvedMethod_getResolvedPossiblyPrivateVirtualMethodAndMirror
#0118   17134 ResolvedMethod_getResolvedInterfaceMethod_2
#0119   35220 ResolvedMethod_getResolvedInterfaceMethodAndMirror_3
#0120    2211 ResolvedMethod_getResolvedInterfaceMethodOffset
#0122     252 ResolvedMethod_isSubjectToPhaseChange
#0125    9146 ResolvedMethod_getRemoteROMString
#0126    2149 ResolvedMethod_fieldOrStaticName
#0127    2817 ResolvedMethod_getRemoteROMClassAndMethods
#0131       2 ResolvedMethod_isUnresolvedCallSiteTableEntry
#0132       1 ResolvedMethod_callSiteTableEntryAddress
#0133       1 ResolvedMethod_getResolvedDynamicMethod
#0141    1198 ResolvedMethod_setWarmCallGraphTooBig
#0147   23467 ResolvedMethod_getResolvedImproperInterfaceMethodAndMirror
#0148    9069 ResolvedMethod_isUnresolvedString
#0149    9069 ResolvedMethod_stringConstant
#0208       7 VM_getSystemClassFromClassName
#0211       1 VM_getClassClassPointer
#0212   20709 VM_setJ2IThunk
#0213    1899 VM_getClassOfMethod
#0217   70073 VM_getClassFromSignature
#0218   79763 VM_jitFieldsAreSame
#0219   15621 VM_jitStaticsAreSame
#0221   46647 VM_classHasBeenReplaced
#0222   21706 VM_classHasBeenExtended
#0229     387 VM_isClassInitialized
#0230   30193 VM_getOSRFrameSizeInBytes
#0232     688 VM_isString1
#0233     368 VM_getMethods
#0239   16612 VM_stackWalkerMaySkipFrames
#0245    3925 VM_getClassFromNewArrayType
#0258     157 VM_getOffsetOfClassFromJavaLangClassField
#0266   28044 VM_getResolvedVirtualMethod
#0270     239 VM_getInstanceFieldOffset
#0273    2443 VM_getClassDepthAndFlagsValue
#0274     967 VM_getMethodFromName
#0278    8847 VM_getReferenceSlotsInClass
#0282      15 VM_getInterpreterVTableSlot
#0291       2 VM_getObjectNewInstanceImplMethod
#0294       1 VM_setInvokeExactJ2IThunk
#0297     105 VM_instanceOfOrCheckCast
#0300      73 VM_getResolvedMethodsAndMirror
#0301       1 VM_getVMInfo
#0401       2 CompInfo_getInvocationCount
#0408       2 CompInfo_setInvocationCountAtomic
#0500   10656 ClassEnv_classFlagsValue
#0800       1 CHTable_getAllClassInfo
#0801    5416 CHTable_getClassInfoUpdates
#0900   13054 IProfiler_profilingSample
#0901    3428 IProfiler_searchForMethodSample
#0903    8570 IProfiler_setCallCount