xiangzhai
commented
5 years ago \cc @BruceForstall
Closed xiangzhai closed 2 years ago
xiangzhai
commented
5 years ago \cc @BruceForstall
sandreenko
commented
5 years ago cc @dotnet/jit-contrib
BruceForstall
commented
5 years ago @noahfalk @kouvel Did the implementation of Tiered Compilation possibly break the interpreter path? (If so, do you have a suggested solution to get the interpreter working again?) Or does this deserve more investigation?
noahfalk
commented
5 years ago The last time I tested interpreter + tiered compilation was probably 2-3 years ago when I refactored it a bit. I don't know if Kount tested it more recently. In the current code you could set a breakpoint here to determine how tiered compilation is setting the jit flags and here to see if those flags made it to the JIT without further alteration and we enter GenerateInterpreterStub.
AndyAyersMS
commented
5 years ago @xiangzhai have you tried what I suggested a while back?
https://github.com/dotnet/coreclr/issues/24824#issuecomment-497048982
xiangzhai
commented
5 years ago Hi @AndyAyersMS
I tried https://github.com/dotnet/coreclr/issues/24824#issuecomment-497166854
diff --git a/src/vm/tieredcompilation.cpp b/src/vm/tieredcompilation.cpp
index 8cdfc48..d13fb86 100644
--- a/src/vm/tieredcompilation.cpp
+++ b/src/vm/tieredcompilation.cpp
@@ -860,9 +860,6 @@ CORJIT_FLAGS TieredCompilationManager::GetJitFlags(NativeCodeVersion nativeCodeV
CORJIT_FLAGS flags;
if (!nativeCodeVersion.GetMethodDesc()->IsEligibleForTieredCompilation())
{
-#ifdef FEATURE_INTERPRETER
- flags.Set(CORJIT_FLAGS::CORJIT_FLAG_MAKEFINALCODE);
-#endif
return flags;
}
@@ -877,9 +874,6 @@ CORJIT_FLAGS TieredCompilationManager::GetJitFlags(NativeCodeVersion nativeCodeV
// fall through
case NativeCodeVersion::OptimizationTierOptimized:
-#ifdef FEATURE_INTERPRETER
- flags.Set(CORJIT_FLAGS::CORJIT_FLAG_MAKEFINALCODE);
-#endif
break;
default:
But it lead to SEGFAULT when mixed Interpreter and JIT as described above https://github.com/dotnet/coreclr/issues/27761#issue-519851291 .
Thanks, Leslie Zhai
xiangzhai
commented
5 years ago Hi @noahfalk
I tracked interpreterFallback https://github.com/dotnet/coreclr/issues/24654#issue-445775637 and g_CORDebuggerControlFlags https://github.com/dotnet/coreclr/issues/24654 for 2.1 branch.
We started porting CoreCLR to MIPS64 based on 2.1 branch, then migrated to historical (about 5 months ago) master https://github.com/dotnet/coreclr/commit/e999f37a83583768aa68ad9fda805088ac9df6b8 So I need to track them again.
Thanks, Leslie Zhai
xiangzhai
commented
5 years ago Hi @noahfalk
./build.sh cmakeargs "-DFEATURE_INTERPRETER=1" skipcrossgen ignorewarnings# corefx && aspnetcore ALLIN1
export CORE_LIBRARIES=/home/loongson/zhaixiang/corefx-3.1-Linux.mips64.Debug
# interpreter
export COMPlus_DumpInterpreterStubs=1
export COMPlus_Interpret=*
export COMPlus_InterpreterJITThreshold=999999
export COMPlus_InterpreterFallback=1
export COMPlus_InterpreterPrintPostMortem=1
export COMPlus_InterpreterDoLoopMethods=1
export COMPlus_TraceInterpreterEntries=1
export COMPlus_TieredCompilation=1
export COMPlus_ZapDisable=1
export COMPlus_JITMinOpts=1
# jit
export COMPlus_JitFunctionTrace=1And X86 and ARM64 use the similar environment variables.
Breakpoint 2, TieredCompilationManager::GetJitFlags (nativeCodeVersion=...) at /home/loongson/zhaixiang/coreclr-mips64-dev/src/vm/tieredcompilation.cpp:860
860 CORJIT_FLAGS flags;
(gdb) n
861 if (!nativeCodeVersion.GetMethodDesc()->IsEligibleForTieredCompilation())
(gdb) n
864 flags.Set(CORJIT_FLAGS::CORJIT_FLAG_MAKEFINALCODE);
(gdb) n
866 return flags;
(gdb) i r v0
v0: 0x40000
(gdb) c
Continuing.
Breakpoint 1, invokeCompileMethodHelper (jitMgr=0x12007cf50, comp=0xffffff89a0, info=0xffffff8810, jitFlags=..., nativeEntry=0xffffff87c0, nativeSizeOfCode=0xffffff87bc) at /home/loongson/zhaixiang/coreclr-mips64-dev/src/vm/jitinterface.cpp:12350
12350 CorJitResult ret = CORJIT_SKIPPED; // Note that CORJIT_SKIPPED is an error exit status code
(gdb) n
12352 comp->setJitFlags(jitFlags);
(gdb) n
12360 if (FAILED(ret) && jitMgr->m_alternateJit
(gdb) n
12393 bool isInterpreterStub = false;
(gdb) n
12394 bool interpreterFallback = (s_InterpreterFallback.val(CLRConfig::INTERNAL_InterpreterFallback) != 0);
(gdb) n
12396 if (interpreterFallback == false)
(gdb) p/x interpreterFallback
$5 = 0x1
(gdb) n
12409 if (FAILED(ret) && jitMgr->m_jit)
(gdb) n
12411 ret = CompileMethodWithEtwWrapper(jitMgr,
(gdb) n
12412 comp,
(gdb) n
12413 info,
(gdb) n
12415 nativeEntry,
(gdb) n
12416 nativeSizeOfCode);
(gdb) n
12411 ret = CompileMethodWithEtwWrapper(jitMgr,
(gdb) n
{ Start Jitting System.AppContext:Setup(long,long,int) (MethodHash=d8570543)
} Jitted Entry 001 at 000000ff`7c826988 method System.AppContext:Setup(long,long,int) size 00000220
12417 }
(gdb) n
12419 if (interpreterFallback == true)
(gdb) n
12423 if (FAILED(ret) && !jitFlags.IsSet(CORJIT_FLAGS::CORJIT_FLAG_IMPORT_ONLY) && !jitFlags.IsSet(CORJIT_FLAGS::CORJIT_FLAG_MAKEFINALCODE))
(gdb) n
12430 }
The flags value is 0x40000, and jitFlags.IsSet(CORJIT_FLAGS::CORJIT_FLAG_MAKEFINALCODE) is false, so Interpreter::GenerateInterpreterStub will NOT be called.
Thanks, Leslie Zhai
noahfalk
commented
5 years ago This code flow:
861 if (!nativeCodeVersion.GetMethodDesc()->IsEligibleForTieredCompilation()) (gdb) n 864 flags.Set(CORJIT_FLAGS::CORJIT_FLAG_MAKEFINALCODE);
suggests that you debugged a method that was not eligible for tiered compilation, and from there it would be by design not to use the interpreter because without tiering we can never switch from interpreter back to JIT.
Two potential cases here that would be useful to distinguish: 1) One specific method that you stepped through didn't support tiering, but others would have supported it (set a breakpoint in the case where the if evaluates false and see if execution ever goes that way) 2) All the methods are reporting that they don't support tiering. Given that COMPlus_TieredCompilation=1, this is not the behavior I would expect. The code here has the rules to determine whether each method will support tiering so we'd want to understand why it always evaluated false.
xiangzhai
commented
5 years ago Hi @noahfalk
Thanks for your kind response!
HelloWorld.x64.JitFunctionTrace.log ARM64 and MIPS64 also goes to case 2. I will investigate why it always evaluated false.
Thanks, Leslie Zhai
xiangzhai
commented
5 years ago Hi @noahfalk
Base on https://github.com/dotnet/coreclr/commit/03b7b318ab6641c20baeb37f963c4599a97d9276
Build and run HelloWorld option:
./build.sh cmakeargs "-DFEATURE_INTERPRETER=1" skipcrossgen ignorewarnings
# corefx
export CORE_LIBRARIES=~/dotnet-sdk-3.0.100-linux-x64/shared/Microsoft.NETCore.App/3.0.0
# interpreter
export COMPlus_DumpInterpreterStubs=1
export COMPlus_Interpret=*
export COMPlus_InterpreterJITThreshold=999999
export COMPlus_InterpreterFallback=1
export COMPlus_InterpreterPrintPostMortem=1
export COMPlus_InterpreterDoLoopMethods=1
export COMPlus_TraceInterpreterEntries=1
export COMPlus_TieredCompilation=1
export COMPlus_ZapDisable=1
# jit
export COMPlus_JitFunctionTrace=1
gdb --args bin/Product/Linux.x64.Debug/corerun ~/HelloWorld.dllPlease apply the patch https://github.com/dotnet/coreclr/pull/27850 suggested by @jkotas at first.
Debug patch:
diff --git a/src/vm/method.cpp b/src/vm/method.cpp
index b36c78c..8d42f10 100644
--- a/src/vm/method.cpp
+++ b/src/vm/method.cpp
@@ -4850,6 +4850,7 @@ bool MethodDesc::DetermineAndSetIsEligibleForTieredCompilation()
{
m_bFlags2 |= enum_flag2_IsEligibleForTieredCompilation;
_ASSERTE(IsVersionable());
+ _ASSERTE(!"WhoReachHere");
return true;
}
#endif
It seems that only HelloWorld.Program:Main(System.String[]) method will support tiering:
{ Start Jitting Method 211 Internal.Runtime.CompilerServices.Unsafe:IsNullRef(byref):bool (MethodHash=b7a571a2) MinOpts
} Jitted Method 211 at 00007fff`7c0b84c0 method Internal.Runtime.CompilerServices.Unsafe:IsNullRef(byref):bool size 0000003d
{ Start Jitting Method 212 HelloWorld.Program:Main(System.String[]) (MethodHash=ab41dfb7) MinOpts
Assert failure(PID 14534 [0x000038c6], Thread: 14534 [0x38c6]): !"WhoReachHere"
File: /home/zhaixiang/coreclr/src/vm/method.cpp Line: 4853
Image: /home/zhaixiang/coreclr/bin/Product/Linux.x64.Debug/corerun
Changed the debug patch:
diff --git a/src/vm/method.cpp b/src/vm/method.cpp
index b36c78c..4b7665e 100644
--- a/src/vm/method.cpp
+++ b/src/vm/method.cpp
@@ -4850,6 +4850,7 @@ bool MethodDesc::DetermineAndSetIsEligibleForTieredCompilation()
{
m_bFlags2 |= enum_flag2_IsEligibleForTieredCompilation;
_ASSERTE(IsVersionable());
+ printf("WhoReachHere\n");
return true;
}
#endif
HelloWorld.WhoReachHere.JitFunctionTrace.log
A small number of methods from System.Private.CoreLib and corefx will support tiering, but lots of them still goto JIT. Compared with HotSpot Tiered compilation http://cr.openjdk.java.net/~vlivanov/talks/2015_JIT_Overview.pdf Bytecode → Interpreter + JITted executable is what we desire.
Or is it able to just enable Interpreter only just like java -Xint?
Thanks, Leslie Zhai
xiangzhai
commented
4 years ago It is not able to work equivalent to java -Xmixed, so please reopen the issue.
BruceForstall
commented
2 years ago We currently have no work ongoing to support the interpreter, so I am going to close this.
Hi,
OpenJDK is able to just enable interpreter:
or just enable JIT compilers:
or mixed:
And what about CoreCLR?
But there are only
Start JittingandJitted Entry, noGenerating interpretation stubat all:Although it is able to hack tieredcompilation https://github.com/dotnet/coreclr/issues/24824#issuecomment-497048982 to force interpreter. There is SEGFAULT issue:
How to mix Interpreter and JIT compiler? Please give me some hint.
Thanks, Leslie Zhai
category:correctness theme:interpreter skill-level:intermediate cost:small