`JIT/jit64/opt/rngchk/RngchkStress2.cs` failing to build with `error CS8078: An expression is too long or complex to compile`

[radical]

Failing on osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit:

Failing build:

/Users/runner/work/1/s/src/tests/JIT/jit64/opt/rngchk/RngchkStress2.cs(527,796): error CS8078: An expression is too long or complex to compile [/Users/runner/work/1/s/src/tests/JIT/jit64/opt/rngchk/RngchkStress2_o.csproj] [/Users/runner/work/1/s/src/tests/build.proj]
##[error]src/tests/JIT/jit64/opt/rngchk/RngchkStress2.cs(527,796): error CS8078: An expression is too long or complex to compile [/Users/runner/work/1/s/src/tests/JIT/jit64/opt/rngchk/RngchkStress2_o.csproj]

This was hit on https://github.com/dotnet/runtime/pull/87522 .

{
  "ErrorMessage": "An expression is too long or complex to compile",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=315357 Result validation: :white_check_mark: Known issue matched with the provided build.

Report

Build	Definition	Step Name	Console log	Pull Request
533692	dotnet/runtime	Build Tests	Log	dotnet/runtime#95758
532325	dotnet/runtime	Build Tests	Log	dotnet/runtime#95758
531205	dotnet/runtime	Build Tests	Log	dotnet/runtime#95758
528402	dotnet/runtime	Build Tests	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	4

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details

---

Failing on `osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit`: [Failing build](https://dev.azure.com/dnceng-public/public/_build/results?buildId=315357&view=logs&jobId=b37acc34-3c47-5685-7f27-b5a5ac18be64&j=28035387-23c9-50dd-d519-66c274947a2c&t=e05a8684-8106-59f6-8d1b-f5f024dfb6f2): ``` /Users/runner/work/1/s/src/tests/JIT/jit64/opt/rngchk/RngchkStress2.cs(527,796): error CS8078: An expression is too long or complex to compile [/Users/runner/work/1/s/src/tests/JIT/jit64/opt/rngchk/RngchkStress2_o.csproj] [/Users/runner/work/1/s/src/tests/build.proj] ##[error]src/tests/JIT/jit64/opt/rngchk/RngchkStress2.cs(527,796): error CS8078: An expression is too long or complex to compile [/Users/runner/work/1/s/src/tests/JIT/jit64/opt/rngchk/RngchkStress2_o.csproj] ``` This was hit on https://github.com/dotnet/runtime/pull/87522 .

Author:	radical
Assignees:	-
Labels:	`area-CodeGen-coreclr`
Milestone:	-

[radical]

log - full stack trace: https://gist.github.com/radical/8ba41143e077b8b6eab371237c1756aa

[JulieLeeMSFT]

@jcouv, thanks for quickly disabling the test. Are you looking into the rootcause of the failure?

[radical]

@jcouv My attempts at disabling the tests didn't quite work. Could you please take a look?

[jcouv]

Thanks for the ping. I'm taking a look. I wasn't able to repro yet.

[jcouv]

@radical I'm trying to repro this issue, but am unable so far. Basically, I copied the deeply nested code from RngchkStress2.cs and put in a compiler unittest. Then I call the API (SemanticModel.GetOperation) reported in the overflow stack trace. (btw, thanks for that stack trace) I'm able to do that in debug mode, which normally consumes a lot more stack. But I get no overflow.

Looking at the code involved in the stack trace (CSharpOperationFactory, which is a big visitor), I see that code has not changed since January, so it feels unlikely that we've increased the stack consumption there.

Some thoughts so far:

• it's possible that the tree that we feed to CSharpOperationFactory has changed, but that also feels unlikely (I looked at the history of "Operation" tests, but saw nothing significant)
• maybe the compiler used is not a recent version that I'm looking at. Doing the repro with an extra #error version in the problematic file would help confirm the version that is used (it will be output as an error message)
• it doesn't look like the toolset compiler is being updated as part of the problematic PR. So I'm thinking that root cause may lie elsewhere. Getting version from #error version before the change would help confirm that (I expect it will be the same as after the change)
• I wonder whether is there something special about the machine or the environment that makes the stack size more constrained?
• the other factor that affects the stack usage is the runtime, which was updated as part of this PR. Would it be possible to capture a dump for this crash?

[jkotas]

This was hit on https://github.com/dotnet/runtime/pull/87522 .

This PR upgraded the .NET SDK used to build the repo to .NET 8 Preview 5.

This intermittent build error is likely caused by tiered PGO instrumentation that was enabled in .NET 8 Preview 5 by default. The PGO instrumentation may cause program to use different amount of stack space from run to run.

[jcouv]

Thanks @jkotas, that makes sense. Unassigned myself and marked the issue as "untriaged" so that it can be assigned to a new owner.

[am11]

cc @jakobbotsch (I think this will go away with new PGO collection/update)

[jakobbotsch]

I think we should just simplify the test if it is expected that Roslyn will rely on stack for deeply nested input programs. Note that the failures are on osx-x64 which, IIRC, have threads with notoriously small stack limits.

It is certainly expected from our side that new JIT changes may cause generated native code to use more stack. In this case it seems realistic that it is related to tiered PGO being enabled in preview 5. I would guess the source is new inlining as a result of having PGO data available.

[jakobbotsch]

FWIW, the test has 193 nested for loops in it, and the stack trace posted above contains CreateBoundForStatementOperation 179 times. So let's just remove 50 levels of nesting.

[akoeplinger]

I also hit this on HugeArray1.cs in https://github.com/dotnet/runtime/pull/89858

/Users/runner/work/1/s/src/tests/JIT/jit64/opt/cse/HugeArray1.cs(10749,63): error CS8078: An expression is too long or complex to compile [/Users/runner/work/1/s/src/tests/JIT/jit64/opt/cse/HugeArray1.csproj]

[EgorBo]

I suspect there is something wrong with RuntimeHelpers.EnsureSufficientExecutionStack() function - when I make it no-op I am able to reproduce Stack Overflow with only 200kb of stack (new Thread(DoWork, 200*1024)) and PGO doesn't affect it (crashes on both PGO=1 and PGO=0)

[EgorBo]

So, on Apple it seems we only get 364kb of stack (it's m_CacheStackBase - m_CacheStackLimit) + EnsureSufficientExecutionStack (that Roslyn actively uses) fails if there is less than 128kb of stack left so we only have 364-128=236kb of stack available before RuntimeHelpers.EnsureSufficientExecutionStack will start throwing exceptions (and Roslyn will start complaining that a program is too complext to compile).

Main thread still has 8mb of stack. Perhaps, Roslyn should spawn threads with custom size of stack? (at least on Apple) cc @jkotas @jaredpar @jakobbotsch

[jkotas]

Perhaps, Roslyn should spawn threads with custom size of stack?

It won't fix the problem with compilation of deeply nested statements failing intermittently due to non-determinism of PGO and tiered compilation. It will just move the threshold where the compilation becomes unreliable.

[EgorBo]

I changed exception type in EnsureSufficientExecutionStack to BadImageException just to crash Roslyn and get a call stack. Unfortunately, I wasn't able to install sos - lldb complains on dylib this file does not represent a loadable dylib (although. I've checked that it's a correct arm64 binary) 🙁 The problem looks to be https://developer.apple.com/forums/thread/729765 that @hoyosjs already filed

lldb thread backtrace

``` (lldb) thread backtrace * thread #26, stop reason = signal SIGABRT * frame #0: 0x0000000189438724 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x000000018946fc28 libsystem_pthread.dylib`pthread_kill + 288 frame #2: 0x000000018937dae8 libsystem_c.dylib`abort + 180 frame #3: 0x0000000101660994 libcoreclr.dylib`PROCAbort + 64 frame #4: 0x00000001016608ac libcoreclr.dylib`TerminateProcess + 176 frame #5: 0x000000010185f71c libcoreclr.dylib`UnwindManagedExceptionPass1(PAL_SEHException&, _CONTEXT*) + 900 frame #6: 0x000000010185f800 libcoreclr.dylib`DispatchManagedException(PAL_SEHException&, bool) + 68 frame #7: 0x00000001017aa518 libcoreclr.dylib`__FCThrow(void*, RuntimeExceptionKind, unsigned int, char16_t const*, char16_t const*, char16_t const*) + 388 frame #8: 0x0000000101815134 libcoreclr.dylib`ReflectionInvocation::EnsureSufficientExecutionStack() + 92 frame #9: 0x00000001195aee20 frame #10: 0x0000000119563a40 frame #11: 0x0000000119562a2c frame #12: 0x0000000119561f94 frame #13: 0x00000001195618ec frame #14: 0x000000011958cb28 frame #15: 0x000000011958c640 frame #16: 0x000000011958c13c frame #17: 0x000000011959a650 frame #18: 0x000000011956fcf0 frame #19: 0x00000001195734f4 frame #20: 0x000000011959a7d8 frame #21: 0x0000000119573328 frame #22: 0x000000011958c96c frame #23: 0x000000011958c13c frame #24: 0x000000011959a650 frame #25: 0x000000011956fcf0 frame #26: 0x00000001195734f4 frame #27: 0x000000011959a7d8 frame #28: 0x0000000119573328 frame #29: 0x000000011958c96c frame #30: 0x000000011958c13c frame #31: 0x000000011959a650 frame #32: 0x000000011956fcf0 frame #33: 0x00000001195734f4 frame #34: 0x000000011959a7d8 frame #35: 0x0000000119573328 frame #36: 0x000000011958c96c frame #37: 0x000000011958c13c frame #38: 0x000000011959a650 frame #39: 0x000000011956fcf0 frame #40: 0x00000001195734f4 frame #41: 0x000000011959a7d8 frame #42: 0x0000000119573328 frame #43: 0x000000011958c96c frame #44: 0x000000011958c13c frame #45: 0x000000011959a650 frame #46: 0x000000011956fcf0 frame #47: 0x00000001195734f4 frame #48: 0x000000011959a7d8 frame #49: 0x0000000119573328 frame #50: 0x000000011958c96c frame #51: 0x000000011958c13c frame #52: 0x000000011959a650 frame #53: 0x000000011956fcf0 frame #54: 0x00000001195734f4 frame #55: 0x000000011959a7d8 frame #56: 0x0000000119573328 frame #57: 0x000000011958c96c frame #58: 0x000000011958c13c frame #59: 0x000000011959a650 frame #60: 0x000000011956fcf0 frame #61: 0x00000001195734f4 frame #62: 0x000000011959a7d8 frame #63: 0x0000000119573328 frame #64: 0x000000011958c96c frame #65: 0x000000011958c13c frame #66: 0x000000011959a650 frame #67: 0x000000011956fcf0 frame #68: 0x00000001195734f4 frame #69: 0x000000011959a7d8 frame #70: 0x0000000119573328 frame #71: 0x000000011958c96c frame #72: 0x000000011958c13c frame #73: 0x000000011959a650 frame #74: 0x000000011956fcf0 frame #75: 0x00000001195734f4 frame #76: 0x000000011959a7d8 frame #77: 0x0000000119573328 frame #78: 0x000000011958c96c frame #79: 0x000000011958c13c frame #80: 0x000000011959a650 frame #81: 0x000000011956fcf0 frame #82: 0x00000001195734f4 frame #83: 0x000000011959a7d8 frame #84: 0x0000000119573328 frame #85: 0x000000011958c96c frame #86: 0x000000011958c13c frame #87: 0x000000011959a650 frame #88: 0x000000011956fcf0 frame #89: 0x00000001195734f4 frame #90: 0x000000011959a7d8 frame #91: 0x0000000119573328 frame #92: 0x000000011958c96c frame #93: 0x000000011958c13c frame #94: 0x000000011959a650 frame #95: 0x000000011956fcf0 frame #96: 0x00000001195734f4 frame #97: 0x000000011959a7d8 frame #98: 0x0000000119573328 frame #99: 0x000000011958c96c frame #100: 0x000000011958c13c frame #101: 0x000000011959a650 frame #102: 0x000000011956fcf0 frame #103: 0x00000001195734f4 frame #104: 0x000000011959a7d8 frame #105: 0x0000000119573328 frame #106: 0x000000011958c96c frame #107: 0x000000011958c13c frame #108: 0x000000011959a650 frame #109: 0x000000011956fcf0 frame #110: 0x00000001195734f4 frame #111: 0x000000011959a7d8 frame #112: 0x0000000119573328 frame #113: 0x000000011958c96c frame #114: 0x000000011958c13c frame #115: 0x000000011959a650 frame #116: 0x000000011956fcf0 frame #117: 0x00000001195734f4 frame #118: 0x000000011959a7d8 frame #119: 0x0000000119573328 frame #120: 0x000000011958c96c frame #121: 0x000000011958c13c frame #122: 0x000000011959a650 frame #123: 0x000000011956fcf0 frame #124: 0x00000001195734f4 frame #125: 0x000000011959a7d8 frame #126: 0x0000000119573328 frame #127: 0x000000011958c96c frame #128: 0x000000011958c13c frame #129: 0x000000011959a650 frame #130: 0x000000011956fcf0 frame #131: 0x00000001195734f4 frame #132: 0x000000011959a7d8 frame #133: 0x0000000119573328 frame #134: 0x000000011958c96c frame #135: 0x000000011958c13c frame #136: 0x000000011959a650 frame #137: 0x000000011956fcf0 frame #138: 0x00000001195734f4 frame #139: 0x000000011959a7d8 frame #140: 0x0000000119573328 frame #141: 0x000000011958c96c frame #142: 0x000000011958c13c frame #143: 0x000000011959a650 frame #144: 0x000000011956fcf0 frame #145: 0x00000001195734f4 frame #146: 0x000000011959a7d8 frame #147: 0x0000000119573328 frame #148: 0x000000011958c96c frame #149: 0x000000011958c13c frame #150: 0x000000011959a650 frame #151: 0x000000011956fcf0 frame #152: 0x00000001195734f4 frame #153: 0x000000011959a7d8 frame #154: 0x0000000119573328 frame #155: 0x000000011958c96c frame #156: 0x000000011958c13c frame #157: 0x000000011959a650 frame #158: 0x000000011956fcf0 frame #159: 0x00000001195734f4 frame #160: 0x000000011959a7d8 frame #161: 0x0000000119573328 frame #162: 0x000000011958c96c frame #163: 0x000000011958c13c frame #164: 0x000000011959a650 frame #165: 0x000000011956fcf0 frame #166: 0x00000001195734f4 frame #167: 0x000000011959a7d8 frame #168: 0x0000000119573328 frame #169: 0x000000011958c96c frame #170: 0x000000011958c13c frame #171: 0x000000011959a650 frame #172: 0x000000011956fcf0 frame #173: 0x00000001195734f4 frame #174: 0x000000011959a7d8 frame #175: 0x0000000119573328 frame #176: 0x000000011958c96c frame #177: 0x000000011958c13c frame #178: 0x000000011959a650 frame #179: 0x000000011956fcf0 frame #180: 0x00000001195734f4 frame #181: 0x000000011959a7d8 frame #182: 0x0000000119573328 frame #183: 0x000000011958c96c frame #184: 0x000000011958c13c frame #185: 0x000000011959a650 frame #186: 0x000000011956fcf0 frame #187: 0x00000001195734f4 frame #188: 0x000000011959a7d8 frame #189: 0x0000000119573328 frame #190: 0x000000011958c96c frame #191: 0x000000011958c13c frame #192: 0x000000011959a650 frame #193: 0x000000011956fcf0 frame #194: 0x00000001195734f4 frame #195: 0x000000011959a7d8 frame #196: 0x0000000119573328 frame #197: 0x000000011958c96c frame #198: 0x000000011958c13c frame #199: 0x000000011959a650 frame #200: 0x000000011956fcf0 frame #201: 0x00000001195734f4 frame #202: 0x000000011959a7d8 frame #203: 0x0000000119573328 frame #204: 0x000000011958c96c frame #205: 0x000000011958c13c frame #206: 0x000000011959a650 frame #207: 0x000000011956fcf0 frame #208: 0x00000001195734f4 frame #209: 0x000000011959a7d8 frame #210: 0x0000000119573328 frame #211: 0x000000011958c96c frame #212: 0x000000011958c13c frame #213: 0x000000011959a650 frame #214: 0x000000011956fcf0 frame #215: 0x00000001195734f4 frame #216: 0x000000011959a7d8 frame #217: 0x0000000119573328 frame #218: 0x000000011958c96c frame #219: 0x000000011958c13c frame #220: 0x000000011959a650 frame #221: 0x000000011956fcf0 frame #222: 0x00000001195734f4 frame #223: 0x000000011959a7d8 frame #224: 0x0000000119573328 frame #225: 0x000000011958c96c frame #226: 0x000000011958c13c frame #227: 0x000000011959a650 frame #228: 0x000000011956fcf0 frame #229: 0x00000001195734f4 frame #230: 0x000000011959a7d8 frame #231: 0x0000000119573328 frame #232: 0x000000011958c96c frame #233: 0x000000011958c13c frame #234: 0x000000011959a650 frame #235: 0x000000011956fcf0 frame #236: 0x00000001195734f4 frame #237: 0x000000011959a7d8 frame #238: 0x0000000119573328 frame #239: 0x000000011958c96c frame #240: 0x000000011958c13c frame #241: 0x000000011959a650 frame #242: 0x000000011956fcf0 frame #243: 0x00000001195734f4 frame #244: 0x000000011959a7d8 frame #245: 0x0000000119573328 frame #246: 0x000000011958c96c frame #247: 0x000000011958c13c frame #248: 0x000000011959a650 frame #249: 0x000000011956fcf0 frame #250: 0x00000001195734f4 frame #251: 0x000000011959a7d8 frame #252: 0x0000000119573328 frame #253: 0x000000011958c96c frame #254: 0x000000011958c13c frame #255: 0x000000011959a650 frame #256: 0x000000011956fcf0 frame #257: 0x00000001195734f4 frame #258: 0x000000011959a7d8 frame #259: 0x0000000119573328 frame #260: 0x000000011958c96c frame #261: 0x000000011958c13c frame #262: 0x000000011959a650 frame #263: 0x000000011956fcf0 frame #264: 0x00000001195734f4 frame #265: 0x000000011959a7d8 frame #266: 0x0000000119573328 frame #267: 0x000000011958c96c frame #268: 0x000000011958c13c frame #269: 0x000000011959a650 frame #270: 0x000000011956fcf0 frame #271: 0x00000001195734f4 frame #272: 0x000000011959a7d8 frame #273: 0x0000000119573328 frame #274: 0x000000011958c96c frame #275: 0x000000011958c13c frame #276: 0x000000011959a650 frame #277: 0x000000011956fcf0 frame #278: 0x00000001195734f4 frame #279: 0x000000011959a7d8 frame #280: 0x0000000119573328 frame #281: 0x000000011958c96c frame #282: 0x000000011958c13c frame #283: 0x000000011959a650 frame #284: 0x000000011956fcf0 frame #285: 0x00000001195734f4 frame #286: 0x000000011959a7d8 frame #287: 0x0000000119573328 frame #288: 0x000000011958c96c frame #289: 0x000000011958c13c frame #290: 0x000000011959a650 frame #291: 0x000000011956fcf0 frame #292: 0x00000001195734f4 frame #293: 0x000000011959a7d8 frame #294: 0x0000000119573328 frame #295: 0x000000011958c96c frame #296: 0x000000011958c13c frame #297: 0x000000011959a650 frame #298: 0x000000011956fcf0 frame #299: 0x00000001195734f4 frame #300: 0x000000011959a7d8 frame #301: 0x0000000119573328 frame #302: 0x000000011958c96c frame #303: 0x000000011958c13c frame #304: 0x000000011959a650 frame #305: 0x000000011956fcf0 frame #306: 0x00000001195734f4 frame #307: 0x000000011959a7d8 frame #308: 0x0000000119573328 frame #309: 0x000000011958c96c frame #310: 0x000000011958c13c frame #311: 0x000000011959a650 frame #312: 0x000000011956fcf0 frame #313: 0x00000001195734f4 frame #314: 0x000000011959a7d8 frame #315: 0x0000000119573328 frame #316: 0x000000011958c96c frame #317: 0x000000011958c13c frame #318: 0x000000011959a650 frame #319: 0x000000011956fcf0 frame #320: 0x00000001195734f4 frame #321: 0x000000011959a7d8 frame #322: 0x0000000119573328 frame #323: 0x000000011958c96c frame #324: 0x000000011958c13c frame #325: 0x000000011959a650 frame #326: 0x000000011956fcf0 frame #327: 0x00000001195734f4 frame #328: 0x000000011959a7d8 frame #329: 0x0000000119573328 frame #330: 0x000000011958c96c frame #331: 0x000000011958c13c frame #332: 0x000000011959a650 frame #333: 0x000000011956fcf0 frame #334: 0x00000001195734f4 frame #335: 0x000000011959a7d8 frame #336: 0x0000000119573328 frame #337: 0x000000011958c96c frame #338: 0x000000011958c13c frame #339: 0x000000011959a650 frame #340: 0x000000011956fcf0 frame #341: 0x00000001195734f4 frame #342: 0x000000011959a7d8 frame #343: 0x0000000119573328 frame #344: 0x000000011958c96c frame #345: 0x000000011958c13c frame #346: 0x000000011959a650 frame #347: 0x000000011956fcf0 frame #348: 0x00000001195734f4 frame #349: 0x000000011959a7d8 frame #350: 0x0000000119573328 frame #351: 0x000000011958c96c frame #352: 0x000000011958c13c frame #353: 0x000000011959a650 frame #354: 0x000000011956fcf0 frame #355: 0x00000001195734f4 frame #356: 0x000000011959a7d8 frame #357: 0x0000000119573328 frame #358: 0x000000011958c96c frame #359: 0x000000011958c13c frame #360: 0x000000011959a650 frame #361: 0x000000011956fcf0 frame #362: 0x00000001195734f4 frame #363: 0x000000011959a7d8 frame #364: 0x0000000119573328 frame #365: 0x000000011958c96c frame #366: 0x000000011958c13c frame #367: 0x000000011959a650 frame #368: 0x000000011956fcf0 frame #369: 0x00000001195734f4 frame #370: 0x000000011959a7d8 frame #371: 0x0000000119573328 frame #372: 0x000000011958c96c frame #373: 0x000000011958c13c frame #374: 0x000000011959a650 frame #375: 0x000000011956fcf0 frame #376: 0x00000001195734f4 frame #377: 0x000000011959a7d8 frame #378: 0x0000000119573328 frame #379: 0x000000011958c96c frame #380: 0x000000011958c13c frame #381: 0x000000011959a650 frame #382: 0x000000011956fcf0 frame #383: 0x00000001195734f4 frame #384: 0x000000011959a7d8 frame #385: 0x0000000119573328 frame #386: 0x000000011958c96c frame #387: 0x000000011958c13c frame #388: 0x000000011959a650 frame #389: 0x000000011956fcf0 frame #390: 0x00000001195734f4 frame #391: 0x000000011959a7d8 frame #392: 0x0000000119573328 frame #393: 0x000000011958c96c frame #394: 0x000000011958c13c frame #395: 0x000000011959a650 frame #396: 0x000000011956fcf0 frame #397: 0x00000001195734f4 frame #398: 0x000000011959a7d8 frame #399: 0x0000000119573328 frame #400: 0x000000011958c96c frame #401: 0x000000011958c13c frame #402: 0x000000011959a650 frame #403: 0x000000011956fcf0 frame #404: 0x00000001195734f4 frame #405: 0x000000011959a7d8 frame #406: 0x0000000119573328 frame #407: 0x000000011958c96c frame #408: 0x000000011958c13c frame #409: 0x000000011959a650 frame #410: 0x000000011956fcf0 frame #411: 0x00000001195734f4 frame #412: 0x000000011959a7d8 frame #413: 0x0000000119573328 frame #414: 0x000000011958c96c frame #415: 0x000000011958c13c frame #416: 0x000000011959a650 frame #417: 0x000000011956fcf0 frame #418: 0x00000001195734f4 frame #419: 0x000000011959a7d8 frame #420: 0x0000000119573328 frame #421: 0x000000011958c96c frame #422: 0x000000011958c13c frame #423: 0x000000011959a650 frame #424: 0x000000011956fcf0 frame #425: 0x00000001195734f4 frame #426: 0x000000011959a7d8 frame #427: 0x0000000119573328 frame #428: 0x000000011958c96c frame #429: 0x000000011958c13c frame #430: 0x000000011959a650 frame #431: 0x000000011956fcf0 frame #432: 0x00000001195734f4 frame #433: 0x000000011959a7d8 frame #434: 0x0000000119573328 frame #435: 0x000000011958c96c frame #436: 0x000000011958c13c frame #437: 0x000000011959a650 frame #438: 0x000000011956fcf0 frame #439: 0x00000001195734f4 frame #440: 0x000000011959a7d8 frame #441: 0x0000000119573328 frame #442: 0x000000011958c96c frame #443: 0x000000011958c13c frame #444: 0x000000011959a650 frame #445: 0x000000011956fcf0 frame #446: 0x00000001195734f4 frame #447: 0x000000011959a7d8 frame #448: 0x0000000119573328 frame #449: 0x000000011958c96c frame #450: 0x000000011958c13c frame #451: 0x000000011959a650 frame #452: 0x000000011956fcf0 frame #453: 0x00000001195734f4 frame #454: 0x000000011959a7d8 frame #455: 0x0000000119573328 frame #456: 0x000000011958c96c frame #457: 0x000000011958c13c frame #458: 0x000000011959a650 frame #459: 0x000000011956fcf0 frame #460: 0x00000001195734f4 frame #461: 0x000000011959a7d8 frame #462: 0x0000000119573328 frame #463: 0x000000011958c96c frame #464: 0x000000011958c13c frame #465: 0x000000011959a650 frame #466: 0x000000011956fcf0 frame #467: 0x00000001195734f4 frame #468: 0x000000011959a7d8 frame #469: 0x0000000119573328 frame #470: 0x000000011958c96c frame #471: 0x000000011958c13c frame #472: 0x000000011959a650 frame #473: 0x000000011956fcf0 frame #474: 0x00000001195734f4 frame #475: 0x000000011959a7d8 frame #476: 0x0000000119573328 frame #477: 0x000000011958c96c frame #478: 0x000000011958c13c frame #479: 0x000000011959a650 frame #480: 0x000000011956fcf0 frame #481: 0x00000001195734f4 frame #482: 0x000000011959a7d8 frame #483: 0x0000000119573328 frame #484: 0x000000011958c96c frame #485: 0x000000011958c13c frame #486: 0x000000011959a650 frame #487: 0x000000011956fcf0 frame #488: 0x00000001195734f4 frame #489: 0x000000011959a7d8 frame #490: 0x0000000119573328 frame #491: 0x000000011958c96c frame #492: 0x000000011958c13c frame #493: 0x000000011959a650 frame #494: 0x000000011956fcf0 frame #495: 0x00000001195734f4 frame #496: 0x000000011959a7d8 frame #497: 0x0000000119573328 frame #498: 0x000000011958c96c frame #499: 0x000000011958c13c frame #500: 0x000000011959a650 frame #501: 0x000000011956fcf0 frame #502: 0x00000001195734f4 frame #503: 0x000000011959a7d8 frame #504: 0x0000000119573328 frame #505: 0x000000011958c96c frame #506: 0x000000011958c13c frame #507: 0x000000011959a650 frame #508: 0x000000011956fcf0 frame #509: 0x00000001195734f4 frame #510: 0x000000011959a7d8 frame #511: 0x0000000119573328 frame #512: 0x000000011958c96c frame #513: 0x000000011958c13c frame #514: 0x000000011959a650 frame #515: 0x000000011956fcf0 frame #516: 0x00000001195734f4 frame #517: 0x000000011959a7d8 frame #518: 0x0000000119573328 frame #519: 0x000000011958c96c frame #520: 0x000000011958c13c frame #521: 0x000000011959a650 frame #522: 0x000000011956fcf0 frame #523: 0x00000001195734f4 frame #524: 0x000000011959a7d8 frame #525: 0x0000000119573328 frame #526: 0x000000011958c96c frame #527: 0x000000011958c13c frame #528: 0x000000011959a650 frame #529: 0x000000011956fcf0 frame #530: 0x00000001195734f4 frame #531: 0x000000011959a7d8 frame #532: 0x0000000119573328 frame #533: 0x000000011958c96c frame #534: 0x000000011958c13c frame #535: 0x000000011959a650 frame #536: 0x000000011956fcf0 frame #537: 0x00000001195734f4 frame #538: 0x000000011959a7d8 frame #539: 0x0000000119573328 frame #540: 0x000000011958c96c frame #541: 0x000000011958c13c frame #542: 0x000000011959a650 frame #543: 0x000000011956fcf0 frame #544: 0x00000001195734f4 frame #545: 0x000000011959a7d8 frame #546: 0x0000000119573328 frame #547: 0x000000011958c96c frame #548: 0x000000011958c13c frame #549: 0x000000011959a650 frame #550: 0x000000011956fcf0 frame #551: 0x00000001195734f4 frame #552: 0x000000011959a7d8 frame #553: 0x0000000119573328 frame #554: 0x000000011958c96c frame #555: 0x000000011958c13c frame #556: 0x000000011959a650 frame #557: 0x000000011956fcf0 frame #558: 0x00000001195734f4 frame #559: 0x000000011959a7d8 frame #560: 0x0000000119573328 frame #561: 0x000000011958c96c frame #562: 0x000000011958c13c frame #563: 0x000000011959a650 frame #564: 0x000000011956fcf0 frame #565: 0x00000001195734f4 frame #566: 0x000000011959a7d8 frame #567: 0x0000000119573328 frame #568: 0x000000011958c96c frame #569: 0x000000011958c13c frame #570: 0x000000011959a650 frame #571: 0x000000011956fcf0 frame #572: 0x00000001195734f4 frame #573: 0x000000011959a7d8 frame #574: 0x0000000119573328 frame #575: 0x000000011958c96c frame #576: 0x000000011958c13c frame #577: 0x000000011959a650 frame #578: 0x000000011956fcf0 frame #579: 0x00000001195734f4 frame #580: 0x000000011959a7d8 frame #581: 0x0000000119573328 frame #582: 0x000000011958c96c frame #583: 0x000000011958c13c frame #584: 0x000000011959a650 frame #585: 0x000000011956fcf0 frame #586: 0x00000001195734f4 frame #587: 0x000000011959a7d8 frame #588: 0x0000000119573328 frame #589: 0x000000011958c96c frame #590: 0x000000011958c13c frame #591: 0x000000011959a650 frame #592: 0x000000011956fcf0 frame #593: 0x00000001195734f4 frame #594: 0x000000011959a7d8 frame #595: 0x0000000119573328 frame #596: 0x000000011958c96c frame #597: 0x000000011958c13c frame #598: 0x000000011959a650 frame #599: 0x000000011956fcf0 frame #600: 0x00000001195734f4 frame #601: 0x000000011959a7d8 frame #602: 0x0000000119573328 frame #603: 0x000000011958c96c frame #604: 0x000000011958c13c frame #605: 0x000000011959a650 frame #606: 0x000000011956fcf0 frame #607: 0x00000001195734f4 frame #608: 0x000000011959a7d8 frame #609: 0x0000000119573328 frame #610: 0x000000011958c96c frame #611: 0x000000011958c13c frame #612: 0x000000011959a650 frame #613: 0x000000011956fcf0 frame #614: 0x00000001195734f4 frame #615: 0x000000011959a7d8 frame #616: 0x0000000119573328 frame #617: 0x000000011958c96c frame #618: 0x000000011958c13c frame #619: 0x000000011959a650 frame #620: 0x000000011956fcf0 frame #621: 0x00000001195734f4 frame #622: 0x000000011959a7d8 frame #623: 0x0000000119573328 frame #624: 0x000000011958c96c frame #625: 0x000000011958c13c frame #626: 0x000000011959a650 frame #627: 0x000000011956fcf0 frame #628: 0x00000001195734f4 frame #629: 0x000000011959a7d8 frame #630: 0x0000000119573328 frame #631: 0x000000011958c96c frame #632: 0x000000011958c13c frame #633: 0x000000011959a650 frame #634: 0x000000011956fcf0 frame #635: 0x00000001195734f4 frame #636: 0x000000011959a7d8 frame #637: 0x0000000119573328 frame #638: 0x000000011958c96c frame #639: 0x000000011958c13c frame #640: 0x000000011959a650 frame #641: 0x000000011956fcf0 frame #642: 0x00000001195734f4 frame #643: 0x000000011959a7d8 frame #644: 0x0000000119573328 frame #645: 0x000000011958c96c frame #646: 0x000000011958c13c frame #647: 0x000000011959a650 frame #648: 0x000000011956fcf0 frame #649: 0x00000001195734f4 frame #650: 0x000000011959a7d8 frame #651: 0x0000000119573328 frame #652: 0x000000011958c96c frame #653: 0x000000011958c13c frame #654: 0x000000011959a650 frame #655: 0x000000011956fcf0 frame #656: 0x00000001195734f4 frame #657: 0x000000011959a7d8 frame #658: 0x0000000119573328 frame #659: 0x000000011958c96c frame #660: 0x000000011958c13c frame #661: 0x000000011959a650 frame #662: 0x000000011956fcf0 frame #663: 0x00000001195734f4 frame #664: 0x000000011959a7d8 frame #665: 0x0000000119573328 frame #666: 0x000000011958c96c frame #667: 0x000000011958c13c frame #668: 0x000000011959a650 frame #669: 0x000000011956fcf0 frame #670: 0x00000001195734f4 frame #671: 0x000000011959a7d8 frame #672: 0x0000000119573328 frame #673: 0x000000011958c96c frame #674: 0x000000011958c13c frame #675: 0x000000011959a650 frame #676: 0x000000011956fcf0 frame #677: 0x00000001195734f4 frame #678: 0x000000011959a7d8 frame #679: 0x0000000119573328 frame #680: 0x000000011958c96c frame #681: 0x000000011958c13c frame #682: 0x000000011959a650 frame #683: 0x000000011956fcf0 frame #684: 0x00000001195734f4 frame #685: 0x000000011959a7d8 frame #686: 0x0000000119573328 frame #687: 0x000000011958c96c frame #688: 0x000000011958c13c frame #689: 0x000000011959a650 frame #690: 0x000000011956fcf0 frame #691: 0x00000001195734f4 frame #692: 0x000000011959a7d8 frame #693: 0x0000000119573328 frame #694: 0x000000011958c96c frame #695: 0x000000011958c13c frame #696: 0x000000011959a650 frame #697: 0x000000011956fcf0 frame #698: 0x00000001195734f4 frame #699: 0x000000011959a7d8 frame #700: 0x0000000119573328 frame #701: 0x000000011958c96c frame #702: 0x000000011958c13c frame #703: 0x000000011959a650 frame #704: 0x000000011956fcf0 frame #705: 0x00000001195734f4 frame #706: 0x000000011959a7d8 frame #707: 0x0000000119573328 frame #708: 0x000000011958c96c frame #709: 0x000000011958c13c frame #710: 0x000000011959a650 frame #711: 0x000000011956fcf0 frame #712: 0x00000001195734f4 frame #713: 0x000000011959a7d8 frame #714: 0x0000000119573328 frame #715: 0x000000011958c96c frame #716: 0x000000011958c13c frame #717: 0x000000011959a650 frame #718: 0x000000011956fcf0 frame #719: 0x00000001195734f4 frame #720: 0x000000011959a7d8 frame #721: 0x0000000119573328 frame #722: 0x000000011958c96c frame #723: 0x000000011958c13c frame #724: 0x000000011959a650 frame #725: 0x000000011956fcf0 frame #726: 0x00000001195734f4 frame #727: 0x000000011959a7d8 frame #728: 0x0000000119573328 frame #729: 0x000000011958c96c frame #730: 0x000000011958c13c frame #731: 0x000000011959a650 frame #732: 0x000000011956fcf0 frame #733: 0x00000001195734f4 frame #734: 0x000000011959a7d8 frame #735: 0x0000000119573328 frame #736: 0x000000011958c96c frame #737: 0x000000011958c13c frame #738: 0x000000011959a650 frame #739: 0x000000011956fcf0 frame #740: 0x00000001195734f4 frame #741: 0x000000011959a7d8 frame #742: 0x0000000119573328 frame #743: 0x000000011958c96c frame #744: 0x000000011958c13c frame #745: 0x000000011959a650 frame #746: 0x000000011956fcf0 frame #747: 0x00000001195734f4 frame #748: 0x000000011959a7d8 frame #749: 0x0000000119573328 frame #750: 0x000000011958c96c frame #751: 0x000000011958c13c frame #752: 0x000000011959a650 frame #753: 0x000000011956fcf0 frame #754: 0x00000001195734f4 frame #755: 0x000000011959a7d8 frame #756: 0x0000000119573328 frame #757: 0x000000011958c96c frame #758: 0x000000011958c13c frame #759: 0x000000011959a650 frame #760: 0x000000011956fcf0 frame #761: 0x00000001195734f4 frame #762: 0x000000011959a7d8 frame #763: 0x0000000119573328 frame #764: 0x000000011958c96c frame #765: 0x000000011958c13c frame #766: 0x000000011959a650 frame #767: 0x000000011956fcf0 frame #768: 0x00000001195734f4 frame #769: 0x000000011959a7d8 frame #770: 0x0000000119573328 frame #771: 0x000000011958c96c frame #772: 0x000000011958c13c frame #773: 0x000000011959a650 frame #774: 0x000000011956fcf0 frame #775: 0x00000001195734f4 frame #776: 0x000000011959a7d8 frame #777: 0x0000000119573328 frame #778: 0x000000011958c96c frame #779: 0x000000011958c13c frame #780: 0x000000011959a650 frame #781: 0x000000011956fcf0 frame #782: 0x00000001195734f4 frame #783: 0x000000011959a7d8 frame #784: 0x0000000119573328 frame #785: 0x000000011958c96c frame #786: 0x000000011958c13c frame #787: 0x000000011959a650 frame #788: 0x000000011956fcf0 frame #789: 0x00000001195734f4 frame #790: 0x000000011959a7d8 frame #791: 0x0000000119573328 frame #792: 0x000000011958c96c frame #793: 0x000000011958c13c frame #794: 0x000000011959a650 frame #795: 0x000000011956fcf0 frame #796: 0x00000001195734f4 frame #797: 0x000000011959a7d8 frame #798: 0x0000000119573328 frame #799: 0x000000011958c96c frame #800: 0x000000011958c13c frame #801: 0x000000011959a650 frame #802: 0x000000011956fcf0 frame #803: 0x00000001195734f4 frame #804: 0x000000011959a7d8 frame #805: 0x0000000119573328 frame #806: 0x000000011958c96c frame #807: 0x000000011958c13c frame #808: 0x000000011959a650 frame #809: 0x000000011956fcf0 frame #810: 0x00000001195734f4 frame #811: 0x000000011959a7d8 frame #812: 0x0000000119573328 frame #813: 0x000000011958c96c frame #814: 0x000000011958c13c frame #815: 0x000000011959a650 frame #816: 0x000000011956fcf0 frame #817: 0x00000001195734f4 frame #818: 0x000000011959a7d8 frame #819: 0x0000000119573328 frame #820: 0x000000011958c96c frame #821: 0x000000011958c13c frame #822: 0x000000011959a650 frame #823: 0x000000011956fcf0 frame #824: 0x00000001195734f4 frame #825: 0x000000011959a7d8 frame #826: 0x0000000119573328 frame #827: 0x000000011958c96c frame #828: 0x000000011958c13c frame #829: 0x000000011959a650 frame #830: 0x000000011956fcf0 frame #831: 0x00000001195734f4 frame #832: 0x000000011959a7d8 frame #833: 0x0000000119573328 frame #834: 0x000000011958c96c frame #835: 0x000000011958c13c frame #836: 0x000000011959a650 frame #837: 0x000000011956fcf0 frame #838: 0x00000001195734f4 frame #839: 0x000000011959a7d8 frame #840: 0x0000000119573328 frame #841: 0x000000011958c96c frame #842: 0x000000011958c13c frame #843: 0x000000011959a650 frame #844: 0x000000011956fcf0 frame #845: 0x00000001195734f4 frame #846: 0x000000011959a7d8 frame #847: 0x0000000119573328 frame #848: 0x000000011958c96c frame #849: 0x000000011958c13c frame #850: 0x000000011959a650 frame #851: 0x000000011956fcf0 frame #852: 0x00000001195734f4 frame #853: 0x000000011959a7d8 frame #854: 0x0000000119573328 frame #855: 0x000000011958c96c frame #856: 0x000000011958c13c frame #857: 0x000000011959a650 frame #858: 0x000000011956fcf0 frame #859: 0x00000001195734f4 frame #860: 0x000000011959a7d8 frame #861: 0x0000000119573328 frame #862: 0x000000011958c96c frame #863: 0x000000011958c13c frame #864: 0x000000011959a650 frame #865: 0x000000011956fcf0 frame #866: 0x00000001195734f4 frame #867: 0x000000011959a7d8 frame #868: 0x0000000119573328 frame #869: 0x000000011958c96c frame #870: 0x000000011958c13c frame #871: 0x000000011959a650 frame #872: 0x000000011956fcf0 frame #873: 0x00000001195734f4 frame #874: 0x000000011959a7d8 frame #875: 0x0000000119573328 frame #876: 0x000000011958c96c frame #877: 0x000000011958c13c frame #878: 0x000000011959a650 frame #879: 0x000000011956fcf0 frame #880: 0x00000001195734f4 frame #881: 0x000000011959a7d8 frame #882: 0x0000000119573328 frame #883: 0x000000011958c96c frame #884: 0x000000011958c13c frame #885: 0x000000011959a650 frame #886: 0x000000011956fcf0 frame #887: 0x00000001195734f4 frame #888: 0x000000011959a7d8 frame #889: 0x0000000119573328 frame #890: 0x000000011958c96c frame #891: 0x000000011958c13c frame #892: 0x000000011959a650 frame #893: 0x000000011956fcf0 frame #894: 0x00000001195734f4 frame #895: 0x000000011959a7d8 frame #896: 0x0000000119573328 frame #897: 0x000000011958c96c frame #898: 0x000000011958c13c frame #899: 0x000000011959a650 frame #900: 0x000000011956fcf0 frame #901: 0x00000001195734f4 frame #902: 0x000000011959a7d8 frame #903: 0x0000000119573328 frame #904: 0x000000011958c96c frame #905: 0x000000011958c13c frame #906: 0x000000011959a650 frame #907: 0x000000011956fcf0 frame #908: 0x00000001195734f4 frame #909: 0x000000011959a7d8 frame #910: 0x0000000119573328 frame #911: 0x000000011958c96c frame #912: 0x000000011958c13c frame #913: 0x000000011959a650 frame #914: 0x000000011956fcf0 frame #915: 0x00000001195734f4 frame #916: 0x000000011959a7d8 frame #917: 0x0000000119573328 frame #918: 0x000000011958c96c frame #919: 0x000000011958c13c frame #920: 0x000000011959a650 frame #921: 0x000000011956fcf0 frame #922: 0x00000001195734f4 frame #923: 0x000000011959a7d8 frame #924: 0x0000000119573328 frame #925: 0x000000011958c96c frame #926: 0x000000011958c13c frame #927: 0x000000011959a650 frame #928: 0x000000011956fcf0 frame #929: 0x00000001195734f4 frame #930: 0x000000011959a7d8 frame #931: 0x0000000119573328 frame #932: 0x000000011958c96c frame #933: 0x000000011958c13c frame #934: 0x000000011959a650 frame #935: 0x000000011956fcf0 frame #936: 0x00000001195734f4 frame #937: 0x000000011959a7d8 frame #938: 0x0000000119573328 frame #939: 0x000000011958c96c frame #940: 0x000000011958c13c frame #941: 0x000000011959a650 frame #942: 0x000000011956fcf0 frame #943: 0x00000001195734f4 frame #944: 0x000000011959a7d8 frame #945: 0x0000000119573328 frame #946: 0x000000011958c96c frame #947: 0x000000011958c13c frame #948: 0x000000011959a650 frame #949: 0x000000011956fcf0 frame #950: 0x00000001195734f4 frame #951: 0x000000011959a7d8 frame #952: 0x0000000119573328 frame #953: 0x000000011958c96c frame #954: 0x000000011958c13c frame #955: 0x000000011959a650 frame #956: 0x000000011956fcf0 frame #957: 0x00000001195734f4 frame #958: 0x000000011959a7d8 frame #959: 0x0000000119573328 frame #960: 0x000000011958c96c frame #961: 0x000000011958c13c frame #962: 0x000000011959a650 frame #963: 0x000000011956fcf0 frame #964: 0x00000001195734f4 frame #965: 0x000000011959a7d8 frame #966: 0x0000000119573328 frame #967: 0x000000011958c96c frame #968: 0x000000011958c13c frame #969: 0x000000011959a650 frame #970: 0x000000011956fcf0 frame #971: 0x00000001195734f4 frame #972: 0x000000011959a7d8 frame #973: 0x0000000119573328 frame #974: 0x000000011958c96c frame #975: 0x000000011958c13c frame #976: 0x000000011959a650 frame #977: 0x000000011956fcf0 frame #978: 0x00000001195734f4 frame #979: 0x000000011959a7d8 frame #980: 0x0000000119573328 frame #981: 0x000000011958c96c frame #982: 0x000000011958c13c frame #983: 0x000000011959a650 frame #984: 0x000000011956fcf0 frame #985: 0x00000001195734f4 frame #986: 0x000000011959a7d8 frame #987: 0x0000000119573328 frame #988: 0x000000011958c96c frame #989: 0x000000011958c13c frame #990: 0x000000011959a650 frame #991: 0x000000011956fcf0 frame #992: 0x00000001195734f4 frame #993: 0x000000011959a7d8 frame #994: 0x0000000119573328 frame #995: 0x000000011958c96c frame #996: 0x000000011958c13c frame #997: 0x000000011959a650 frame #998: 0x000000011956fcf0 frame #999: 0x00000001195734f4 frame #1000: 0x000000011959a7d8 frame #1001: 0x0000000119573328 frame #1002: 0x000000011958c96c frame #1003: 0x000000011958c13c frame #1004: 0x000000011959a650 frame #1005: 0x000000011956fcf0 frame #1006: 0x00000001195734f4 frame #1007: 0x000000011959a7d8 frame #1008: 0x0000000119573328 frame #1009: 0x000000011958c96c frame #1010: 0x000000011958c13c frame #1011: 0x000000011959a650 frame #1012: 0x000000011956fcf0 frame #1013: 0x00000001195734f4 frame #1014: 0x000000011959a7d8 frame #1015: 0x0000000119573328 frame #1016: 0x000000011958c96c frame #1017: 0x000000011958c13c frame #1018: 0x000000011959a650 frame #1019: 0x000000011956fcf0 frame #1020: 0x00000001195734f4 frame #1021: 0x000000011959a7d8 frame #1022: 0x0000000119573328 frame #1023: 0x000000011958c96c frame #1024: 0x000000011958c13c frame #1025: 0x000000011959a650 frame #1026: 0x000000011956fcf0 frame #1027: 0x00000001195734f4 frame #1028: 0x000000011959a7d8 frame #1029: 0x0000000119573328 frame #1030: 0x000000011958c96c frame #1031: 0x000000011958c13c frame #1032: 0x000000011959a650 frame #1033: 0x000000011956fcf0 frame #1034: 0x00000001195734f4 frame #1035: 0x000000011959a7d8 frame #1036: 0x0000000119573328 frame #1037: 0x000000011958c96c frame #1038: 0x000000011958c13c frame #1039: 0x000000011959a650 frame #1040: 0x000000011956fcf0 frame #1041: 0x00000001195734f4 frame #1042: 0x000000011959a7d8 frame #1043: 0x0000000119573328 frame #1044: 0x000000011958c96c frame #1045: 0x000000011958c13c frame #1046: 0x000000011959a650 frame #1047: 0x000000011956fcf0 frame #1048: 0x00000001195734f4 frame #1049: 0x000000011959a7d8 frame #1050: 0x0000000119573328 frame #1051: 0x000000011958c96c frame #1052: 0x000000011958c13c frame #1053: 0x000000011959a650 frame #1054: 0x000000011956fcf0 frame #1055: 0x00000001195734f4 frame #1056: 0x000000011959a7d8 frame #1057: 0x0000000119573328 frame #1058: 0x000000011958c96c frame #1059: 0x000000011958c13c frame #1060: 0x000000011959a650 frame #1061: 0x000000011956fcf0 frame #1062: 0x00000001195734f4 frame #1063: 0x000000011959a7d8 frame #1064: 0x0000000119573328 frame #1065: 0x000000011958c96c frame #1066: 0x000000011958c13c frame #1067: 0x000000011959a650 frame #1068: 0x000000011956fcf0 frame #1069: 0x00000001195734f4 frame #1070: 0x000000011959a7d8 frame #1071: 0x0000000119573328 frame #1072: 0x000000011958c96c frame #1073: 0x000000011958c13c frame #1074: 0x000000011959a650 frame #1075: 0x000000011956fcf0 frame #1076: 0x00000001195734f4 frame #1077: 0x000000011959a7d8 frame #1078: 0x0000000119573328 frame #1079: 0x000000011958c96c frame #1080: 0x000000011958c13c frame #1081: 0x000000011959a650 frame #1082: 0x000000011956fcf0 frame #1083: 0x00000001195734f4 frame #1084: 0x000000011959a7d8 frame #1085: 0x0000000119573328 frame #1086: 0x000000011958c96c frame #1087: 0x000000011958c13c frame #1088: 0x000000011959a650 frame #1089: 0x000000011956fcf0 frame #1090: 0x00000001195734f4 frame #1091: 0x000000011959a7d8 frame #1092: 0x0000000119573328 frame #1093: 0x000000011958c96c frame #1094: 0x000000011958c13c frame #1095: 0x000000011959a650 frame #1096: 0x000000011956fcf0 frame #1097: 0x00000001195734f4 frame #1098: 0x000000011959a7d8 frame #1099: 0x0000000119573328 frame #1100: 0x000000011958c96c frame #1101: 0x000000011958c13c frame #1102: 0x000000011959a650 frame #1103: 0x000000011956fcf0 frame #1104: 0x00000001195734f4 frame #1105: 0x000000011959a7d8 frame #1106: 0x0000000119573328 frame #1107: 0x000000011958c96c frame #1108: 0x000000011958c13c frame #1109: 0x000000011959a650 frame #1110: 0x000000011956fcf0 frame #1111: 0x00000001195734f4 frame #1112: 0x000000011959a7d8 frame #1113: 0x0000000119573328 frame #1114: 0x000000011958c96c frame #1115: 0x000000011958c13c frame #1116: 0x000000011959a650 frame #1117: 0x000000011956fcf0 frame #1118: 0x00000001195734f4 frame #1119: 0x000000011959a7d8 frame #1120: 0x0000000119573328 frame #1121: 0x000000011958c96c frame #1122: 0x000000011958c13c frame #1123: 0x000000011959a650 frame #1124: 0x000000011956fcf0 frame #1125: 0x00000001195734f4 frame #1126: 0x000000011959a7d8 frame #1127: 0x0000000119573328 frame #1128: 0x000000011958c96c frame #1129: 0x000000011958c13c frame #1130: 0x000000011959a650 frame #1131: 0x000000011956fcf0 frame #1132: 0x00000001195734f4 frame #1133: 0x000000011959a7d8 frame #1134: 0x0000000119573328 frame #1135: 0x000000011958c96c frame #1136: 0x000000011958c13c frame #1137: 0x000000011959a650 frame #1138: 0x000000011956fcf0 frame #1139: 0x00000001195734f4 frame #1140: 0x000000011959a7d8 frame #1141: 0x0000000119573328 frame #1142: 0x000000011958c96c frame #1143: 0x000000011958c13c frame #1144: 0x000000011959a650 frame #1145: 0x000000011956fcf0 frame #1146: 0x00000001195734f4 frame #1147: 0x000000011959a7d8 frame #1148: 0x0000000119573328 frame #1149: 0x000000011958c96c frame #1150: 0x000000011958c13c frame #1151: 0x000000011959a650 frame #1152: 0x000000011956fcf0 frame #1153: 0x00000001195734f4 frame #1154: 0x000000011959a7d8 frame #1155: 0x0000000119573328 frame #1156: 0x000000011958c96c frame #1157: 0x000000011958c13c frame #1158: 0x000000011959a650 frame #1159: 0x000000011956fcf0 frame #1160: 0x00000001195734f4 frame #1161: 0x000000011959a7d8 frame #1162: 0x0000000119573328 frame #1163: 0x000000011958c96c frame #1164: 0x000000011958c13c frame #1165: 0x000000011959a650 frame #1166: 0x000000011956fcf0 frame #1167: 0x00000001195734f4 frame #1168: 0x000000011959a7d8 frame #1169: 0x0000000119573328 frame #1170: 0x000000011958c96c frame #1171: 0x000000011958c13c frame #1172: 0x000000011959a650 frame #1173: 0x000000011956fcf0 frame #1174: 0x00000001195734f4 frame #1175: 0x000000011959a7d8 frame #1176: 0x0000000119573328 frame #1177: 0x000000011958c96c frame #1178: 0x000000011958c13c frame #1179: 0x000000011959a650 frame #1180: 0x000000011956fcf0 frame #1181: 0x00000001195734f4 frame #1182: 0x000000011959a7d8 frame #1183: 0x0000000119573328 frame #1184: 0x000000011958c96c frame #1185: 0x000000011958c13c frame #1186: 0x000000011959a650 frame #1187: 0x000000011956fcf0 frame #1188: 0x00000001195734f4 frame #1189: 0x000000011959a7d8 frame #1190: 0x0000000119573328 frame #1191: 0x000000011958c96c frame #1192: 0x000000011958c13c frame #1193: 0x000000011959a650 frame #1194: 0x000000011956fcf0 frame #1195: 0x00000001195734f4 frame #1196: 0x000000011959a7d8 frame #1197: 0x0000000119573328 frame #1198: 0x000000011958c96c frame #1199: 0x000000011958c13c frame #1200: 0x000000011959a650 frame #1201: 0x000000011956fcf0 frame #1202: 0x00000001195734f4 frame #1203: 0x000000011959a7d8 frame #1204: 0x0000000119573328 frame #1205: 0x000000011958c96c frame #1206: 0x000000011958c13c frame #1207: 0x000000011959a650 frame #1208: 0x000000011956fcf0 frame #1209: 0x00000001195734f4 frame #1210: 0x000000011959a7d8 frame #1211: 0x0000000119573328 frame #1212: 0x000000011958c96c frame #1213: 0x000000011958c13c frame #1214: 0x000000011959a650 frame #1215: 0x000000011956fcf0 frame #1216: 0x00000001195734f4 frame #1217: 0x000000011959a7d8 frame #1218: 0x0000000119573328 frame #1219: 0x000000011958c96c frame #1220: 0x000000011958c13c frame #1221: 0x000000011959a650 frame #1222: 0x000000011956fcf0 frame #1223: 0x00000001195734f4 frame #1224: 0x000000011959a7d8 frame #1225: 0x0000000119573328 frame #1226: 0x000000011958c96c frame #1227: 0x000000011958c13c frame #1228: 0x000000011959a650 frame #1229: 0x000000011956fcf0 frame #1230: 0x00000001195734f4 frame #1231: 0x000000011959a7d8 frame #1232: 0x0000000119573328 frame #1233: 0x000000011958c96c frame #1234: 0x000000011958c13c frame #1235: 0x000000011959a650 frame #1236: 0x000000011956fcf0 frame #1237: 0x00000001195734f4 frame #1238: 0x000000011959a7d8 frame #1239: 0x0000000119573328 frame #1240: 0x000000011958c96c frame #1241: 0x000000011958c13c frame #1242: 0x000000011959a650 frame #1243: 0x000000011956fcf0 frame #1244: 0x00000001195734f4 frame #1245: 0x000000011959a7d8 frame #1246: 0x0000000119573328 frame #1247: 0x000000011958c96c frame #1248: 0x000000011958c13c frame #1249: 0x000000011959a650 frame #1250: 0x000000011956fcf0 frame #1251: 0x00000001195734f4 frame #1252: 0x000000011959a7d8 frame #1253: 0x0000000119573328 frame #1254: 0x000000011958c96c frame #1255: 0x000000011958c13c frame #1256: 0x000000011959a650 frame #1257: 0x000000011956fcf0 frame #1258: 0x00000001195734f4 frame #1259: 0x000000011959a7d8 frame #1260: 0x0000000119573328 frame #1261: 0x000000011958c96c frame #1262: 0x000000011958c13c frame #1263: 0x000000011959a650 frame #1264: 0x000000011956fcf0 frame #1265: 0x00000001195734f4 frame #1266: 0x000000011959a7d8 frame #1267: 0x0000000119573328 frame #1268: 0x000000011958c96c frame #1269: 0x000000011958c13c frame #1270: 0x000000011959a650 frame #1271: 0x000000011956fcf0 frame #1272: 0x00000001195734f4 frame #1273: 0x000000011959a7d8 frame #1274: 0x0000000119573328 frame #1275: 0x000000011958c96c frame #1276: 0x000000011958c13c frame #1277: 0x000000011959a650 frame #1278: 0x000000011956fcf0 frame #1279: 0x000000011957fc1c frame #1280: 0x000000011957f740 frame #1281: 0x000000011957b63c frame #1282: 0x0000000119556c30 frame #1283: 0x0000000119555b74 frame #1284: 0x0000000119586468 frame #1285: 0x000000011955a7ac frame #1286: 0x0000000119556904 frame #1287: 0x0000000119555b74 frame #1288: 0x00000001195535fc frame #1289: 0x0000000118f66704 frame #1290: 0x0000000118f662f0 frame #1291: 0x0000000119552e88 frame #1292: 0x0000000118f55920 frame #1293: 0x0000000118f5518c frame #1294: 0x0000000118f54dec frame #1295: 0x0000000118f27924 frame #1296: 0x0000000118f2724c frame #1297: 0x0000000118f26de8 frame #1298: 0x0000000118f266d4 frame #1299: 0x0000000118f22ce0 frame #1300: 0x000000011a0da9b8 frame #1301: 0x00000001018f9084 libcoreclr.dylib`CallDescrWorkerInternal + 132 frame #1302: 0x0000000101775eec libcoreclr.dylib`DispatchCallSimple(unsigned long*, unsigned int, unsigned long long, unsigned int) + 268 frame #1303: 0x00000001017881a8 libcoreclr.dylib`ThreadNative::KickOffThread_Worker(void*) + 148 frame #1304: 0x0000000101749910 libcoreclr.dylib`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 256 frame #1305: 0x0000000101749e0c libcoreclr.dylib`ManagedThreadBase::KickOff(void (*)(void*), void*) + 32 frame #1306: 0x0000000101788280 libcoreclr.dylib`ThreadNative::KickOffThread(void*) + 172 frame #1307: 0x0000000101664750 libcoreclr.dylib`CorUnix::CPalThread::ThreadEntry(void*) + 380 frame #1308: 0x000000018946ffa8 libsystem_pthread.dylib`_pthread_start + 148 ```

[hoyosjs]

@EgorBo do you have a copy of the dump?

[EgorBo]

Perhaps, Roslyn should spawn threads with custom size of stack?

It won't fix the problem with compilation of deeply nested statements failing intermittently due to non-determinism of PGO and tiered compilation. It will just move the threshold where the compilation becomes unreliable.

Sure, I just want to make sure that none of methods in the trace have unusual stack consumption due to PGO. What I'm saying that 236kb of stack just sounds too low for a code compilation task that involves recursions

@EgorBo do you have a copy of the dump?

I guess I can just follow your advice and compile lldb from sources locally to get sos working

[EgorBo]

Yes, I was able to load SOS with locally built LLDB (from https://github.com/llvm/llvm-project)

[EgorBo]

Here is the managed stack with stack pointers - the biggest delta between two pointers for managed calls is 544 bytes so nothing unusual, PGO also only contributes a few kb on top of it.

https://gist.github.com/EgorBo/ddf3d254d840bd61174b43d3527929fb

[EgorBo]

Summary: The problem looks to be macOS specific, when Roslyn compiles one of these tests it consumes too much of stack and fails (it doesn't crash with the unrecoverable StackOverflow because it relies on EnsureSufficientExecutionStack() API).

• HugeArray1.cs
• hugeSimpleExpr1.cs
• RngchkStress2.cs

The typical default stack size for 64bit OSes looks like this:	OS	Main thread	Secondary threads
Windows	1.5Mb*	1.5Mb*
Linux	8Mb	8Mb
macOS	8Mb	512Kb

(* - overriden by .NET) These thresholds may vary (esp on linux) or can be configured OS-wise. There is also ability to se stack size directly via new Thread(..., stackSize) or globally via DOTNET_DefaultStackSize and DEFAULT_STACK_SIZE)

So, for macOS we only have 512Kb of stack, 128Kb are reseved to be able to show traces for SO (we only get 384kb here with m_CacheStackBase - m_CacheStackLimit), then EnsureSufficientExecutionStack() requires at least 128kb of stack available. Then, VM's functions (e.g. to kick off a new thread) may occupy several kbs on the call stack so we're left with only ~230kb of stack for Roslyn to do the job. It looks like PGO increases stack consumption for some functions but nothing unusual (none of functions consume more than 544 bytes of stack).

Possible actions: 1) Disable these tests on macOS only (some of them are currently disabled globally) 2) Use e.g. DOTNET_DefaultStackSize=0x100000 (1Mb) for macOS in those tests (tricky for tests groups?) 3) Simplify them (not sure how do that for HugeArray1.cs and hugeSimpleExpr1.cs) 4) Roslyn might want to increase stack size for macOS too - ~230kb sounds too low for such a heavy task 5) Propose an API RuntimeHelpers.EnsureSufficientExecutionStack(long size) to override the default, a bit too conservative, 128kb limit (or request more if a user uses large stackallocs) 6) Override the default stack size on macOS to some fixed number (at least for the secondary threads)

[hamarb123]

I personally would like RuntimeHelpers.EnsureSufficientExecutionStack(long size). 128k is quite a lot, but it's probably not a good idea to change it. This way I would be able to check something more meaningful than "is there enough stack that I could run basically any main function in a sensible program which doesn't have a stackalloc bug". Ideally I would actually be able to check "is there enough room for my actual function, which I can guess probably doesn't quite need 128kB of stack".

From my experimentation and checking the PE file header, the stack size on Windows seems to be 1.5MB for .NET executables on both main and secondary threads, but for standard Windows executables it's just 1MB by default (for all threads I believe, but I've only tested main for native). Is there a reason to not just increase the secondary thread default on macOS to 1.5/1MB, since we're already not following the default on Windows? Note that the memory should only be reserved until it's used, so this shouldn't even realistically increase memory usage even.

Or we could standardise across these platforms to some specific number (e.g. 2/4/8MB), at least for the secondary threads (you can change it for primary thread on windows in the optional part of the PE header - which is how we set it presumably as the number matches, but I have no idea if similar options exist for macOS and Linux).

[jkotas]

Ideally I would actually be able to check "is there enough room for my actual function, which I can guess probably doesn't quite need 128kB of stack".

The 128kB limit is there for exception handling, garbage collection and JIT to run on the thread in addition to your function.

[jkotas]

Is there a reason to not just increase the secondary thread default on macOS to 1.5/1MB,

IIRC, it was discussed before but I am not able to find the paper trail. @janvorli Do you remember?

[MichalPetryka]

I think .Net should just guarantee some minimum stack size (like 2MB) on all platforms for consistency.

[EgorBo]

Moving to 9.0.0 as, I guess, there is nothing we can do for .NET 8.0 here?

[jaredpar]

Roslyn might want to increase stack size for macOS too - ~230kb sounds too low for such a heavy task

Disagree with this direction. What makes Roslyn special compared to any other application that uses the thread pool? Other than it's blocking a runtime test?

[jakobbotsch]

Disagree with this direction. What makes Roslyn special compared to any other application that uses the thread pool? Other than it's blocking a runtime test?

Roslyn is a bit special in this regard since it has unbounded recursion that is dependent on user provided input.

[jaredpar]

Roslyn is a bit special in this regard since it has unbounded recursion that is dependent on user provided input.

Guessing Roslyn is not alone here.

[jakobbotsch]

Guessing Roslyn is not alone here.

Agreed, but I would expect this to be a much smaller class of applications than applications that use the thread pool. In any case the to idea make the stack limits consistent across platforms seems good to me.

I would like to understand what exactly has changed since we are seeing this while building a number of different tests, even some innocent looking ones. I hit this while building https://github.com/dotnet/runtime/blob/261e3a40cd9faa240da82aacd8d821469703e339/src/tests/JIT/Methodical/VT/etc/ctor_recurse.cs locally today, and I have seen it on https://github.com/dotnet/runtime/blob/1aded3f65f6fd13a60eb43c191f4ea8aa27bf5dc/src/tests/JIT/Regression/JitBlue/Runtime_85602/Runtime_85602.cs in CI recently as well. These do not look super complex, so it's a weird coincidence if nothing major has changed.

[jaredpar]

Historically we've seen this due to a combination of factors:

1. The runtime and roslyn have a number of code samples that run right up against the limits.
2. The runtime makes changes, like improve inlining, that change stack size

Taken together this can lead to the compiler blowing its stack where it previously did not. This typically plays out in a given .NET release by the following sequence of events:

Runtime makes change that improves inlining. That change gets merged into a .NET SDK. The darc PR to merge that SDK into dotnet/roslyn gets blocked because it breaks the EndToEnd tests we have that verify our nesting limits.

Pretty sure we've seen this play out in runtime before but it's been a while.

These do not look super complex, so it's a weird coincidence if nothing major has changed.

I agree that neither of those look very complex. Those hitting the limits would, in my mind, be cause for investigation as to what changed.

FWIW the compiler actually got significantly better in .NET 8 in terms of level of nesting we can handle. In most cases it improved by 4X compared to .NET 7. So this should actually be going the other direction. Always possible you hit a nesting path we don't test though.

[EgorBo]

Minimal repro (it blows up even on Windows after certain iteration - looks like Tier0 codegen is fine):

Prerequsites:

• Remove xunit from RngchkStress2.cs
• Add <PackageReference Include="Microsoft.CodeAnalysis" Version="4.7.0-2.final" />

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Emit;

class Program
{
    const string repo = @"C:/prj/runtime";
    const string config = "windows.x64.Release";
    const string coreRoot = repo + $@"/artifacts/tests/coreclr/{config}/Tests/Core_Root/";
    const string sourceFilePath = repo + @"/src/tests/JIT/jit64/opt/rngchk/RngchkStress2.cs";

    static async Task Main()
    {
        string sourceCode = File.ReadAllText(sourceFilePath);
        for (int i = 0; i < 100; i++)
        {
            const int stackSize = 384 * 1024;
            var thread = new Thread(() => CompileToDll(sourceCode, "Output.dll"), stackSize);
            thread.Start();
            await Task.Delay(100);
            thread.Join();
        }
        Console.WriteLine("Started compilation on background thread...");
        Console.ReadLine();
    }

    private static void CompileToDll(string sourceCode, string outputDllPath)
    {
        SyntaxTree syntaxTree = CSharpSyntaxTree.ParseText(sourceCode);
        string assemblyName = Path.GetRandomFileName();
        CSharpCompilation compilation = CSharpCompilation.Create(
            assemblyName,
            syntaxTrees: new[] { syntaxTree },
            references: new[] {
                MetadataReference.CreateFromFile(coreRoot + "System.Private.CoreLib.dll"),
                MetadataReference.CreateFromFile(coreRoot + "System.Runtime.dll"),
                MetadataReference.CreateFromFile(coreRoot + "System.Console.dll")
            },
            options: new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary));
        EmitResult result = compilation.Emit(outputDllPath);
        if (result.Success)
        {
            Console.WriteLine($"DLL was successfully created at: {outputDllPath}");
        }
        else
        {
            Console.WriteLine("Compilation failed!");
            foreach (Diagnostic diagnostic in result.Diagnostics)
                if (diagnostic.Severity == DiagnosticSeverity.Error)
                    Console.WriteLine(diagnostic.GetMessage());
        }
    }
}

[EgorBo]

One thing I noticed that If I spawn a new Thread with stackSize=512 on Windows then I get (int64_t)m_CacheStackBase - (int64_t)m_CacheStackLimit; = 512kb so exactly what is requested. On macOS at the same time I get 384kb here for secondary threads when I don't specify any limit

[janvorli]

@EgorBo PAL doesn't attempt to override the default stack size except for MUSL Linuxes where it is so low that it is unusable for .NET (tens of kilobytes). But if the size is specified when creating the thread, it should be honored. I am not sure if I read your comment correctly. Did you mean that when you set stack size to 512kB on macOS, you only get 384 kB?

[EgorBo]

@EgorBo PAL doesn't attempt to override the default stack size except for MUSL Linuxes where it is so low that it is unusable for .NET (tens of kilobytes). But if the size is specified when creating the thread, it should be honored. I am not sure if I read your comment correctly. Did you mean that when you set stack size to 512kB on macOS, you only get 384 kB?

I haven't tested explicit stack size on macOS (can try once I boot to it) but the default one seems to be 384kb. I only tested the explicit on Windows and it gave me what I requested - 512kb.

I was measuring the subraction of these two on both platforms.

[janvorli]

According to Apple's doc, the default stack size for the main thread is 8MB and for the secondary threads it is 512kB. https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Multithreading/CreatingThreads/CreatingThreads.html

However, the m_CacheStackLimit on macOS for secondary threads is not necessarily reliable. We get it from the pthread_get_stacksize_np and there are reports of it giving various results based on the macOS version.

[hamarb123]

According to Apple's doc, the default stack size for the main thread is 8MB and for the secondary threads it is 512kB.

Note we already don't follow the default on Windows (we went from 1MB -> 1.5MB for all threads), I see no reason to not increase it for secondary threads on macOS also since 512kB is clearly unacceptably low when we expect 128kB to be used for stack traces, and another 128kB to be available for EnsureSufficientExecutionStack.

[hamarb123]

Would a PR to change the stack size on macOS be taken? We could also make all desktop platforms have consistent stack size if that's desired too.

[janvorli]

@hamarb123 I would be happy to accept a PR that unifies default secondary stack sizes on 64 bit platforms. Even for Windows where by an accidental change the default stack size was changed from 4MB to 1.5MB about 8 years ago (see https://github.com/dotnet/runtime/issues/96347). @jkotas I wonder if we should move all the 64 bit targets to use 4MB default stack size or be more conservative. It seems that the 1.5MB that we were using before was sufficient in most of the cases (based on the low number of reports of stack overflow issues), so maybe jumping to 4MB might be unnecessarily large step.

[jaredpar]

Lowering the stack limit on Windows would almost certainly lead to customer dissatisfaction. The C# compiler is naturally recursive and it's limits in certain areas are tied to the default thread stack size. There are a non-trivial number of customers out there who run right on the edge of what we can support today (and we have extensive tests to make sure we don't regress this experience because of previous incidents). Any reduction in stack size would end up breaking our ability to build those customers code bases.

[janvorli]

@jaredpar I was definitely not suggesting to lower the current limit, we now have it set to 1.5MB, my point was whether we should move it to 4MB or just say 2MB.

[jaredpar]

@janvorli

Gotcha

I wonder if we should move all the 64 bit targets to use 4MB default stack size or be more conservative

I was reading the second half of that and interpreting "be more conservative" as potentially considering a lower stack size. Definitely happy with any change that makes Mac bigger.

[jkotas]

@jkotas I wonder if we should move all the 64 bit targets to use 4MB default stack size or be more conservative.

If macOS is the only one giving us trouble, I would fix macOS to match Windows and leave it at that.

The max stack size is a perf trade-off. The larger max stack size, the more (unused) memory is potentially committed to stacks. Growing max stack size is potentially replacing stack overflow with out of memory.

Some algorithms (like our own async - look for TryEnsureSufficientExecutionStack) try to use the stack as much as possible and gracefully cut over when the current thread stack runs out. These algorithms will make sure that the whole available stack is committed.

[janvorli]

Good point about the async stuff committing the stack space.

[hamarb123]

@jkotas I wonder if we should move all the 64 bit targets to use 4MB default stack size or be more conservative.

If macOS is the only one giving us trouble, I would fix macOS to match Windows and leave it at that.

The max stack size is a perf trade-off. The larger max stack size, the more (unused) memory is potentially committed to stacks. Growing max stack size is potentially replacing stack overflow with out of memory.

Some algorithms (like our own async - look for TryEnsureSufficientExecutionStack) try to use the stack as much as possible and gracefully cut over when the current thread stack runs out. These algorithms will make sure that the whole available stack is committed.

Do we have these sorts of problems on Linux today, which uses 8MB for every thread? It's also worth mentioning, that we are pushing people to use the stack more and more every .NET version (which I think is a good thing), e.g., with stackalloc first, recommendations to use structs instead of classes, inline arrays & collection expressions, upcoming params ReadOnlySpan<...> - all these features are slowly consuming more and more stack space for the same code.

I propose that we change Windows back to 4MB (which it used to be), or we could do 2MB for secondary threads if we think 4 is too high, and change macOS to match it:

OS	Main thread	Secondary threads
Windows	1.5MB -> 2 or 4MB	1.5MB -> 2 or 4MB
Linux	8MB	8MB
macOS	8MB	512kB -> 2 or 4MB

[jaredpar]

Some algorithms (like our own async - look for TryEnsureSufficientExecutionStack) try to use the stack as much as possible and gracefully cut over when the current thread stack runs out. These algorithms will make sure that the whole available stack is committed.

Do we have these sorts of problems on Linux today, which uses 8MB for every thread

FWIW The C# compiler uses TryEnsureSufficientExecutionStack pretty liberally across all threads. It hasn't caused any issues that I'm aware of. At the same time the compiler is a known memory hog.