dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.07k stars 4.69k forks source link

Runtime crashes on OSX running compiler unit tests #97186

Open jaredpar opened 8 months ago

jaredpar commented 8 months ago

Description

The runtime is crashing on OSX when running the compiler unit tests. There is no specific test that causes the crash but the crash is very consistent (approaching 100%).

Unfortunately, due to this being a OSX dump, the dump files are too large for us to upload a full dump. The best we can get is a mini dump and json report:

This is blocking our ability to test on OSX because there is no specific test to disable to work around the crash that we can see. The same tests pass just fine on Windows and Linux.

Reproduction Steps

Run the C# compiler tests in CI. Any build in this pipeline with the main branch filter will demonstrate the crash.

https://dnceng-public.visualstudio.com/public/_build?definitionId=95&_a=summary&branchFilter=319%2C319

Expected behavior

Unit tests pass

Actual behavior

Crashes with the following stack trace:

__wait4
PROCCreateCrashDump(std::__1::vector<char const*, std::__1::allocator<char const*>>&, char*, int, bool)
PROCCreateCrashDumpIfEnabled
invoke_previous_action(sigaction*, int, __siginfo*, void*, bool)
_sigtramp
WKS::gc_heap::mark_ro_segments()
WKS::gc_heap::mark_phase(int, int)
WKS::gc_heap::gc1()
WKS::gc_heap::garbage_collect(int)
WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason)
WKS::gc_heap::try_allocate_more_space(alloc_context*, unsigned long, unsigned int, int)
WKS::GCHeap::Alloc(gc_alloc_context*, unsigned long, unsigned int)
Alloc(unsigned long, GC_ALLOC_FLAGS)
AllocateObject(MethodTable*)
MethodTable::FastBox(void**)
JIT_Box(CORINFO_CLASS_STRUCT_*, void*)
Microsoft.CodeAnalysis.UnitTests.Collections.WrapStructural_SimpleInt.Equals(Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt)
System.Linq.Enumerable.Contains[[Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests]](System.Collections.Generic.IEnumerable`1<Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt>, Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, System.Collections.Generic.IEqualityComparer`1<Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt>)
Microsoft.CodeAnalysis.UnitTests.Collections.TestBase`1[[Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests]].CreateSortedSet(System.Collections.Generic.IEnumerable`1<Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt>, Int32, Int32)
Microsoft.CodeAnalysis.UnitTests.Collections.TestBase`1[[Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests]].CreateEnumerable(EnumerableType, System.Collections.Generic.IEnumerable`1<Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt>, Int32, Int32, Int32)
Microsoft.CodeAnalysis.UnitTests.Collections.ISet_Generic_Tests`1[[Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests]].ISet_Generic_Overlaps(EnumerableType, Int32, Int32, Int32, Int32)
DynamicClass.InvokeStub_ISet_Generic_Tests`1.ISet_Generic_Overlaps(System.Object, System.Span`1<System.Object>)
System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].CallTestMethod(System.Object)
Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef)
Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib]].<InvokeTestMethodAsync>b__1()
Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet]](<AggregateAsync>d__4 ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet]](<AggregateAsync>d__4 ByRef)
Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1<System.Threading.Tasks.Task>)
Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib]].<InvokeTestMethodAsync>b__0()
Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core]](<RunAsync>d__9 ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core]](<RunAsync>d__9 ByRef)
Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1<System.Threading.Tasks.Task>)
Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef)
Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].InvokeTestMethodAsync(System.Object)
Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(System.Object)
Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<RunAsync>b__47_0>d<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<RunAsync>b__47_0>d<System.__Canon> ByRef)
Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].<RunAsync>b__47_0()
Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.Decimal> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.Decimal> ByRef)
Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.Decimal>>)
Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator)
Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]](<InvokeTestAsync>d__4 ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]](<InvokeTestAsync>d__4 ByRef)
Xunit.Sdk.XunitTestRunner.InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)
Xunit.Sdk.TestRunner`1+<>c__DisplayClass43_0[[System.__Canon, System.Private.CoreLib]].<RunAsync>b__0()
Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.__Canon> ByRef)
Xunit.Sdk.ExceptionAggregator.RunAsync[[System.__Canon, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>)
Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__43<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__43<System.__Canon> ByRef)
Xunit.Sdk.TestRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
Xunit.Sdk.XunitTestCaseRunner.RunTestAsync()
Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef)
Xunit.Sdk.TestCaseRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
Xunit.Sdk.XunitTestCase.RunAsync(Xunit.Abstractions.IMessageSink, Xunit.Sdk.IMessageBus, System.Object[], Xunit.Sdk.ExceptionAggregator, System.Threading.CancellationTokenSource)
Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(Xunit.Sdk.IXunitTestCase)
Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef)
Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCasesAsync()
Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef)
Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(Xunit.Abstractions.ITestMethod, Xunit.Abstractions.IReflectionMethodInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Object[])
Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef)
Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestMethodsAsync()
Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__37<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__37<System.__Canon> ByRef)
Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>)
Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestClassesAsync>d__28<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestClassesAsync>d__28<System.__Canon> ByRef)
Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestClassesAsync()
Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__27<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__27<System.__Canon> ByRef)
Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Threading.CancellationTokenSource)
Xunit.Sdk.TestAssemblyRunner`1+<RunTestCollectionsAsync>d__42[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunTestCollectionsAsync>d__42[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCollectionsAsync>d__42<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunTestCollectionsAsync>d__42[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCollectionsAsync>d__42<System.__Canon> ByRef)
Xunit.Sdk.TestAssemblyRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCollectionsAsync(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource)
Xunit.Sdk.XunitTestAssemblyRunner.<>n__0(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource)
Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14, xunit.execution.dotnet]](<RunTestCollectionsAsync>d__14 ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14, xunit.execution.dotnet]](<RunTestCollectionsAsync>d__14 ByRef)
Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionsAsync(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource)
Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib]].MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__41<System.__Canon> ByRef)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__41<System.__Canon> ByRef)
Xunit.Sdk.TestAssemblyRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8.MoveNext()
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8, xunit.execution.dotnet]](<RunTestCases>d__8 ByRef)
System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start[[Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8, xunit.execution.dotnet]](<RunTestCases>d__8 ByRef)
Xunit.Sdk.XunitTestFrameworkExecutor.RunTestCases(System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions)
Xunit.Sdk.TestFrameworkExecutor`1[[System.__Canon, System.Private.CoreLib]].RunTests(System.Collections.Generic.IEnumerable`1<Xunit.Abstractions.ITestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions)
Xunit.Xunit2.RunTests(System.Collections.Generic.IEnumerable`1<Xunit.Abstractions.ITestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions)
Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, Xunit.Runner.VisualStudio.TestPlatformContext, Xunit.Runner.VisualStudio.RunSettings, Xunit.IMessageSinkWithTypes, Xunit.Runner.VisualStudio.AssemblyRunInfo)
Xunit.Runner.VisualStudio.VsTestRunner+<>c__DisplayClass20_0.<RunTests>b__4(Xunit.Runner.VisualStudio.AssemblyRunInfo)
System.Collections.Generic.List`1[[System.__Canon, System.Private.CoreLib]].ForEach(System.Action`1<System.__Canon>)
Xunit.Runner.VisualStudio.VsTestRunner.RunTests(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, Xunit.Runner.VisualStudio.TestPlatformContext, Xunit.Runner.VisualStudio.RunSettings, System.Func`1<System.Collections.Generic.List`1<Xunit.Runner.VisualStudio.AssemblyRunInfo>>)
Xunit.Runner.VisualStudio.VsTestRunner.Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor.RunTests(System.Collections.Generic.IEnumerable`1<System.String>, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
Microsoft.VisualStudio.TestPlatform.Common.ExtensionDecorators.SerialTestRunDecorator.RunTests(System.Collections.Generic.IEnumerable`1<System.String>, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.Utilities.LazyExtension`2<Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor,Microsoft.VisualStudio.TestPlatform.Common.Interfaces.ITestExecutorCapabilities>, System.Tuple`2<System.Uri,System.String>, Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Adapter.RunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(System.Collections.Generic.IEnumerable`1<System.Tuple`2<System.Uri,System.String>>, Int64)
Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestsInternal()
Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTests()
Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.ExecutionManager.StartTestRun(System.Collections.Generic.Dictionary`2<System.String,System.Collections.Generic.IEnumerable`1<System.String>>, System.String, System.String, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ClientProtocol.TestExecutionContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ITestCaseEventsHandler, Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IInternalTestRunEventsHandler)
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c__DisplayClass45_4.<OnMessageReceived>b__3()
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c.<.ctor>b__31_2(System.Action)
Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib]].SafeProcessJob(System.__Canon)
Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib]].BackgroundJobProcessor(System.String)
Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib]].<.ctor>b__16_0()
System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
System.Threading.Thread.StartCallback()
DispatchCallSimple(unsigned long*, unsigned int, unsigned long long, unsigned int)
ThreadNative::KickOffThread_Worker(void*)
ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)
ManagedThreadBase::KickOff(void (*)(void*), void*)
ThreadNative::KickOffThread(void*)
CorUnix::CPalThread::ThreadEntry(void*)
_pthread_start

Regression?

Yes this is a recent regression that came around the time of adopting the .NET 8 SDK GA

Known Workarounds

None

Configuration

Complicated

Other information

At the moment we are collecting OSX dumps with the following settings:

DOTNET_DbgEnableMiniDump=1
DOTNET_DbgMiniDumpName=/cores/crash.%d.%e.dmp
DOTNET_DbgMiniDumpType=1
DOTNET_EnableCrashReport=1

Happy to adjust this as need to help with the investigation.

ghost commented 8 months ago

Tagging subscribers to this area: @dotnet/gc See info in area-owners.md if you want to be subscribed.

Issue Details
### Description The runtime is crashing on OSX when running the compiler unit tests. There is no specific test that causes the crash but the crash is very consistent (approaching 100%). Unfortunately, due to this being a OSX dump, the dump files are too large for us to upload a full dump. The best we can get is a mini dump and json report: - [crash.dmp][1] - [crash.dmp.json][2] This is blocking our ability to test on OSX because there is no specific test to disable to work around the crash that we can see. The same tests pass just fine on Windows and Linux. [1]: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-roslyn-refs-pull-71706-merge-f38fe728c96a430d94/Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests_Microsoft.CodeAnalysis.Rebuild.UnitTests_Microsoft.CodeAnalysis.Scripting.UnitTests_Microsoft.CodeAnaly..._3/1/crash.34178.dotnet.dmp?helixlogtype=result [2]: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-roslyn-refs-pull-71706-merge-f38fe728c96a430d94/Microsoft.CodeAnalysis.CSharp.Syntax.UnitTests_Microsoft.CodeAnalysis.Rebuild.UnitTests_Microsoft.CodeAnalysis.Scripting.UnitTests_Microsoft.CodeAnaly..._3/1/crash.34178.dotnet.dmp.crashreport.json?helixlogtype=result ### Reproduction Steps Run the C# compiler tests in CI. Any build in this pipeline with the `main` branch filter will demonstrate the crash. https://dnceng-public.visualstudio.com/public/_build?definitionId=95&_a=summary&branchFilter=319%2C319 ### Expected behavior Unit tests pass ### Actual behavior Crashes with the following stack trace: ``` __wait4 PROCCreateCrashDump(std::__1::vector>&, char*, int, bool) PROCCreateCrashDumpIfEnabled invoke_previous_action(sigaction*, int, __siginfo*, void*, bool) _sigtramp WKS::gc_heap::mark_ro_segments() WKS::gc_heap::mark_phase(int, int) WKS::gc_heap::gc1() WKS::gc_heap::garbage_collect(int) WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) WKS::gc_heap::try_allocate_more_space(alloc_context*, unsigned long, unsigned int, int) WKS::GCHeap::Alloc(gc_alloc_context*, unsigned long, unsigned int) Alloc(unsigned long, GC_ALLOC_FLAGS) AllocateObject(MethodTable*) MethodTable::FastBox(void**) JIT_Box(CORINFO_CLASS_STRUCT_*, void*) Microsoft.CodeAnalysis.UnitTests.Collections.WrapStructural_SimpleInt.Equals(Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt) System.Linq.Enumerable.Contains[[Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests]](System.Collections.Generic.IEnumerable`1, Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, System.Collections.Generic.IEqualityComparer`1) Microsoft.CodeAnalysis.UnitTests.Collections.TestBase`1[[Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests]].CreateSortedSet(System.Collections.Generic.IEnumerable`1, Int32, Int32) Microsoft.CodeAnalysis.UnitTests.Collections.TestBase`1[[Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests]].CreateEnumerable(EnumerableType, System.Collections.Generic.IEnumerable`1, Int32, Int32, Int32) Microsoft.CodeAnalysis.UnitTests.Collections.ISet_Generic_Tests`1[[Microsoft.CodeAnalysis.UnitTests.Collections.SimpleInt, Microsoft.CodeAnalysis.UnitTests]].ISet_Generic_Overlaps(EnumerableType, Int32, Int32, Int32, Int32) DynamicClass.InvokeStub_ISet_Generic_Tests`1.ISet_Generic_Overlaps(System.Object, System.Span`1) System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].CallTestMethod(System.Object) Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<b__1>d[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<b__1>d ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1+<b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<b__1>d ByRef) Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib]].b__1() Xunit.Sdk.ExecutionTimer+d__4.MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+d__4, xunit.execution.dotnet]](d__4 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExecutionTimer+d__4, xunit.execution.dotnet]](d__4 ByRef) Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1) Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_1[[System.__Canon, System.Private.CoreLib]].b__0() Xunit.Sdk.ExceptionAggregator+d__9.MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+d__9, xunit.core]](d__9 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExceptionAggregator+d__9, xunit.core]](d__9 ByRef) Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1) Xunit.Sdk.TestInvoker`1+d__48[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__48 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.TestInvoker`1+d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__48 ByRef) Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].InvokeTestMethodAsync(System.Object) Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(System.Object) Xunit.Sdk.TestInvoker`1+<b__47_0>d[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<b__47_0>d ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.TestInvoker`1+<b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<b__47_0>d ByRef) Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].b__47_0() Xunit.Sdk.ExceptionAggregator+d__10`1[[System.Decimal, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](d__10`1 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.ExceptionAggregator+d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](d__10`1 ByRef) Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib]](System.Func`1>) Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].RunAsync() Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator) Xunit.Sdk.XunitTestRunner+d__4.MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+d__4, xunit.execution.dotnet]](d__4 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTestRunner+d__4, xunit.execution.dotnet]](d__4 ByRef) Xunit.Sdk.XunitTestRunner.InvokeTestAsync(Xunit.Sdk.ExceptionAggregator) Xunit.Sdk.TestRunner`1+<>c__DisplayClass43_0[[System.__Canon, System.Private.CoreLib]].b__0() Xunit.Sdk.ExceptionAggregator+d__10`1[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](d__10`1 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.ExceptionAggregator+d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](d__10`1 ByRef) Xunit.Sdk.ExceptionAggregator.RunAsync[[System.__Canon, System.Private.CoreLib]](System.Func`1>) Xunit.Sdk.TestRunner`1+d__43[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestRunner`1+d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__43 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestRunner`1+d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__43 ByRef) Xunit.Sdk.TestRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() Xunit.Sdk.XunitTestCaseRunner.RunTestAsync() Xunit.Sdk.TestCaseRunner`1+d__19[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCaseRunner`1+d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__19 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCaseRunner`1+d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__19 ByRef) Xunit.Sdk.TestCaseRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() Xunit.Sdk.XunitTestCase.RunAsync(Xunit.Abstractions.IMessageSink, Xunit.Sdk.IMessageBus, System.Object[], Xunit.Sdk.ExceptionAggregator, System.Threading.CancellationTokenSource) Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(Xunit.Sdk.IXunitTestCase) Xunit.Sdk.TestMethodRunner`1+d__32[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__32 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__32 ByRef) Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCasesAsync() Xunit.Sdk.TestMethodRunner`1+d__31[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__31 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__31 ByRef) Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(Xunit.Abstractions.ITestMethod, Xunit.Abstractions.IReflectionMethodInfo, System.Collections.Generic.IEnumerable`1, System.Object[]) Xunit.Sdk.TestClassRunner`1+d__38[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__38 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__38 ByRef) Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestMethodsAsync() Xunit.Sdk.TestClassRunner`1+d__37[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__37 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__37 ByRef) Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1) Xunit.Sdk.TestCollectionRunner`1+d__28[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__28 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__28 ByRef) Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestClassesAsync() Xunit.Sdk.TestCollectionRunner`1+d__27[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__27 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__27 ByRef) Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1, System.Threading.CancellationTokenSource) Xunit.Sdk.TestAssemblyRunner`1+d__42[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestAssemblyRunner`1+d__42[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__42 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestAssemblyRunner`1+d__42[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__42 ByRef) Xunit.Sdk.TestAssemblyRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCollectionsAsync(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource) Xunit.Sdk.XunitTestAssemblyRunner.<>n__0(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource) Xunit.Sdk.XunitTestAssemblyRunner+d__14.MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestAssemblyRunner+d__14, xunit.execution.dotnet]](d__14 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTestAssemblyRunner+d__14, xunit.execution.dotnet]](d__14 ByRef) Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionsAsync(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource) Xunit.Sdk.TestAssemblyRunner`1+d__41[[System.__Canon, System.Private.CoreLib]].MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestAssemblyRunner`1+d__41[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__41 ByRef) System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestAssemblyRunner`1+d__41[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](d__41 ByRef) Xunit.Sdk.TestAssemblyRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync() Xunit.Sdk.XunitTestFrameworkExecutor+d__8.MoveNext() System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestFrameworkExecutor+d__8, xunit.execution.dotnet]](d__8 ByRef) System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start[[Xunit.Sdk.XunitTestFrameworkExecutor+d__8, xunit.execution.dotnet]](d__8 ByRef) Xunit.Sdk.XunitTestFrameworkExecutor.RunTestCases(System.Collections.Generic.IEnumerable`1, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions) Xunit.Sdk.TestFrameworkExecutor`1[[System.__Canon, System.Private.CoreLib]].RunTests(System.Collections.Generic.IEnumerable`1, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions) Xunit.Xunit2.RunTests(System.Collections.Generic.IEnumerable`1, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions) Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, Xunit.Runner.VisualStudio.TestPlatformContext, Xunit.Runner.VisualStudio.RunSettings, Xunit.IMessageSinkWithTypes, Xunit.Runner.VisualStudio.AssemblyRunInfo) Xunit.Runner.VisualStudio.VsTestRunner+<>c__DisplayClass20_0.b__4(Xunit.Runner.VisualStudio.AssemblyRunInfo) System.Collections.Generic.List`1[[System.__Canon, System.Private.CoreLib]].ForEach(System.Action`1) Xunit.Runner.VisualStudio.VsTestRunner.RunTests(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, Xunit.Runner.VisualStudio.TestPlatformContext, Xunit.Runner.VisualStudio.RunSettings, System.Func`1>) Xunit.Runner.VisualStudio.VsTestRunner.Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor.RunTests(System.Collections.Generic.IEnumerable`1, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle) Microsoft.VisualStudio.TestPlatform.Common.ExtensionDecorators.SerialTestRunDecorator.RunTests(System.Collections.Generic.IEnumerable`1, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle) Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.Utilities.LazyExtension`2, System.Tuple`2, Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Adapter.RunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle) Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(System.Collections.Generic.IEnumerable`1>, Int64) Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestsInternal() Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTests() Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.ExecutionManager.StartTestRun(System.Collections.Generic.Dictionary`2>, System.String, System.String, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ClientProtocol.TestExecutionContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ITestCaseEventsHandler, Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IInternalTestRunEventsHandler) Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c__DisplayClass45_4.b__3() Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c.<.ctor>b__31_2(System.Action) Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib]].SafeProcessJob(System.__Canon) Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib]].BackgroundJobProcessor(System.String) Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib]].<.ctor>b__16_0() System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) System.Threading.Thread.StartCallback() DispatchCallSimple(unsigned long*, unsigned int, unsigned long long, unsigned int) ThreadNative::KickOffThread_Worker(void*) ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) ManagedThreadBase::KickOff(void (*)(void*), void*) ThreadNative::KickOffThread(void*) CorUnix::CPalThread::ThreadEntry(void*) _pthread_start ``` ### Regression? Yes this is a recent regression that came around the time of adopting the .NET 8 SDK GA ### Known Workarounds None ### Configuration Complicated ### Other information At the moment we are collecting OSX dumps with the following settings: ```sh DOTNET_DbgEnableMiniDump=1 DOTNET_DbgMiniDumpName=/cores/crash.%d.%e.dmp DOTNET_DbgMiniDumpType=1 DOTNET_EnableCrashReport=1 ``` Happy to adjust this as need to help with the investigation.
Author: jaredpar
Assignees: -
Labels: `area-GC-coreclr`, `untriaged`, `needs-area-label`
Milestone: -
cshung commented 8 months ago

I have got mac devices, is it possible to reproduce these locally?

EgorBo commented 8 months ago

GC Regions aren't enabled for OSX?

jaredpar commented 8 months ago

I have got mac devices, is it possible to reproduce these locally?

We tried running the tests locally on Macs and the issue did not reproduce. Unsure if it's specific to this hardware or some other factor. It reproduces very reliably in CI.

cshung commented 8 months ago

GC Regions aren't enabled for OSX?

Yes, it is not enabled for OSX, because of the large dump issue.

EgorBo commented 8 months ago

Then maybe the issue can be reproduced on other platforms by disabling regions?

cshung commented 8 months ago

We certainly can disable region using DOTNET_GCName=clrgc, this should work on any platforms.

cshung commented 8 months ago

Do we know how to read the dump? When I tried to open the dump in Visual Studio, it complained wrong file format. When I tried to open it in WinDBG, it reported 25 threads, lm showed no modules. I tried to open it with lldb on osx, this time it shows 21 threads, but all thread back stack traces are just numbers.

The crash report is more interesting, it does indicate mark_ro_segment is causing a trap

     {
      "is_managed" : "false",
      "module_address" : "0x7ff818c5a000",
      "stack_pointer" : "0x70000a07df40",
      "native_address" : "0x7ff818c5d5ed",
      "native_offset" : "0x1d",
      "native_image_offset" : "0x35ed",
      "unmanaged_name" : "_sigtramp",
      "native_module" : "libsystem_platform.dylib"
     },
     {
      "is_managed" : "false",
      "module_address" : "0x101394000",
      "stack_pointer" : "0x70000a07e490",
      "native_address" : "0x101676e05",
      "native_offset" : "0x2b5",
      "native_image_offset" : "0x2e2e05",
      "unmanaged_name" : "WKS::gc_heap::mark_ro_segments()",
      "native_module" : "libcoreclr.dylib"
     },

The native_image_offset is potentially useful, it might led us to the instruction where it failed, but I am not sure how to proceed from there.

jaredpar commented 8 months ago

Do we know how to read the dump? When I tried to open the dump in Visual Studio, it complained wrong file format.

My understanding from chatting with @hoyosjs is that Mac dumps are only openable on Mac.

We certainly can disable region using DOTNET_GCName=clrgc, this should work on any platforms.

I queued up a PR that disables this for Linux. That should be fairly apples to apples with our Mac runs. If it does crash the dump should upload cause it doesn't have the same size issues that Mac's do.

jaredpar commented 8 months ago

Got a complete run on Linux with DOTNET_GCName=libclrgc.so and everything passed.

cshung commented 8 months ago

Got a complete run on Linux with DOTNET_GCName=libclrgc.so and everything passed.

That's sad, which means we have to work with OSX then. Is it possible to capture stress log and/or run with customized clrgc with instrumentation?

wfurt commented 8 months ago

I see local test crashes as well after I sync & rebuild runtime (after git clean -dfx)

(lldb) clrstack -a
OS Thread Id: 0x10aba58 (1)
        Child SP               IP Call Site
00007FF7BFEF9B38 0000000104664b33 [HelperMethodFrame_PROTECTOBJ: 00007ff7bfef9b38] System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
00007FF7BFEF9DE0 000000012E28696D System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(System.Object, IntPtr*) [/Users/furt/github/wfurt-runtime2/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs @ 36]
    PARAMETERS:
        this (0x00007FF7BFEF9DF8) = 0x0000000190476978
        obj (0x00007FF7BFEF9DF0) = 0x0000000190476730
        args (0x00007FF7BFEF9DE8) = 0x0000000000000000

00007FF7BFEF9E10 000000012E26E18F System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs @ 57]
    PARAMETERS:
        this (0x00007FF7BFEF9E78) = 0x0000000190476978
        obj (0x00007FF7BFEF9E70) = 0x0000000190476730
        invokeAttr (0x00007FF7BFEF9E6C) = 0x9047673000000000
    LOCALS:
        0x00007FF7BFEF9E60 = 0x0000000000000000
        0x00007FF7BFEF9E58 = 0x000000019048fb68

00007FF7BFEF9E90 000000012E28537B System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs @ 127]
    PARAMETERS:
        this (0x00007FF7BFEF9EE8) = 0x00000001902b6bf8
        obj (0x00007FF7BFEF9EE0) = 0x0000000190476730
        invokeAttr (0x00007FF7BFEF9EDC) = 0x9047673000000000
        binder (0x00007FF7BFEF9ED0) = 0x0000000000000000
        parameters (0x00007FF7BFEF9EC8) = 0x0000000190052128
        culture (0x00007FF7BFEF9EC0) = 0x0000000000000000
    LOCALS:
        0x00007FF7BFEF9EBC = 0x0000000000000000

00007FF7BFEF9F00 000000012E285267 System.Reflection.MethodBase.Invoke(System.Object, System.Object[]) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBase.cs @ 56]
    PARAMETERS:
        this (0x00007FF7BFEF9F18) = 0x00000001902b6bf8
        obj (0x00007FF7BFEF9F10) = 0x0000000190476730
        parameters (0x00007FF7BFEF9F08) = 0x0000000190052128

00007FF7BFEF9F30 000000012EFB056C Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].CallTestMethod(System.Object) [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 150]
    PARAMETERS:
        this (0x00007FF7BFEF9F48) = 0x00000001904763e0
        testClassInstance (0x00007FF7BFEF9F40) = 0x0000000190476730

00007FF7BFEF9F60 000000012EFB018F Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 269]
    PARAMETERS:
        this (0x00007FF7BFEFA190) = 0x00007ff7bfefa298
    LOCALS:
        0x00007FF7BFEFA184 = 0x00000000ffffffff
        0x00007FF7BFEFA178 = 0x00000001904767f0
        0x00007FF7BFEFA174 = 0x0000000000000000
        0x00007FF7BFEFA170 = 0x0000000000000000
        0x00007FF7BFEFA168 = 0x0000000000000000
        0x00007FF7BFEFA160 = 0x0000000000000000
        0x00007FF7BFEFA158 = 0x0000000000000000
        0x00007FF7BFEFA150 = 0x0000000000000000
        0x00007FF7BFEFA148 = 0x0000000000000000
        0x00007FF7BFEFA140 = 0x0000000000000000

00007FF7BFEFA1B0 000000012EFAFA09 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFA218) = 0x00007ff7bfefa298
    LOCALS:
        0x00007FF7BFEFA210 = 0x0000000190064118
        0x00007FF7BFEFA208 = 0x0000000000000000
        0x00007FF7BFEFA200 = 0x0000000190476820
        0x00007FF7BFEFA1F8 = 0x0000000000000000

00007FF7BFEFA240 000000012EFAF930 System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0+<<InvokeTestMethodAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<InvokeTestMethodAsync>b__1>d<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilder.cs @ 33]
    PARAMETERS:
        this (0x00007FF7BFEFA260) = 0x00007ff7bfefa2a8
        stateMachine (0x00007FF7BFEFA250) = 0x00007ff7bfefa298

00007FF7BFEFA280 000000012EFAF8A6 Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0[[System.__Canon, System.Private.CoreLib]].<InvokeTestMethodAsync>b__1()
    PARAMETERS:
        this (0x00007FF7BFEFA2C0) = 0x00000001904767f0
    LOCALS:
        0x00007FF7BFEFA298 = 0x00000001904767f0

00007FF7BFEFA2E0 000000012EFAF65E Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4.MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs @ 48]
    PARAMETERS:
        this (0x00007FF7BFEFA328) = 0x00007ff7bfefa3d0
    LOCALS:
        0x00007FF7BFEFA324 = 0x00000000ffffffff
        0x00007FF7BFEFA318 = 0x0000000190476468
        0x00007FF7BFEFA310 = 0x0000000000000000
        0x00007FF7BFEFA308 = 0x0000000000000000

00007FF7BFEFA340 000000012EFAF541 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet]](<AggregateAsync>d__4 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFA398) = 0x00007ff7bfefa3d0
    LOCALS:
        0x00007FF7BFEFA390 = 0x0000000190064118
        0x00007FF7BFEFA388 = 0x0000000000000000
        0x00007FF7BFEFA380 = 0x0000000190476820
        0x00007FF7BFEFA378 = 0x0000000000000000

00007FF7BFEFA3B0 000000012EFAF4AC System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExecutionTimer+<AggregateAsync>d__4, xunit.execution.dotnet]](<AggregateAsync>d__4 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilder.cs @ 33]
    PARAMETERS:
        this (0x00007FF7BFEFA3B8) = 0x00007ff7bfefa3f0
        stateMachine (0x00007FF7BFEFA3B0) = 0x00007ff7bfefa3d0

00007FF7BFEFA3D0 000000012EFAF460 Xunit.Sdk.ExecutionTimer.AggregateAsync(System.Func`1<System.Threading.Tasks.Task>)
    PARAMETERS:
        this (0x00007FF7BFEFA408) = 0x0000000190476468
        asyncAction (0x00007FF7BFEFA400) = 0x0000000190476910
    LOCALS:
        0x00007FF7BFEFA3D0 = 0x0000000190476910

00007FF7BFEFA420 000000012EFAF3F3 Xunit.Sdk.TestInvoker`1+<>c__DisplayClass48_0[[System.__Canon, System.Private.CoreLib]].<InvokeTestMethodAsync>b__0() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 242]
    PARAMETERS:
        this (0x00007FF7BFEFA458) = 0x00000001904767f0
    LOCALS:
        0x00007FF7BFEFA450 = 0x0000000190476910

00007FF7BFEFA470 000000012EFAF1B1 Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9.MoveNext() [/_/src/xunit.core/Sdk/ExceptionAggregator.cs @ 90]
    PARAMETERS:
        this (0x00007FF7BFEFA4C8) = 0x00007ff7bfefa578
    LOCALS:
        0x00007FF7BFEFA4C4 = 0x00000000ffffffff
        0x00007FF7BFEFA4B8 = 0x0000000190476368
        0x00007FF7BFEFA4B0 = 0x0000000000000000
        0x00007FF7BFEFA4A8 = 0x0000000000000000
        0x00007FF7BFEFA4A0 = 0x0000000000000000

00007FF7BFEFA4E0 000000012EFAF0BB System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core]](<RunAsync>d__9 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFA538) = 0x00007ff7bfefa578
    LOCALS:
        0x00007FF7BFEFA530 = 0x0000000190064118
        0x00007FF7BFEFA528 = 0x0000000000000000
        0x00007FF7BFEFA520 = 0x0000000190476820
        0x00007FF7BFEFA518 = 0x0000000000000000

00007FF7BFEFA550 000000012EFAF02C System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__9, xunit.core]](<RunAsync>d__9 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilder.cs @ 33]
    PARAMETERS:
        this (0x00007FF7BFEFA558) = 0x00007ff7bfefa590
        stateMachine (0x00007FF7BFEFA550) = 0x00007ff7bfefa578

00007FF7BFEFA570 000000012EFAEFE1 Xunit.Sdk.ExceptionAggregator.RunAsync(System.Func`1<System.Threading.Tasks.Task>)
    PARAMETERS:
        this (0x00007FF7BFEFA5A8) = 0x0000000190476368
        code (0x00007FF7BFEFA5A0) = 0x00000001904768d0
    LOCALS:
        0x00007FF7BFEFA578 = 0x00000001904768d0

00007FF7BFEFA5C0 000000012EFAE86B Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 241]
    PARAMETERS:
        this (0x00007FF7BFEFA670) = 0x00007ff7bfefa778
    LOCALS:
        0x00007FF7BFEFA664 = 0x00000000ffffffff
        0x00007FF7BFEFA658 = 0x00000001904763e0
        0x00007FF7BFEFA648 = 0x0000000000000000
        0x00007FF7BFEFA640 = 0x00000001904767f0
        0x00007FF7BFEFA638 = 0x0000000000000000
        0x00007FF7BFEFA630 = 0x0000000000000000

00007FF7BFEFA690 000000012EFAE5BF System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFA6F8) = 0x00007ff7bfefa778
    LOCALS:
        0x00007FF7BFEFA6F0 = 0x0000000190064118
        0x00007FF7BFEFA6E8 = 0x0000000000000000
        0x00007FF7BFEFA6E0 = 0x0000000000000000
        0x00007FF7BFEFA6D8 = 0x0000000000000000

00007FF7BFEFA720 000000012EFAE4E0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.TestInvoker`1+<InvokeTestMethodAsync>d__48[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<InvokeTestMethodAsync>d__48<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFA740) = 0x00007ff7bfefa798
        stateMachine (0x00007FF7BFEFA730) = 0x00007ff7bfefa778

00007FF7BFEFA760 000000012EFAE456 Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].InvokeTestMethodAsync(System.Object)
    PARAMETERS:
        this (0x00007FF7BFEFA7B0) = 0x00000001904763e0
        testClassInstance (0x00007FF7BFEFA7A8) = 0x0000000190476730
    LOCALS:
        0x00007FF7BFEFA778 = 0x00000001904763e0

00007FF7BFEFA7D0 000000012EFAE28A Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(System.Object) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestInvoker.cs @ 112]
    PARAMETERS:
        this (0x00007FF7BFEFA808) = 0x00000001904763e0
        testClassInstance (0x00007FF7BFEFA800) = 0x0000000190476730

00007FF7BFEFA820 000000012EFACCAC Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 206]
    PARAMETERS:
        this (0x00007FF7BFEFA910) = 0x00007ff7bfefaa28
    LOCALS:
        0x00007FF7BFEFA904 = 0x00000000ffffffff
        0x00007FF7BFEFA8F8 = 0x00000001904763e0
        0x00007FF7BFEFA8E8 = 0x0000000000000000
        0x00007FF7BFEFA8E0 = 0x0000000190469258
        0x00007FF7BFEFA8D8 = 0x0000000000000000
        0x00007FF7BFEFA8D0 = 0x0000000000000000

00007FF7BFEFA930 000000012EFAC7F3 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<RunAsync>b__47_0>d<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFA9A8) = 0x00007ff7bfefaa28
    LOCALS:
        0x00007FF7BFEFA9A0 = 0x0000000190064118
        0x00007FF7BFEFA998 = 0x0000000000000000
        0x00007FF7BFEFA990 = 0x0000000000000000
        0x00007FF7BFEFA988 = 0x0000000000000000

00007FF7BFEFA9D0 000000012EFAC710 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.TestInvoker`1+<<RunAsync>b__47_0>d[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<<RunAsync>b__47_0>d<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFA9F0) = 0x00007ff7bfefaa48
        stateMachine (0x00007FF7BFEFA9E0) = 0x00007ff7bfefaa28

00007FF7BFEFAA10 000000012EFAC68B Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].<RunAsync>b__47_0()
    PARAMETERS:
        this (0x00007FF7BFEFAA60) = 0x00000001904763e0
    LOCALS:
        0x00007FF7BFEFAA28 = 0x00000001904763e0

00007FF7BFEFAA80 000000012EFAC42F Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]].MoveNext() [/_/src/xunit.core/Sdk/ExceptionAggregator.cs @ 107]
    PARAMETERS:
        this (0x00007FF7BFEFAAF8) = 0x00007ff7bfefaba8
    LOCALS:
        0x00007FF7BFEFAAF4 = 0x00000000ffffffff
        0x00007FF7BFEFAAE8 = 0x0000000190476368
        0x00007FF7BFEFAAD8 = 0x0000000000000000
        0x00007FF7BFEFAAD0 = 0x0000000000000000
        0x00007FF7BFEFAAC8 = 0x0000000000000000
        0x00007FF7BFEFAAC0 = 0x0000000000000000

00007FF7BFEFAB10 000000012EFAC32B System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.Decimal> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFAB68) = 0x00007ff7bfefaba8
    LOCALS:
        0x00007FF7BFEFAB60 = 0x0000000190064118
        0x00007FF7BFEFAB58 = 0x0000000000000000
        0x00007FF7BFEFAB50 = 0x0000000000000000
        0x00007FF7BFEFAB48 = 0x0000000000000000

00007FF7BFEFAB80 000000012EFAC29C System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Decimal, System.Private.CoreLib]].Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.Decimal, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.Decimal> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFAB88) = 0x00007ff7bfefabc0
        stateMachine (0x00007FF7BFEFAB80) = 0x00007ff7bfefaba8

00007FF7BFEFABA0 000000012EFAC211 Xunit.Sdk.ExceptionAggregator.RunAsync[[System.Decimal, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.Decimal>>)
    PARAMETERS:
        this (0x00007FF7BFEFABD8) = 0x0000000190476368
        code (0x00007FF7BFEFABD0) = 0x0000000190476500
    LOCALS:
        0x00007FF7BFEFABA8 = 0x0000000190476500

00007FF7BFEFABF0 000000012EFAC171 Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib]].RunAsync() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs @ 189]
    PARAMETERS:
        this (0x00007FF7BFEFAC08) = 0x00000001904763e0

00007FF7BFEFAC20 000000012EFABB4B Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(Xunit.Sdk.ExceptionAggregator) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs @ 84]
    PARAMETERS:
        this (0x00007FF7BFEFAC98) = 0x00000001904760f0
        aggregator (0x00007FF7BFEFAC90) = 0x0000000190476368

00007FF7BFEFACB0 000000012EFAB7BB Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4.MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestRunner.cs @ 67]
    PARAMETERS:
        this (0x00007FF7BFEFAD70) = 0x00007ff7bfefae68
    LOCALS:
        0x00007FF7BFEFAD6C = 0x00000000ffffffff
        0x00007FF7BFEFAD60 = 0x00000001904760f0
        0x00007FF7BFEFAD58 = 0x0000000000000000
        0x00007FF7BFEFAD50 = 0x0000000190475a58
        0x00007FF7BFEFAD4C = 0x0000000000000000
        0x00007FF7BFEFAD40 = 0x0000000000000000
        0x00007FF7BFEFAD38 = 0x0000000000000000
        0x00007FF7BFEFAD30 = 0x0000000000000000

00007FF7BFEFADB0 000000012EFAB4E2 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]](<InvokeTestAsync>d__4 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFAE18) = 0x00007ff7bfefae68
    LOCALS:
        0x00007FF7BFEFAE10 = 0x0000000190064118
        0x00007FF7BFEFAE08 = 0x0000000000000000
        0x00007FF7BFEFAE00 = 0x0000000000000000
        0x00007FF7BFEFADF8 = 0x0000000000000000

00007FF7BFEFAE30 000000012EFAB450 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTestRunner+<InvokeTestAsync>d__4, xunit.execution.dotnet]](<InvokeTestAsync>d__4 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFAE48) = 0x00007ff7bfefae90
        stateMachine (0x00007FF7BFEFAE38) = 0x00007ff7bfefae68

00007FF7BFEFAE60 000000012EFAB3FA Xunit.Sdk.XunitTestRunner.InvokeTestAsync(Xunit.Sdk.ExceptionAggregator)
    PARAMETERS:
        this (0x00007FF7BFEFAEA8) = 0x00000001904760f0
        aggregator (0x00007FF7BFEFAEA0) = 0x0000000190476368
    LOCALS:
        0x00007FF7BFEFAE68 = 0x00000001904760f0

00007FF7BFEFAEC0 000000012EFAB36A Xunit.Sdk.TestRunner`1+<>c__DisplayClass43_0[[System.__Canon, System.Private.CoreLib]].<RunAsync>b__0() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestRunner.cs @ 149]
    PARAMETERS:
        this (0x00007FF7BFEFAEC8) = 0x00000001904762b8

00007FF7BFEFAEE0 000000012EFAB018 Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.core/Sdk/ExceptionAggregator.cs @ 107]
    PARAMETERS:
        this (0x00007FF7BFEFAF80) = 0x00007ff7bfefb088
    LOCALS:
        0x00007FF7BFEFAF74 = 0x00000000ffffffff
        0x00007FF7BFEFAF68 = 0x0000000190476368
        0x00007FF7BFEFAF60 = 0x0000000000000000
        0x00007FF7BFEFAF58 = 0x0000000000000000
        0x00007FF7BFEFAF50 = 0x0000000000000000
        0x00007FF7BFEFAF48 = 0x0000000000000000

00007FF7BFEFAFA0 000000012EFAAEF9 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFB008) = 0x00007ff7bfefb088
    LOCALS:
        0x00007FF7BFEFB000 = 0x0000000190064118
        0x00007FF7BFEFAFF8 = 0x0000000000000000
        0x00007FF7BFEFAFF0 = 0x0000000000000000
        0x00007FF7BFEFAFE8 = 0x0000000000000000

00007FF7BFEFB030 000000012EFAAE20 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.ExceptionAggregator+<RunAsync>d__10`1[[System.__Canon, System.Private.CoreLib]], xunit.core]](<RunAsync>d__10`1<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFB050) = 0x00007ff7bfefb0a0
        stateMachine (0x00007FF7BFEFB040) = 0x00007ff7bfefb088

00007FF7BFEFB070 000000012EFAAC41 Xunit.Sdk.ExceptionAggregator.RunAsync[[System.__Canon, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>)
    PARAMETERS:
        this (0x00007FF7BFEFB0C0) = 0x0000000190476368
        code (0x00007FF7BFEFB0B0) = 0x00000001904763a0
    LOCALS:
        0x00007FF7BFEFB088 = 0x00000001904763a0

00007FF7BFEFB0E0 000000012EFAA160 Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestRunner.cs @ 149]
    PARAMETERS:
        this (0x00007FF7BFEFB240) = 0x00007ff7bfefb358
    LOCALS:
        0x00007FF7BFEFB234 = 0x00000000ffffffff
        0x00007FF7BFEFB228 = 0x00000001904760f0
        0x00007FF7BFEFB220 = 0x0000000000000000
        0x00007FF7BFEFB218 = 0x0000000000000000
        0x00007FF7BFEFB210 = 0x0000000000000000
        0x00007FF7BFEFB208 = 0x0000000000000000
        0x00007FF7BFEFB200 = 0x0000000000000000
        0x00007FF7BFEFB1F8 = 0x0000000000000000

00007FF7BFEFB260 000000012EFA9D03 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__43<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFB2D8) = 0x00007ff7bfefb358
    LOCALS:
        0x00007FF7BFEFB2D0 = 0x0000000190064118
        0x00007FF7BFEFB2C8 = 0x0000000000000000
        0x00007FF7BFEFB2C0 = 0x0000000000000000
        0x00007FF7BFEFB2B8 = 0x0000000000000000

00007FF7BFEFB300 000000012EFA9C20 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestRunner`1+<RunAsync>d__43[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__43<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFB320) = 0x00007ff7bfefb380
        stateMachine (0x00007FF7BFEFB310) = 0x00007ff7bfefb358

00007FF7BFEFB340 000000012EFA9B85 Xunit.Sdk.TestRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
    PARAMETERS:
        this (0x00007FF7BFEFB390) = 0x00000001904760f0
    LOCALS:
        0x00007FF7BFEFB358 = 0x00000001904760f0

00007FF7BFEFB3B0 000000012EFA7ED6 Xunit.Sdk.XunitTestCaseRunner.RunTestAsync() [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestCaseRunner.cs @ 139]
    PARAMETERS:
        this (0x00007FF7BFEFB438) = 0x0000000190475e40

00007FF7BFEFB450 000000012EFA781A Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestCaseRunner.cs @ 82]
    PARAMETERS:
        this (0x00007FF7BFEFB520) = 0x00007ff7bfefb620
    LOCALS:
        0x00007FF7BFEFB514 = 0x00000000ffffffff
        0x00007FF7BFEFB508 = 0x0000000190475e40
        0x00007FF7BFEFB500 = 0x0000000000000000
        0x00007FF7BFEFB4F8 = 0x0000000190469258
        0x00007FF7BFEFB4F0 = 0x0000000000000000
        0x00007FF7BFEFB4E8 = 0x0000000000000000
        0x00007FF7BFEFB4E0 = 0x0000000000000000

00007FF7BFEFB540 000000012EFA752F System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFB5A8) = 0x00007ff7bfefb620
    LOCALS:
        0x00007FF7BFEFB5A0 = 0x0000000190064118
        0x00007FF7BFEFB598 = 0x0000000000000000
        0x00007FF7BFEFB590 = 0x0000000000000000
        0x00007FF7BFEFB588 = 0x0000000000000000

00007FF7BFEFB5D0 000000012EFA7450 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCaseRunner`1+<RunAsync>d__19[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__19<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFB5F0) = 0x00007ff7bfefb638
        stateMachine (0x00007FF7BFEFB5E0) = 0x00007ff7bfefb620

00007FF7BFEFB610 000000012EFA73BF Xunit.Sdk.TestCaseRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
    PARAMETERS:
        this (0x00007FF7BFEFB650) = 0x0000000190475e40
    LOCALS:
        0x00007FF7BFEFB620 = 0x0000000190475e40

00007FF7BFEFB670 000000012EFA666B Xunit.Sdk.XunitTestCase.RunAsync(Xunit.Abstractions.IMessageSink, Xunit.Sdk.IMessageBus, System.Object[], Xunit.Sdk.ExceptionAggregator, System.Threading.CancellationTokenSource) [/_/src/xunit.execution/Sdk/Frameworks/XunitTestCase.cs @ 162]
    PARAMETERS:
        this (0x00007FF7BFEFB6D8) = 0x00000001902c7e10
        diagnosticMessageSink (0x00007FF7BFEFB6D0) = 0x0000000190050620
        messageBus (0x00007FF7BFEFB6C8) = 0x00000001904494f8
        constructorArguments (0x00007FF7BFEFB6C0) = 0x0000000190475a58
        aggregator (0x00007FF7BFEFB6B8) = 0x0000000190475e08
        cancellationTokenSource (0x00007FF7BFEFB6B0) = 0x000000019043cc30

00007FF7BFEFB6F0 000000012EFA6539 Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(Xunit.Sdk.IXunitTestCase) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestMethodRunner.cs @ 45]
    PARAMETERS:
        this (0x00007FF7BFEFB728) = 0x0000000190475c70
        testCase (0x00007FF7BFEFB720) = 0x00000001902c7e10

00007FF7BFEFB740 000000012EFA60D8 Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs @ 136]
    PARAMETERS:
        this (0x00007FF7BFEFB808) = 0x00007ff7bfefb948
    LOCALS:
        0x00007FF7BFEFB7FC = 0x00000000ffffffff
        0x00007FF7BFEFB7F0 = 0x0000000190475c70
        0x00007FF7BFEFB7E8 = 0x0000000000000000
        0x00007FF7BFEFB7E0 = 0x00000001902c7e10
        0x00007FF7BFEFB7D8 = 0x0000000000000000
        0x00007FF7BFEFB7D0 = 0x0000000000000000
        0x00007FF7BFEFB7C8 = 0x0000000000000000

00007FF7BFEFB850 000000012EFA5E03 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFB8C8) = 0x00007ff7bfefb948
    LOCALS:
        0x00007FF7BFEFB8C0 = 0x0000000190064118
        0x00007FF7BFEFB8B8 = 0x0000000000000000
        0x00007FF7BFEFB8B0 = 0x0000000000000000
        0x00007FF7BFEFB8A8 = 0x0000000000000000

00007FF7BFEFB8F0 000000012EFA5D20 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunTestCasesAsync>d__32[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCasesAsync>d__32<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFB910) = 0x00007ff7bfefb970
        stateMachine (0x00007FF7BFEFB900) = 0x00007ff7bfefb948

00007FF7BFEFB930 000000012EFA5C85 Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCasesAsync()
    PARAMETERS:
        this (0x00007FF7BFEFB980) = 0x0000000190475c70
    LOCALS:
        0x00007FF7BFEFB948 = 0x0000000190475c70

00007FF7BFEFB9A0 000000012EFA564C Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestMethodRunner.cs @ 106]
    PARAMETERS:
        this (0x00007FF7BFEFBA90) = 0x00007ff7bfefbb98
    LOCALS:
        0x00007FF7BFEFBA84 = 0x00000000ffffffff
        0x00007FF7BFEFBA78 = 0x0000000190475c70
        0x00007FF7BFEFBA70 = 0x0000000000000000
        0x00007FF7BFEFBA68 = 0x0000000000000000
        0x00007FF7BFEFBA60 = 0x0000000000000000
        0x00007FF7BFEFBA58 = 0x0000000000000000

00007FF7BFEFBAB0 000000012EFA5429 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFBB18) = 0x00007ff7bfefbb98
    LOCALS:
        0x00007FF7BFEFBB10 = 0x0000000190064118
        0x00007FF7BFEFBB08 = 0x0000000000000000
        0x00007FF7BFEFBB00 = 0x0000000000000000
        0x00007FF7BFEFBAF8 = 0x0000000000000000

00007FF7BFEFBB40 000000012EFA5350 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestMethodRunner`1+<RunAsync>d__31[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__31<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFBB60) = 0x00007ff7bfefbbb0
        stateMachine (0x00007FF7BFEFBB50) = 0x00007ff7bfefbb98

00007FF7BFEFBB80 000000012EFA52C0 Xunit.Sdk.TestMethodRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
    PARAMETERS:
        this (0x00007FF7BFEFBBC0) = 0x0000000190475c70
    LOCALS:
        0x00007FF7BFEFBB98 = 0x0000000190475c70

00007FF7BFEFBBE0 000000012EFA4E6E Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(Xunit.Abstractions.ITestMethod, Xunit.Abstractions.IReflectionMethodInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Object[]) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestClassRunner.cs @ 168]
    PARAMETERS:
        this (0x00007FF7BFEFBC58) = 0x0000000190469ed0
        testMethod (0x00007FF7BFEFBC50) = 0x00000001902c7578
        method (0x00007FF7BFEFBC48) = 0x00000001902b9218
        testCases (0x00007FF7BFEFBC40) = 0x0000000190475bb0
        constructorArguments (0x00007FF7BFEFBC38) = 0x0000000190475a58

00007FF7BFEFBC70 000000012EFA285C Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]].MoveNext()
    PARAMETERS:
        this (0x00007FF7BFEFBF48) = 0x00007ff7bfefc080
    LOCALS:
        0x00007FF7BFEFBF3C = 0x00000000ffffffff
        0x00007FF7BFEFBF30 = 0x0000000190469ed0
        0x00007FF7BFEFBF28 = 0x0000000000000000
        0x00007FF7BFEFBF20 = 0x00000001904757a8
        0x00007FF7BFEFBF18 = 0x0000000000000000
        0x00007FF7BFEFBF10 = 0x0000000190475bb0
        0x00007FF7BFEFBF08 = 0x0000000000000000
        0x00007FF7BFEFBF00 = 0x0000000000000000
        0x00007FF7BFEFBEF8 = 0x0000000000000000

00007FF7BFEFBF90 000000012EFA17CD System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFC008) = 0x00007ff7bfefc080
    LOCALS:
        0x00007FF7BFEFC000 = 0x0000000190064118
        0x00007FF7BFEFBFF8 = 0x0000000000000000
        0x00007FF7BFEFBFF0 = 0x0000000000000000
        0x00007FF7BFEFBFE8 = 0x0000000000000000

00007FF7BFEFC030 000000012EFA1630 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+<RunTestMethodsAsync>d__38[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFC050) = 0x00007ff7bfefc0b0
        stateMachine (0x00007FF7BFEFC040) = 0x00007ff7bfefc080

00007FF7BFEFC070 000000012EFA155F Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestMethodsAsync()
    PARAMETERS:
        this (0x00007FF7BFEFC0C0) = 0x0000000190469ed0
    LOCALS:
        0x00007FF7BFEFC080 = 0x0000000190469ed0

00007FF7BFEFC0E0 000000012EF9D712 Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestClassRunner.cs @ 171]
    PARAMETERS:
        this (0x00007FF7BFEFC1E0) = 0x00007ff7bfefc2e0
    LOCALS:
        0x00007FF7BFEFC1D4 = 0x00000000ffffffff
        0x00007FF7BFEFC1C8 = 0x0000000190469ed0
        0x00007FF7BFEFC1C0 = 0x0000000000000000
        0x00007FF7BFEFC1B8 = 0x0000000190011090
        0x00007FF7BFEFC1B0 = 0x0000000000000000
        0x00007FF7BFEFC1A8 = 0x0000000000000000
        0x00007FF7BFEFC1A0 = 0x0000000000000000

00007FF7BFEFC200 000000012EF9D3DF System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__37<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFC268) = 0x00007ff7bfefc2e0
    LOCALS:
        0x00007FF7BFEFC260 = 0x0000000190064118
        0x00007FF7BFEFC258 = 0x0000000000000000
        0x00007FF7BFEFC250 = 0x0000000000000000
        0x00007FF7BFEFC248 = 0x0000000000000000

00007FF7BFEFC290 000000012EF9D300 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__37<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFC2B0) = 0x00007ff7bfefc2f8
        stateMachine (0x00007FF7BFEFC2A0) = 0x00007ff7bfefc2e0

00007FF7BFEFC2D0 000000012EF9D26F Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
    PARAMETERS:
        this (0x00007FF7BFEFC310) = 0x0000000190469ed0
    LOCALS:
        0x00007FF7BFEFC2E0 = 0x0000000190469ed0

00007FF7BFEFC330 000000012EF9CD42 Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(Xunit.Abstractions.ITestClass, Xunit.Abstractions.IReflectionTypeInfo, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestCollectionRunner.cs @ 158]
    PARAMETERS:
        this (0x00007FF7BFEFC3A8) = 0x0000000190469b28
        testClass (0x00007FF7BFEFC3A0) = 0x00000001902b4898
        class (0x00007FF7BFEFC398) = 0x00000001902b4880
        testCases (0x00007FF7BFEFC390) = 0x0000000190469e10

00007FF7BFEFC3C0 000000012EF9C653 Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]].MoveNext()
    PARAMETERS:
        this (0x00007FF7BFEFC578) = 0x00007ff7bfefc6b8
    LOCALS:
        0x00007FF7BFEFC56C = 0x00000000ffffffff
        0x00007FF7BFEFC560 = 0x0000000190469b28
        0x00007FF7BFEFC558 = 0x0000000000000000
        0x00007FF7BFEFC550 = 0x0000000190469e10
        0x00007FF7BFEFC548 = 0x0000000000000000
        0x00007FF7BFEFC540 = 0x0000000000000000
        0x00007FF7BFEFC538 = 0x0000000000000000

00007FF7BFEFC5C0 000000012EF9B733 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestClassesAsync>d__28<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFC638) = 0x00007ff7bfefc6b8
    LOCALS:
        0x00007FF7BFEFC630 = 0x0000000190064118
        0x00007FF7BFEFC628 = 0x0000000000000000
        0x00007FF7BFEFC620 = 0x0000000000000000
        0x00007FF7BFEFC618 = 0x0000000000000000

00007FF7BFEFC660 000000012EF9B650 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestClassesAsync>d__28<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFC680) = 0x00007ff7bfefc6e0
        stateMachine (0x00007FF7BFEFC670) = 0x00007ff7bfefc6b8

00007FF7BFEFC6A0 000000012EF9B5B5 Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestClassesAsync()
    PARAMETERS:
        this (0x00007FF7BFEFC6F0) = 0x0000000190469b28
    LOCALS:
        0x00007FF7BFEFC6B8 = 0x0000000190469b28

00007FF7BFEFC710 000000012EF980F3 Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs @ 101]
    PARAMETERS:
        this (0x00007FF7BFEFC810) = 0x00007ff7bfefc910
    LOCALS:
        0x00007FF7BFEFC804 = 0x00000000ffffffff
        0x00007FF7BFEFC7F8 = 0x0000000190469b28
        0x00007FF7BFEFC7F0 = 0x0000000000000000
        0x00007FF7BFEFC7E8 = 0x0000000190011090
        0x00007FF7BFEFC7E0 = 0x0000000000000000
        0x00007FF7BFEFC7D8 = 0x0000000000000000
        0x00007FF7BFEFC7D0 = 0x0000000000000000

00007FF7BFEFC830 000000012EF9734F System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__27<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFC898) = 0x00007ff7bfefc910
    LOCALS:
        0x00007FF7BFEFC890 = 0x0000000190064118
        0x00007FF7BFEFC888 = 0x0000000000000000
        0x00007FF7BFEFC880 = 0x0000000000000000
        0x00007FF7BFEFC878 = 0x0000000000000000

00007FF7BFEFC8C0 000000012EF971A0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__27<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFC8E0) = 0x00007ff7bfefc928
        stateMachine (0x00007FF7BFEFC8D0) = 0x00007ff7bfefc910

00007FF7BFEFC900 000000012EF96ACF Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
    PARAMETERS:
        this (0x00007FF7BFEFC940) = 0x0000000190469b28
    LOCALS:
        0x00007FF7BFEFC910 = 0x0000000190469b28

00007FF7BFEFC960 000000012EF94F95 Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(Xunit.Sdk.IMessageBus, Xunit.Abstractions.ITestCollection, System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, System.Threading.CancellationTokenSource) [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs @ 235]
    PARAMETERS:
        this (0x00007FF7BFEFC9B8) = 0x000000019043cb58
        messageBus (0x00007FF7BFEFC9B0) = 0x00000001904494f8
        testCollection (0x00007FF7BFEFC9A8) = 0x00000001902b6210
        testCases (0x00007FF7BFEFC9A0) = 0x00000001904698a8
        cancellationTokenSource (0x00007FF7BFEFC998) = 0x000000019043cc30

00007FF7BFEFC9D0 000000012EF8E52E Xunit.Sdk.TestAssemblyRunner`1+<RunTestCollectionsAsync>d__42[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestAssemblyRunner.cs @ 231]
    PARAMETERS:
        this (0x00007FF7BFEFCA98) = 0x00007ff7bfefcbf8
    LOCALS:
        0x00007FF7BFEFCA8C = 0x00000000ffffffff
        0x00007FF7BFEFCA80 = 0x000000019043cb58
        0x00007FF7BFEFCA78 = 0x0000000000000000
        0x00007FF7BFEFCA70 = 0x0000000190469ad0
        0x00007FF7BFEFCA68 = 0x0000000000000000
        0x00007FF7BFEFCA60 = 0x0000000000000000
        0x00007FF7BFEFCA58 = 0x0000000000000000

00007FF7BFEFCAE0 000000012EF8DC23 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunTestCollectionsAsync>d__42[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCollectionsAsync>d__42<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFCB78) = 0x00007ff7bfefcbf8
    LOCALS:
        0x00007FF7BFEFCB70 = 0x0000000190064118
        0x00007FF7BFEFCB68 = 0x0000000000000000
        0x00007FF7BFEFCB60 = 0x0000000000000000
        0x00007FF7BFEFCB58 = 0x0000000000000000

00007FF7BFEFCBA0 000000012EF8DA30 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunTestCollectionsAsync>d__42[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunTestCollectionsAsync>d__42<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFCBC0) = 0x00007ff7bfefcc28
        stateMachine (0x00007FF7BFEFCBB0) = 0x00007ff7bfefcbf8

00007FF7BFEFCBE0 000000012EF8D994 Xunit.Sdk.TestAssemblyRunner`1[[System.__Canon, System.Private.CoreLib]].RunTestCollectionsAsync(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource)
    PARAMETERS:
        this (0x00007FF7BFEFCC60) = 0x000000019043cb58
        messageBus (0x00007FF7BFEFCC58) = 0x00000001904494f8
        cancellationTokenSource (0x00007FF7BFEFCC50) = 0x000000019043cc30
    LOCALS:
        0x00007FF7BFEFCBF8 = 0x000000019043cb58

00007FF7BFEFCC80 000000012EF8D678 Xunit.Sdk.XunitTestAssemblyRunner.<>n__0(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource)
    PARAMETERS:
        this (0x00007FF7BFEFCC98) = 0x000000019043cb58
        messageBus (0x00007FF7BFEFCC90) = 0x00000001904494f8
        cancellationTokenSource (0x00007FF7BFEFCC88) = 0x000000019043cc30

00007FF7BFEFCCB0 000000012EF8C155 Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14.MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/XunitTestAssemblyRunner.cs @ 165]
    PARAMETERS:
        this (0x00007FF7BFEFCF20) = 0x00007ff7bfefd060
    LOCALS:
        0x00007FF7BFEFCF1C = 0x00000000ffffffff
        0x00007FF7BFEFCF10 = 0x000000019043cb58
        0x00007FF7BFEFCF08 = 0x0000000000000000
        0x00007FF7BFEFCF00 = 0x0000000000000000
        0x00007FF7BFEFCEF8 = 0x0000000000000000
        0x00007FF7BFEFCEF0 = 0x0000000000000000
        0x00007FF7BFEFCED8 = 0x0000000000000000
        0x00007FF7BFEFCED0 = 0x0000000000000000
        0x00007FF7BFEFCEC8 = 0x0000000000000000
        0x00007FF7BFEFCEC0 = 0x0000000000000000
        0x00007FF7BFEFCEB8 = 0x0000000000000000
        0x00007FF7BFEFCEB0 = 0x0000000000000000
        0x00007FF7BFEFCEA8 = 0x0000000000000000

00007FF7BFEFCF60 000000012EF8B454 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14, xunit.execution.dotnet]](<RunTestCollectionsAsync>d__14 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFD018) = 0x00007ff7bfefd060
    LOCALS:
        0x00007FF7BFEFD010 = 0x0000000190064118
        0x00007FF7BFEFD008 = 0x0000000000000000
        0x00007FF7BFEFD000 = 0x0000000000000000
        0x00007FF7BFEFCFF8 = 0x0000000000000000

00007FF7BFEFD030 000000012EF8B3B0 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.XunitTestAssemblyRunner+<RunTestCollectionsAsync>d__14, xunit.execution.dotnet]](<RunTestCollectionsAsync>d__14 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFD048) = 0x00007ff7bfefd0a8
        stateMachine (0x00007FF7BFEFD038) = 0x00007ff7bfefd060

00007FF7BFEFD060 000000012EF8B355 Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionsAsync(Xunit.Sdk.IMessageBus, System.Threading.CancellationTokenSource)
    PARAMETERS:
        this (0x00007FF7BFEFD0F8) = 0x000000019043cb58
        messageBus (0x00007FF7BFEFD0F0) = 0x00000001904494f8
        cancellationTokenSource (0x00007FF7BFEFD0E8) = 0x000000019043cc30
    LOCALS:
        0x00007FF7BFEFD060 = 0x000000019043cc30

00007FF7BFEFD110 000000012ED7883F Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib]].MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/Runners/TestAssemblyRunner.cs @ 196]
    PARAMETERS:
        this (0x00007FF7BFEFD260) = 0x00007ff7bfefd388
    LOCALS:
        0x00007FF7BFEFD254 = 0x00000000ffffffff
        0x00007FF7BFEFD248 = 0x000000019043cb58
        0x00007FF7BFEFD240 = 0x0000000000000000
        0x00007FF7BFEFD238 = 0x0000000190449470
        0x00007FF7BFEFD230 = 0x0000000190063558
        0x00007FF7BFEFD228 = 0x0000000190469258
        0x00007FF7BFEFD220 = 0x0000000000000000
        0x00007FF7BFEFD218 = 0x0000000000000000
        0x00007FF7BFEFD210 = 0x0000000000000000
        0x00007FF7BFEFD208 = 0x0000000000000000

00007FF7BFEFD280 000000012ED78468 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__41<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFD308) = 0x00007ff7bfefd388
    LOCALS:
        0x00007FF7BFEFD300 = 0x0000000190064118
        0x00007FF7BFEFD2F8 = 0x0000000000000000
        0x00007FF7BFEFD2F0 = 0x0000000000000000
        0x00007FF7BFEFD2E8 = 0x0000000000000000

00007FF7BFEFD330 000000012ED78380 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Xunit.Sdk.TestAssemblyRunner`1+<RunAsync>d__41[[System.__Canon, System.Private.CoreLib]], xunit.execution.dotnet]](<RunAsync>d__41<System.__Canon> ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 35]
    PARAMETERS:
        this (0x00007FF7BFEFD350) = 0x00007ff7bfefd3b8
        stateMachine (0x00007FF7BFEFD340) = 0x00007ff7bfefd388

00007FF7BFEFD370 000000012ED782A5 Xunit.Sdk.TestAssemblyRunner`1[[System.__Canon, System.Private.CoreLib]].RunAsync()
    PARAMETERS:
        this (0x00007FF7BFEFD3D0) = 0x000000019043cb58
    LOCALS:
        0x00007FF7BFEFD388 = 0x000000019043cb58

00007FF7BFEFD3F0 000000012ED77BDF Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8.MoveNext() [/_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkExecutor.cs @ 95]
    PARAMETERS:
        this (0x00007FF7BFEFD448) = 0x00007ff7bfefd528
    LOCALS:
        0x00007FF7BFEFD444 = 0x00000000ffffffff
        0x00007FF7BFEFD438 = 0x0000000190063668
        0x00007FF7BFEFD430 = 0x0000000000000000
        0x00007FF7BFEFD428 = 0x0000000000000000

00007FF7BFEFD460 000000012ED77A6E System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8, xunit.execution.dotnet]](<RunTestCases>d__8 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
    PARAMETERS:
        stateMachine (0x00007FF7BFEFD4D8) = 0x00007ff7bfefd528
    LOCALS:
        0x00007FF7BFEFD4D0 = 0x0000000190064118
        0x00007FF7BFEFD4C8 = 0x0000000000000000
        0x00007FF7BFEFD4C0 = 0x0000000000000000
        0x00007FF7BFEFD4B8 = 0x0000000000000000

00007FF7BFEFD4F0 000000012ED779DC System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start[[Xunit.Sdk.XunitTestFrameworkExecutor+<RunTestCases>d__8, xunit.execution.dotnet]](<RunTestCases>d__8 ByRef) [/Users/furt/github/wfurt-runtime2/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncVoidMethodBuilder.cs @ 40]
    PARAMETERS:
        this (0x00007FF7BFEFD4F8) = 0x00007ff7bfefd558
        stateMachine (0x00007FF7BFEFD4F0) = 0x00007ff7bfefd528

00007FF7BFEFD510 000000012ED7791A Xunit.Sdk.XunitTestFrameworkExecutor.RunTestCases(System.Collections.Generic.IEnumerable`1<Xunit.Sdk.IXunitTestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions)
    PARAMETERS:
        this (0x00007FF7BFEFD588) = 0x0000000190063668
        testCases (0x00007FF7BFEFD580) = 0x000000019043cb18
        executionMessageSink (0x00007FF7BFEFD578) = 0x000000019043ca40
        executionOptions (0x00007FF7BFEFD570) = 0x0000000190034ef8
    LOCALS:
        0x00007FF7BFEFD528 = 0x0000000190063668

00007FF7BFEFD5A0 000000012ED77869 Xunit.Sdk.TestFrameworkExecutor`1[[System.__Canon, System.Private.CoreLib]].RunTests(System.Collections.Generic.IEnumerable`1<Xunit.Abstractions.ITestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions) [/_/src/xunit.execution/Sdk/Frameworks/TestFrameworkExecutor.cs @ 100]
    PARAMETERS:
        this (0x00007FF7BFEFD5D0) = 0x0000000190063668
        testCases (0x00007FF7BFEFD5C8) = 0x0000000190393dd8
        executionMessageSink (0x00007FF7BFEFD5C0) = 0x000000019043ca40
        executionOptions (0x00007FF7BFEFD5B8) = 0x0000000190034ef8

00007FF7BFEFD5F0 000000012ED7776F Xunit.Xunit2.RunTests(System.Collections.Generic.IEnumerable`1<Xunit.Abstractions.ITestCase>, Xunit.Abstractions.IMessageSink, Xunit.Abstractions.ITestFrameworkExecutionOptions) [/_/src/xunit.runner.utility/Frameworks/v2/Xunit2.cs @ 110]
    PARAMETERS:
        this (0x00007FF7BFEFD618) = 0x00000001900504c8
        testCases (0x00007FF7BFEFD610) = 0x0000000190393dd8
        messageSink (0x00007FF7BFEFD608) = 0x000000019043bbc0
        executionOptions (0x00007FF7BFEFD600) = 0x0000000190034ef8

00007FF7BFEFD630 000000012E29C866 Xunit.ConsoleClient.ConsoleRunner.ExecuteAssembly(System.Object, Xunit.XunitProjectAssembly, Boolean, Boolean, System.Nullable`1<Boolean>, System.Nullable`1<Int32>, Boolean, Boolean, System.Nullable`1<Xunit.AppDomainSupport>, Boolean, Boolean, Xunit.XunitFilters, Boolean) [/_/src/Microsoft.DotNet.XUnitConsoleRunner/src/ConsoleRunner.cs @ 396]
    PARAMETERS:
        this (<CLR reg>) = 0x0000000190071720
        consoleLock = <no data>
        assembly (<CLR reg>) = 0x0000000190073268
        serialize = <no data>
        needsXml = <no data>
        parallelizeTestCollections = <no data>
        maxThreadCount = <no data>
        diagnosticMessages = <no data>
        noColor = <no data>
        appDomains = <no data>
        failSkips = <no data>
        stopOnFail (0x00007FF7BFEFD7E8) = 0x0000000000000000
        filters = <no data>
        internalDiagnosticMessages (0x00007FF7BFEFD7F8) = 0x0000000000000000
    LOCALS:
        0x00007FF7BFEFD748 = 0x0000000190034a18
        <no data>
        0x00007FF7BFEFD738 = 0x0000000190034ef8
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        0x00007FF7BFEFD728 = 0x00000001901da158
        0x00007FF7BFEFD720 = 0x00000001900502c8
        0x00007FF7BFEFD718 = 0x00000001900503a0
        <no data>
        <no data>
        <no data>
        0x00007FF7BFEFD710 = 0x000000019043b860
        <no data>
        <no data>

00007FF7BFEFD7C0 000000012E298288 Xunit.ConsoleClient.ConsoleRunner.RunProject(Xunit.XunitProject, Boolean, System.Nullable`1<Boolean>, System.Nullable`1<Boolean>, System.Nullable`1<Int32>, Boolean, Boolean, System.Nullable`1<Xunit.AppDomainSupport>, Boolean, Boolean, Boolean) [/_/src/Microsoft.DotNet.XUnitConsoleRunner/src/ConsoleRunner.cs @ 293]
    PARAMETERS:
        this (0x00007FF7BFEFDAA0) = 0x0000000190071720
        project (0x00007FF7BFEFDA98) = 0x000000019002ad50
        serialize (0x00007FF7BFEFDA94) = 0x0000000000000000
        parallelizeAssemblies (0x00007FF7BFEFDA88) = 0x0000000000000001
        parallelizeTestCollections (0x00007FF7BFEFDA80) = 0x0000000100000001
        maxThreadCount (0x00007FF7BFEFDA78) = 0x0000000000000000
        diagnosticMessages (0x00007FF7BFEFDAE0) = 0x0000000000000000
        noColor (0x00007FF7BFEFDAE8) = 0x0000000000000001
        appDomains (0x00007FF7BFEFDAF0) = 0x0000000000000000
        failSkips (0x00007FF7BFEFDAF8) = 0x0000000000000000
        stopOnFail (0x00007FF7BFEFDB00) = 0x0000000000000000
        internalDiagnosticMessages (0x00007FF7BFEFDB08) = 0x0000000000000000
    LOCALS:
        0x00007FF7BFEFDA70 = 0x0000000190034438
        0x00007FF7BFEFDA68 = 0x0000000190034480
        0x00007FF7BFEFDA60 = 0x00000001900344e8
        0x00007FF7BFEFDA58 = 0x0000000190034890
        0x00007FF7BFEFDA50 = 0x0000000000000000
        0x00007FF7BFEFDA48 = 0x0000000000000000
        0x00007FF7BFEFDA40 = 0x0000000000000000
        0x00007FF7BFEFDA38 = 0x0000000000000000
        0x00007FF7BFEFDA30 = 0x0000000000000000
        0x00007FF7BFEFDA28 = 0x0000000190034960
        0x00007FF7BFEFDA20 = 0x0000000190073268
        0x00007FF7BFEFDA18 = 0x0000000000000000
        0x00007FF7BFEFDA10 = 0x0000000000000000

00007FF7BFEFDAE0 000000012E258C45 Xunit.ConsoleClient.ConsoleRunner.EntryPoint(System.String[]) [/_/src/Microsoft.DotNet.XUnitConsoleRunner/src/ConsoleRunner.cs @ 81]
    PARAMETERS:
        this (0x00007FF7BFEFDC18) = 0x0000000190071720
        args (0x00007FF7BFEFDC10) = 0x000000019000ef60
    LOCALS:
        0x00007FF7BFEFDC08 = 0x000000019002b8e0
        0x00007FF7BFEFDC00 = 0x0000000190032900
        0x00007FF7BFEFDBF8 = 0x000000019002bce0
        0x00007FF7BFEFDBF4 = 0x0000000000000000
        0x00007FF7BFEFDBF0 = 0x0000000000000000
        0x00007FF7BFEFDBE8 = 0x0000000000000000

00007FF7BFEFDC30 000000012DCC6AF9 Xunit.ConsoleClient.Program.Main(System.String[]) [/_/src/Microsoft.DotNet.XUnitConsoleRunner/src/Program.cs @ 16]
    PARAMETERS:
        args (0x00007FF7BFEFDC78) = 0x000000019000ef60
    LOCALS:
        0x00007FF7BFEFDC70 = 0x000000019000f930
        0x00007FF7BFEFDC68 = 0x000000019000f960
        0x00007FF7BFEFDC60 = 0x000000019000f9b8
        0x00007FF7BFEFDC5C = 0x0000000000000000

I'm not 100% sure if this is the same issue but I can provide more info if needed.

cshung commented 8 months ago

@wfurt, can we have the native stack?

wfurt commented 8 months ago

this?

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x0000000104664b33 libcoreclr.dylib`PAL_ThrowExceptionFromContext + 19
    frame #1: 0x000000010489e86e libcoreclr.dylib`UnwindManagedExceptionPass2(PAL_SEHException&, _CONTEXT*) + 990
    frame #2: 0x000000010489e9ed libcoreclr.dylib`UnwindManagedExceptionPass1(PAL_SEHException&, _CONTEXT*) + 349
    frame #3: 0x000000010489ed1e libcoreclr.dylib`DispatchManagedException(PAL_SEHException&, bool) + 78
    frame #4: 0x0000000104810781 libcoreclr.dylib`IL_Throw(Object*) + 481
    frame #5: 0x000000012efb57c2
    frame #6: 0x000000012efb56f2
    frame #7: 0x000000012efb2de5
    frame #8: 0x000000012efb2948
    frame #9: 0x000000012efb25fd
    frame #10: 0x000000012efb05ed
    frame #11: 0x0000000104952e81 libcoreclr.dylib`CallDescrWorkerInternal + 124
    frame #12: 0x00000001047b18cc libcoreclr.dylib`CallDescrWorkerWithHandler(CallDescrData*, int) + 92
    frame #13: 0x000000010485729b libcoreclr.dylib`RuntimeMethodHandle::InvokeMethod(Object*, void**, SignatureNative*, bool) + 1931
    frame #14: 0x000000012e28696d
    frame #15: 0x000000012e26e18f
    frame #16: 0x000000012e28537b
    frame #17: 0x000000012e285267
    frame #18: 0x000000012efb056c
    frame #19: 0x000000012efb018f
    frame #20: 0x000000012efafa09
    frame #21: 0x000000012efaf930
    frame #22: 0x000000012efaf8a6
    frame #23: 0x000000012efaf65e
    frame #24: 0x000000012efaf541
    frame #25: 0x000000012efaf4ac
    frame #26: 0x000000012efaf460
    frame #27: 0x000000012efaf3f3
    frame #28: 0x000000012efaf1b1
    frame #29: 0x000000012efaf0bb
    frame #30: 0x000000012efaf02c
    frame #31: 0x000000012efaefe1
    frame #32: 0x000000012efae86b
    frame #33: 0x000000012efae5bf
    frame #34: 0x000000012efae4e0
    frame #35: 0x000000012efae456
    frame #36: 0x000000012efae28a
    frame #37: 0x000000012efaccac
    frame #38: 0x000000012efac7f3
    frame #39: 0x000000012efac710
    frame #40: 0x000000012efac68b
    frame #41: 0x000000012efac42f
    frame #42: 0x000000012efac32b
    frame #43: 0x000000012efac29c
    frame #44: 0x000000012efac211
    frame #45: 0x000000012efac171
    frame #46: 0x000000012efabb4b
    frame #47: 0x000000012efab7bb
    frame #48: 0x000000012efab4e2
    frame #49: 0x000000012efab450
    frame #50: 0x000000012efab3fa
    frame #51: 0x000000012efab36a
    frame #52: 0x000000012efab018
    frame #53: 0x000000012efaaef9
    frame #54: 0x000000012efaae20
    frame #55: 0x000000012efaac41
    frame #56: 0x000000012efaa160
    frame #57: 0x000000012efa9d03
    frame #58: 0x000000012efa9c20
    frame #59: 0x000000012efa9b85
    frame #60: 0x000000012efa7ed6
    frame #61: 0x000000012efa781a
    frame #62: 0x000000012efa752f
    frame #63: 0x000000012efa7450
    frame #64: 0x000000012efa73bf
    frame #65: 0x000000012efa666b
    frame #66: 0x000000012efa6539
    frame #67: 0x000000012efa60d8
    frame #68: 0x000000012efa5e03
    frame #69: 0x000000012efa5d20
    frame #70: 0x000000012efa5c85
    frame #71: 0x000000012efa564c
    frame #72: 0x000000012efa5429
    frame #73: 0x000000012efa5350
    frame #74: 0x000000012efa52c0
    frame #75: 0x000000012efa4e6e
    frame #76: 0x000000012efa285c
    frame #77: 0x000000012efa17cd
    frame #78: 0x000000012efa1630
    frame #79: 0x000000012efa155f
    frame #80: 0x000000012ef9d712
    frame #81: 0x000000012ef9d3df
    frame #82: 0x000000012ef9d300
    frame #83: 0x000000012ef9d26f
    frame #84: 0x000000012ef9cd42
    frame #85: 0x000000012ef9c653
    frame #86: 0x000000012ef9b733
    frame #87: 0x000000012ef9b650
    frame #88: 0x000000012ef9b5b5
    frame #89: 0x000000012ef980f3
    frame #90: 0x000000012ef9734f
    frame #91: 0x000000012ef971a0
    frame #92: 0x000000012ef96acf
    frame #93: 0x000000012ef94f95
    frame #94: 0x000000012ef8e52e
    frame #95: 0x000000012ef8dc23
    frame #96: 0x000000012ef8da30
    frame #97: 0x000000012ef8d994
    frame #98: 0x000000012ef8d678
    frame #99: 0x000000012ef8c155
    frame #100: 0x000000012ef8b454
    frame #101: 0x000000012ef8b3b0
    frame #102: 0x000000012ef8b355
    frame #103: 0x000000012ed7883f
    frame #104: 0x000000012ed78468
    frame #105: 0x000000012ed78380
    frame #106: 0x000000012ed782a5
    frame #107: 0x000000012ed77bdf
    frame #108: 0x000000012ed77a6e
    frame #109: 0x000000012ed779dc
    frame #110: 0x000000012ed7791a
    frame #111: 0x000000012ed77869
    frame #112: 0x000000012ed7776f
    frame #113: 0x000000012e29c866
    frame #114: 0x000000012e298288
    frame #115: 0x000000012e258c45
    frame #116: 0x000000012dcc6af9
    frame #117: 0x0000000104952e81 libcoreclr.dylib`CallDescrWorkerInternal + 124
    frame #118: 0x00000001047b21e3 libcoreclr.dylib`MethodDescCallSite::CallTargetWorker(unsigned long long const*, unsigned long long*, int) + 1587
    frame #119: 0x00000001046ae076 libcoreclr.dylib`RunMain(MethodDesc*, short, int*, PtrArray**) + 742
    frame #120: 0x00000001046ae462 libcoreclr.dylib`Assembly::ExecuteMainMethod(PtrArray**, int) + 450
    frame #121: 0x00000001046d6f61 libcoreclr.dylib`CorHost2::ExecuteAssembly(unsigned int, char16_t const*, int, char16_t const**, unsigned int*) + 689
    frame #122: 0x000000010469b9b2 libcoreclr.dylib`coreclr_execute_assembly + 226
    frame #123: 0x0000000103198b84 libhostpolicy.dylib`coreclr_t::execute_assembly(int, char const**, char const*, unsigned int*) + 132
    frame #124: 0x00000001031ddf8d libhostpolicy.dylib`run_app_for_context(hostpolicy_context_t const&, int, char const**) + 1117
    frame #125: 0x00000001031de212 libhostpolicy.dylib`run_app(int, char const**) + 98
    frame #126: 0x00000001031dee72 libhostpolicy.dylib`corehost_main + 434
    frame #127: 0x0000000100557f0b libhostfxr.dylib`execute_app(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, corehost_init_t*, int, char const**) + 779
    frame #128: 0x000000010054fc50 libhostfxr.dylib`(anonymous namespace)::read_config_and_execute(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, host_startup_info_t const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::unordered_map<known_options, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>, known_options_hash, std::__1::equal_to<known_options>, std::__1::allocator<std::__1::pair<known_options const, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>>>> const&, int, char const**, host_mode_t, bool, char*, int, int*) + 272
    frame #129: 0x000000010054ca2a libhostfxr.dylib`fx_muxer_t::handle_exec_host_command(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, host_startup_info_t const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::unordered_map<known_options, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>, known_options_hash, std::__1::equal_to<known_options>, std::__1::allocator<std::__1::pair<known_options const, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>>>> const&, int, char const**, int, host_mode_t, bool, char*, int, int*) + 426
    frame #130: 0x000000010054bd87 libhostfxr.dylib`fx_muxer_t::execute(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, int, char const**, host_startup_info_t const&, char*, int, int*) + 423
    frame #131: 0x0000000100543a8e libhostfxr.dylib`hostfxr_main_startupinfo + 190
    frame #132: 0x0000000100011476 dotnet`exe_start(int, char const**) + 1446
    frame #133: 0x00000001000118a8 dotnet`main + 216
    frame #134: 0x00007ff80bdd141f dyld`start + 1903

I can reproduce while running under lldb. I'm working on binary search now to isolate breaking change.

cshung commented 8 months ago

@wfurt, yes, this is what I meant by native stack. This doesn't look like the GC issue we are investigating though, instead,this looks like a managed exception.

wfurt commented 8 months ago

OK. The part I'm trying understand is what it would crash the whole test process. I tried several test suites and I get the same results. I could try the coreclr tests as well as Mac is my primary machine.

jaredpar commented 8 months ago

Any updates here? This bug is blocking our OSX testing as it results in a near 100% failure rate.

cshung commented 8 months ago

I wasn't actively working on it for the last few days. I am happy to if I can get myself some useful information that I use.

jaredpar commented 8 months ago

We've offered to change our repo to give any information possible. We are bound by the limits of Helix so full dumps aren't possible.

I wasn't actively working on it for the last few days

This is not very encouraging. There appears to be a runtime bug that blocks the compiler from executing correctly on one of our key operating systems.

cshung commented 8 months ago

We've offered to change our repo to give any information possible.

Is it possible to run the CI with an instrumented runtime that emit logs instead of dump files to workaround the limitation that dumps are unavailable?

jaredpar commented 8 months ago

Is it possible to run the CI with an instrumented runtime that emit logs instead of dump files to workaround the limitation that dumps are unavailable?

Yes. Is there a runtime available that is instrumented that we can use + instructions on how to use it?

cshung commented 8 months ago

Is it possible to run the CI with an instrumented runtime that emit logs instead of dump files to workaround the limitation that dumps are unavailable?

Yes. Is there a runtime available that is instrumented that we can use + instructions on how to use it?

There is none yet, but I can work on instrumenting it. Since we always crash on mark_ro_segment, we would like to know which line is failing there, it can be as easily as printf("line number\n"); between every line there. Assuming I am logging to stdout, will that information be captured?

jaredpar commented 8 months ago

Assuming I am logging to stdout, will that information be captured?

I think so. The stdout for xunit is captured in the logs so I would expect stdout in general to be captured. The dotnet test is a multi-process event which is why I have a slight bit of uncertainty.

cshung commented 8 months ago

I am working on the instrumentation now, when I am done, I will push a branch to my fork where I have the code changes. As of now, the changes are concentrated in gc.cpp, so it will suffice to just change the GC.

To make the managed process use the new GC, we need to build a libclrgc.dylib next to the loaded libcoreclr.dylib, and then set the environment variable DOTNET_GCName to libclrgc.dyilb, that should make the processes use the new GC.

For the first instrumentation round, all I really cared is the very tip of the log when it crashes. Should I worry about the possibility that multiple processes share the same stdout and the log could come from multiple processes?

jaredpar commented 8 months ago

To make the managed process use the new GC, we need to build a libclrgc.dylib next to the loaded libcoreclr.dylib, and then set the environment variable DOTNET_GCName to libclrgc.dyilb, that should make the processes use the new GC.

Let me know when you have it ready and I'll hack our build to use it.

Should I worry about the possibility that multiple processes share the same stdout and the log could come from multiple processes?

Think we'll have to just try and see for now.

cshung commented 8 months ago

I have got the instrumentation ready, can we try this branch?

https://github.com/cshung/runtime/tree/public/mark-ro-instrumentation

Don't worry about rebasing it, clrgc is meant to work across versions.

jaredpar commented 8 months ago

@cshung

How do we get an OSX dotnet host out of that repo? If you have one locally could you just share it out and I'll pubish it around.

cshung commented 8 months ago

@cshung

How do we get an OSX dotnet host out of that repo? If you have one locally could you just share it out and I'll pubish it around.

I am hoping it suffice to just replace the GC, we did quite some work in .net 6/7 time frame to make that possible. By just replacing the GC, we have better chance to reproduce the issue since none of the other things (e.g. vm/jit/libraries) are changed.

Are you hitting some issues with that? I maybe able to figure out what that is.

If you also wanted/needed to replace the dotnet host, then I am not sure how to do. @elinor-fung might know how.

jkotas commented 8 months ago

dotnet host

Build the branch using ./build.sh -c release. The host will be dropped under artifacts/bin/osx-x64.Release/corehost.

As @cshung said, it should not be necessary. Dropping private libclrgc.dylib and setting env variable should be sufficient.

jaredpar commented 8 months ago

The problem is that our devs don't have OSX machines so building ourselves isn't going to help. Our OSX machine access is through CI only. That is why I was asking if you could share the built GC out and I can take it from there.

jkotas commented 8 months ago

@cshung Can you queue draft PR with your change? It is the easiest way to get the instrumented GC binary built and shared.

jaredpar commented 8 months ago

@agocke was awesome enough to grab the most recent Roslyn failure bits from Helix, drop them on his Mac and run the tests in a loop for three hours. Eventually he got a full crash dump (compressed) for the problem. The dump should be accessible to anyone at Microsoft. Warning: it's 12GB.

cshung commented 8 months ago

Thanks @jaredpar and @agocke, I am trying to open the dump. I am able to download the dump and open it on a osx arm64 machine, but I can't extract useful information out of it.

Here is what I tried:

andrewau@boron dotnet % lldb ./dotnet --core ~/Desktop/core\ -\ Copy.97526
(lldb) target create "./dotnet" --core "/Users/andrewau/Desktop/core - Copy.97526"
Core file '/Users/andrewau/Desktop/core - Copy.97526' (arm64) was loaded.
(lldb) bt
* thread #10, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x1187ce2e8)
  * frame #0: 0x00000001061338f8
(lldb) bt all
  thread #1
    frame #0: 0x00007ff8a45d6a78
  thread #2
    frame #0: 0x00007ff7ffdd3294
    frame #1: 0x00007ff7ffddf524
    frame #2: 0x00007ff7ffde0e40
  thread #3
    frame #0: 0x00007ff8a45d6a78
  thread #4
    frame #0: 0x00007ff8a45d6a78
  thread #5
    frame #0: 0x00007ff8a45d6a78
  thread #6
    frame #0: 0x00007ff8a45d6a78
  thread #7
    frame #0: 0x00007ff8a45d6a78
  thread #8
    frame #0: 0x00007ff8a45d6a78
  thread #9
    frame #0: 0x00007ff8a45d6a78
* thread #10, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x1187ce2e8)
  * frame #0: 0x00000001061338f8
  thread #11
    frame #0: 0x00007ff8a45d6a78
  thread #12
    frame #0: 0x00007ff8a45d6a78
  thread #13
    frame #0: 0x00007ff8a45d6a78
  thread #14
    frame #0: 0x00007ff8a45d6a78
  thread #15
    frame #0: 0x00007ff8a45d6a78
  thread #16
    frame #0: 0x00007ff8a45d6a78
  thread #17
    frame #0: 0x00007ff8a45d6a78
  thread #18
    frame #0: 0x00007ff8a45d6a78
  thread #19
    frame #0: 0x00007ff8a45d6a78
  thread #20
    frame #0: 0x00007ff8a45d6a78
  thread #21
    frame #0: 0x00007ff8a45d6a78
  thread #22
    frame #0: 0x00007ff8a45d6a78
  thread #23
    frame #0: 0x00007ff8a45d6a78
  thread #24
    frame #0: 0x00007ff8a45d6a78
  thread #25
    frame #0: 0x00007ff8a45d6a78
  thread #26
    frame #0: 0x00007ff8a45d6a78
  thread #27
    frame #0: 0x00007ff8a45d6a78
(lldb) target modules list
[  0] 2026C5B3-94DF-38EC-9053-6CF87776BE3E 0x0000000100000000 /Users/andrewau/toolssw/dotnet/dotnet 
[  1] 324E4AD9-E01F-3183-B09F-3E20B326643A 0x00000001800b7000 /usr/lib/dyld 
[  2] 375254EB-5A65-35D0-B6B4-773E6763D289 0x0000000180352000 /usr/lib/libc++.1.dylib 
[  3] C9A4714E-2961-3FBD-8E74-8615B3A5777E 0x000000018cccd000 /usr/lib/libSystem.B.dylib 
[  4] E812122A-DA5B-3F1A-A15C-936C0B9D26BE 0x00000001803e0000 /usr/lib/libc++abi.dylib 
[  5] B0558732-E048-33CE-A6D7-2EC929BFCA1D 0x000000018ccc7000 /usr/lib/system/libcache.dylib 
[  6] 1A4B497A-69D2-3048-8F47-133482DABAFA 0x000000018cc82000 /usr/lib/system/libcommonCrypto.dylib 
[  7] DE627B59-A9DD-30C5-8F6D-BEF3949C64A1 0x000000018ccae000 /usr/lib/system/libcompiler_rt.dylib 
[  8] 11CC75EB-6597-33C6-B725-607F7E0B0A28 0x000000018cca4000 /usr/lib/system/libcopyfile.dylib 
[  9] 61DEF451-18FC-3FA7-B05A-F80B0BF36404 0x00000001801b3000 /usr/lib/system/libcorecrypto.dylib 
[ 10] A53D555D-F748-3010-83FE-385C660A81BD 0x0000000180289000 /usr/lib/system/libdispatch.dylib 
[ 11] 33108EF5-2DC1-3E6D-B3BE-C55520E1594F 0x0000000180444000 /usr/lib/system/libdyld.dylib 
[ 12] FCB5DECB-A48D-3B82-9E12-03F2AFA3B0F9 0x000000018ccbd000 /usr/lib/system/libkeymgr.dylib 
[ 13] 8C3F607B-5FA9-3C80-AC97-66F2BBCF2F85 0x000000018cc5a000 /usr/lib/system/libmacho.dylib 
[ 14] 9A7AD27E-0ECF-3D69-AD56-4C41E0AC19DE 0x000000018c1f3000 /usr/lib/system/libquarantine.dylib 
[ 15] DE28FFB5-D896-38E2-B585-E9847A7918C3 0x000000018ccba000 /usr/lib/system/libremovefile.dylib 
[ 16] 092396DB-281D-3FA0-9E50-43F0B11AA972 0x0000000185b0e000 /usr/lib/system/libsystem_asl.dylib 
[ 17] 41DD58AD-E581-3BC7-A2C4-314046BA471F 0x000000018014c000 /usr/lib/system/libsystem_blocks.dylib 
[ 18] AB8A0A0B-9A1E-3284-9243-5A0334468AE5 0x00000001802d3000 /usr/lib/system/libsystem_c.dylib 
[ 19] 15F74475-A05D-333E-85B8-8BCFF0170FAC 0x000000018ccb2000 /usr/lib/system/libsystem_collections.dylib 
[ 20] 136CD597-BE32-3806-8C7E-B144FCC956C6 0x000000018b4ce000 /usr/lib/system/libsystem_configuration.dylib 
[ 21] 9176BF94-EF84-3C52-93FD-7014FD23E850 0x000000018a54e000 /usr/lib/system/libsystem_containermanager.dylib 
[ 22] 9DF87415-44A0-31F3-BC0B-BFBC6D5BFC2B 0x000000018c8f9000 /usr/lib/system/libsystem_coreservices.dylib 
[ 23] A7805817-0D12-3663-8B9B-EE13AA69A1C4 0x00000001837e2000 /usr/lib/system/libsystem_darwin.dylib 
[ 24] 458D27F4-5C6F-312C-A552-437FE7B0F705 0x00000002217ba000 /usr/lib/system/libsystem_darwindirectory.dylib 
[ 25] 4FCBB463-F915-38A4-AB32-159AF74479E9 0x000000018ccbe000 /usr/lib/system/libsystem_dnssd.dylib 
[ 26] E361D92B-A13D-3278-A67E-059BCC1A1942 0x00000001802d0000 /usr/lib/system/libsystem_featureflags.dylib 
[ 27] 0B3340BE-7CBE-356F-9ACE-7A062DC0554E 0x0000000180471000 /usr/lib/system/libsystem_info.dylib 
[ 28] DBAEE968-94BC-390A-9277-2F0D6696F804 0x000000018cc1f000 /usr/lib/system/libsystem_m.dylib 
[ 29] 690A8B04-8E64-3332-B5A5-56A3D5C1C43F 0x0000000180252000 /usr/lib/system/libsystem_malloc.dylib 
[ 30] DB1022B7-FDD1-3045-A287-3149C1B3D7DA 0x0000000185a7e000 /usr/lib/system/libsystem_networkextension.dylib 
[ 31] 60ECA172-D5DA-3743-AE08-2E397A8C7754 0x0000000183c55000 /usr/lib/system/libsystem_notify.dylib 
[ 32] 5F138857-5FE4-30BE-8B63-FDD645DF23A1 0x000000018b4d3000 /usr/lib/system/libsystem_sandbox.dylib 
[ 33] C2EA7F93-A263-39A2-BCDC-F26DCE9DFA32 0x000000018ccb7000 /usr/lib/system/libsystem_secinit.dylib 
[ 34] CA94FC21-BC40-3B43-B65D-B87ECE9E1D48 0x00000001803fc000 /usr/lib/system/libsystem_kernel.dylib 
[ 35] E1840DAA-9C10-393D-A3A3-94DDE6FE0011 0x000000018046a000 /usr/lib/system/libsystem_platform.dylib 
[ 36] A7D94C96-7B1F-3229-9BEA-048D037C3292 0x0000000180437000 /usr/lib/system/libsystem_pthread.dylib 
[ 37] 16E4B671-D97C-3908-9127-307A83DF3DCD 0x00000001874e3000 /usr/lib/system/libsystem_symptoms.dylib 
[ 38] 1F45FD2F-3ABC-3AB3-9A6F-CC9C727ACBDA 0x0000000180198000 /usr/lib/system/libsystem_trace.dylib 
[ 39] 0C443A31-8851-30E2-9FAD-B4C7633F5135 0x000000018cc90000 /usr/lib/system/libunwind.dylib 
[ 40] 272DC84F-1871-3ADD-8126-2155D6EFF7AB 0x0000000180151000 /usr/lib/system/libxpc.dylib 
[ 41] 9BAB9556-7A2A-30A8-ACDE-010BA8E2367D 0x0000000180068000 /usr/lib/libobjc.A.dylib 
[ 42] B4164258-BFB1-3871-A2E2-528C373AEF77 0x000000018cc9c000 /usr/lib/liboah.dylib 

It looks like the stack unwind is failing because we don't have symbols for libcoreclr.dylib, and it is not even listed on the module list, am I missing something like matching binaries or symbols?

agocke commented 8 months ago

Feel free to reach out to @hoyosjs for instructions on debugging on Mac

hoyosjs commented 8 months ago

Was this done on emulation? I see rosetta loaded and we don't support dumps using emulation. However, the dump and runtime in it are arm64, so I am confused. And how was this dump collected? Overall, the dump seems to be in a pretty bad state.

cshung commented 8 months ago

@hoyosjs confirmed with me that the second dump is unfortunately unusable. We might be able to get some info from the 1st dump though. I will work with him on that.

Regardless, can we run the instrumented bits? It is quite likely that I will need to know why a bad frozen segment get into the heap to begin with, and that cannot be solved by the dump because the crime scene is long gone for that and logging will be the preferred method for debugging.

jaredpar commented 8 months ago

@cshung

Regardless, can we run the instrumented bits?

Sure thing. Was hoping the dump file would be enough. If it's not though I can queue up the instrumented GC.

Pipeline Run

cshung commented 8 months ago

Pipeline Run

Thank you! It looks like the run is completed, but the only failure is "Validate Generated Syntax Files" failing with a "The system cannot find the path specified." on Windows?

jaredpar commented 8 months ago

@cshung Yes. I've re-run it a few times and it's passing every time. Worried whatever timing issues exists is essentially undone by adding printf into the code. Trying a few more runs.

cshung commented 8 months ago

Is there some way we can observe that the new GC is actually in use in the pipeline? I am going through the log but I can't see any of the logging statement, or did we just skip the log if the tests pass?

jaredpar commented 8 months ago

Is there some way we can observe that the new GC is actually in use in the pipeline?

I don't know ... can we? 😄

I am going through the log but I can't see any of the logging statement, or did we just skip the log if the tests pass?

The log is not easy to find when tests pass. Have to start digging through the Helix API directly to find it. J

jaredpar commented 8 months ago

Tracked down the console log and it does seem like something is going sideways here. Our mac logic which was working one week ago is now no longer hitting. Yay infra. Digging into it.

cshung commented 8 months ago

Just a link for easy access. https://github.com/dotnet/roslyn/pull/71724

mangod9 commented 5 months ago

can this be closed now since this hasn't reproed in a while?

jaredpar commented 5 months ago

Yeah. Can re-open if we see it again.

jjonescz commented 5 months ago

I think this happens regularly in roslyn-CI main runs - all these red circles are failing macOS legs:

image

mangod9 commented 5 months ago

@cshung, do you still have the private with logging which you had built to diagnose these?

cshung commented 5 months ago

@cshung, do you still have the private with logging which you had built to diagnose these?

Yes, the instrumentation branch is still there

mangod9 commented 3 months ago

Is there any update on the investigation here? Have we been able to run with instrumentation enabled?

jjonescz commented 3 months ago

This actually stopped happening when roslyn was updated from .NET 8 to .NET 9 SDK in https://github.com/dotnet/roslyn/pull/73408.