Random segmentation fault in managed code on 32-bit ARM linux with dotnet 8.0

Description

I'm encountering random segmentation faults (and sometimes AccessViolationException and NullReferenceException) when running a dotnet 8.0 console app on 32-bit ARM linux. It seems to mainly occur for me when it attempts to connect to a SignalR hub as a client.

Reproduction Steps

I have pushed a small repro app pair here.

Note that in addition to the crashing console app, it also contains a trivially simple SignalR webapp that I've deployed to azure when reproducing. (Note the "[CHANGE THIS TO YOUR WEBAPP URL]" line in the console app)

The console app code just tries to make a SignalR connection over websockets and send/receive a few MessagePack messages before exiting.

When running the console app repeatedly, it has a random chance of encountering the issue, after something like 10 to 100 attempts. (a helper bash loop .sh is included)

Expected behavior

Can run repeatedly without crashing.

Actual behavior

Approximately 5% of the time it fails with simply Segmentation fault.

I have uploaded a core crash dump file of one of the segfault occurrences here

On some occasions it randomly has a NullReferenceException, with additional stack trace info:

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.GetStateMachineBox[TStateMachine](TStateMachine& stateMachine, Task`1& taskField)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.HandshakeAsync(ConnectionState startingConnectionState, Int32 protocolVersion, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncCore(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncCore(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsyncInner(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.StartAsync(CancellationToken cancellationToken)
   at Program.<Main>$(String[] args)
   at Program.<Main>(String[] args)

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AwaitUnsafeOnCompleted[TAwaiter](TAwaiter& awaiter, IAsyncStateMachineBox box)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
Fatal error. Internal CLR error. (0x80131506)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Task+<>c.<ThrowAsync>b__128_1(System.Object)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

Regression?

It seems to work fine (even with thousands of attempts) when changing the csproj back to dotnet 6.0 instead of 8.0.

(I can't test dotnet 7.0 on this ARM device due to the higher glibc requirement 7.0 has)

Known Workarounds

Downgrading the csproj to dotnet 6.0

Configuration

Custom linux OS running on 32-bit ARM linux IoT device.

Problem seems specific to 32-bit ARM. Ran repeatedly on a different custom ARM64 device with no issues.

Other information

On some rare occasions with very similar code (from my team's proprietary app), it encountered this AccessViolationException, which I suspect is stemming from the same cause:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ClearStateUponCompletion()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<Int32>, Int32)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Int32)
   at System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ThreadPool+<>c.<.cctor>b__52_0(System.Object)
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SignalCompletion()
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Int32)
   at System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ThreadPool+<>c.<.cctor>b__52_0(System.Object)
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1[[System.Boolean, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SignalCompletion()
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1[[System.Boolean, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Boolean)
   at System.Net.Sockets.SocketAsyncEventArgs.OnCompletedInternal()
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32, System.Net.Sockets.SocketFlags)
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError)
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, System.Memory`1<Byte>, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError)
   at System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ProcessAsyncOperation(System.__Canon)
   at System.Net.Sockets.SocketAsyncContext+ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Is it possible to check if your Linux OS flavor uses 64bit time_t? I am aware that couple of distributions moved to 64bit to avoid the year 2038 problem, which creates a binary break for ARM32 platform. This announcement may be relevant https://github.com/dotnet/core/discussions/9285.

I have seen our tests suite occasionally crash on ARM32 Debian 12 (on main/9.0), possibly the same issue (but we disabled runs on that specific platform due to https://github.com/dotnet/runtime/issues/101444#issuecomment-2116482502 so I did not investigate further).

Looking at the result of a printf("size of time_t is: %d\n", sizeof(time_t)); check using our platform-specific gcc and running it on our hardware, it says 4 bytes, so it seems it's 32-bit time_t.

it says 4 bytes, so it seems it's 32-bit time_t.

Unfortunately I am not sure that this test necessarily works. It still depends on the _TIME_BITS build option and the system would somehow have to set it globally and unconditionally (or for the OpenSSL build, specifically).

Is there something I can try to run to more conclusively answer the time_t question?

I think this may be the same issue that I have been tracking down on our ARM32 (nxp imx6q) embedded platform running Linux.

The fault can be reproduced using an unmodified dotnet 8 webapi sample program:

dotnet new webapi -n crash
dotnet publish --self-contained -c Release -f net8.0 -r linux-arm crash

For me the segmentation fault only seems to occur on the first web request, and is much more likely to happen with multiple simultaneous requests.

I've used this script running on the device to automate the process:

#!/bin/bash
PROGRAM=crash
fail=0
RUNS=100

for run in $(seq 1 $RUNS); do
  echo "Run $run ($fail failures)"
  ./$PROGRAM > /dev/null &
  sleep 5
  curl -k --parallel --parallel-immediate --parallel-max 50  http://localhost:5000/weatherforecast?[1-16] > /dev/null
  sleep 5

  PID=$(pidof $PROGRAM)
  if [ "$PID" == "" ]; then
    echo "CRASHED!"
        ((fail++))
  else
        kill $PID
        sleep 2
  fi
done

echo "Program crashes $fail / $RUNS"

With a single request, the failure rate is only about 3%, but goes up to about 25% with 4 or more requests.

.net7 shows the same fault, but .net6 works without any problems. ARM64 and Ubuntu in WSL did not show the problem.

The kernel (5.4.147) is compiled with CONFIG_64BIT_TIME, but glibc is only version 2.31 I can't see any indication that _TIME_BITS is being defined anywhere, so I think it is just 32 bit.

I can collect a coredump if that will help.

Sometimes I get the AccessViolationException instead of a segmentation fault:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Runtime.CompilerServices.CastHelpers.IsInstanceOfClass(Void*, System.Object)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder+<BindModelAsync>d__8.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder+<BindModelAsync>d__8, Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<BindModelAsync>d__8 ByRef)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1[[Microsoft.AspNetCore.Mvc.ModelBinding.ModelBindingResult, Microsoft.AspNetCore.Mvc.Abstractions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].Start[[Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder+<BindModelAsync>d__8, Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<BindModelAsync>d__8 ByRef)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(Microsoft.AspNetCore.Mvc.ActionContext, Microsoft.AspNetCore.Mvc.ModelBinding.IModelBinder, Microsoft.AspNetCore.Mvc.ModelBinding.IValueProvider, Microsoft.AspNetCore.Mvc.Abstractions.ParameterDescriptor, Microsoft.AspNetCore.Mvc.ModelBinding.ModelMetadata, System.Object, System.Object)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider+<>c__DisplayClass0_0+<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider+<>c__DisplayClass0_0+<<CreateBinderDelegate>g__Bind|0>d, Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<<CreateBinderDelegate>g__Bind|0>d ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider+<>c__DisplayClass0_0+<<CreateBinderDelegate>g__Bind|0>d, Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<<CreateBinderDelegate>g__Bind|0>d ByRef)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider+<>c__DisplayClass0_0.<CreateBinderDelegate>g__Bind|0(Microsoft.AspNetCore.Mvc.ControllerContext, System.Object, System.Collections.Generic.Dictionary`2<System.String,System.Object>)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.BindArgumentsAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State ByRef, Scope ByRef, System.Object ByRef, Boolean ByRef)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State ByRef, Scope ByRef, System.Object ByRef, Boolean ByRef)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Mvc.Routing.ControllerRequestDelegateFactory+<>c__DisplayClass12_0.<CreateRequestDelegate>b__0(Microsoft.AspNetCore.Http.HttpContext)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.SetRoutingAndContinue(Microsoft.AspNetCore.Http.HttpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<<Invoke>g__AwaitMatcher|10_0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<<Invoke>g__AwaitMatcher|10_0>d, Microsoft.AspNetCore.Routing, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<<Invoke>g__AwaitMatcher|10_0>d, Microsoft.AspNetCore.Routing, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<<Invoke>g__AwaitMatcher|10_0>d, Microsoft.AspNetCore.Routing, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecuteFromThreadPool(System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
Aborted

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetException(System.Exception, System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult> ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetException(System.Exception)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__8[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__8[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__8[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__8[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>, System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection+<ProcessRequestsAsync>d__12`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection+<ProcessRequestsAsync>d__12`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection+<ProcessRequestsAsync>d__12`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection+<ProcessRequestsAsync>d__12`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>, System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequestsAsync>d__237`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequestsAsync>d__237`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequestsAsync>d__237`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequestsAsync>d__237`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>, System.Threading.Tasks.VoidTaskResult)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__238`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__238`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__238`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__238`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.AspNetCore.Server.Kestrel.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecuteFromThreadPool(System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
Aborted

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Collections.Generic.Dictionary`2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Linq.Expressions.Compiler.VariableStorageKind, System.Linq.Expressions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor(Int32, System.Collections.Generic.IEqualityComparer`1<System.__Canon>)
   at System.Collections.Generic.Dictionary`2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Linq.Expressions.Compiler.VariableStorageKind, System.Linq.Expressions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]]..ctor()
   at System.Text.Json.Serialization.Metadata.ReflectionEmitMemberAccessor.CreatePropertyGetter[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Reflection.PropertyInfo)
   at System.Text.Json.Serialization.Metadata.ReflectionEmitCachingMemberAccessor+<>c__12`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<CreatePropertyGetter>b__12_0(System.ValueTuple`3<System.String,System.Type,System.Reflection.MemberInfo>)
   at System.Text.Json.Serialization.Metadata.ReflectionEmitCachingMemberAccessor+Cache`1+<>c__6`1[[System.ValueTuple`3[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<GetOrAdd>b__6_0(System.ValueTuple`3<System.__Canon,System.__Canon,System.__Canon>, System.Func`2<System.ValueTuple`3<System.__Canon,System.__Canon,System.__Canon>,System.__Canon>)
   at System.Collections.Concurrent.ConcurrentDictionary`2[[System.ValueTuple`3[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetOrAdd[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.ValueTuple`3<System.__Canon,System.__Canon,System.__Canon>, System.Func`3<System.ValueTuple`3<System.__Canon,System.__Canon,System.__Canon>,System.__Canon,System.__Canon>, System.__Canon)
   at System.Text.Json.Serialization.Metadata.ReflectionEmitCachingMemberAccessor+Cache`1[[System.ValueTuple`3[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetOrAdd[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.ValueTuple`3<System.__Canon,System.__Canon,System.__Canon>, System.Func`2<System.ValueTuple`3<System.__Canon,System.__Canon,System.__Canon>,System.__Canon>)
   at System.Text.Json.Serialization.Metadata.ReflectionEmitCachingMemberAccessor.CreatePropertyGetter[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Reflection.PropertyInfo)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.DeterminePropertyAccessors[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1<System.__Canon>, System.Reflection.MemberInfo, Boolean)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].DetermineReflectionPropertyAccessors(System.Reflection.MemberInfo, Boolean)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.PopulatePropertyInfo(System.Text.Json.Serialization.Metadata.JsonPropertyInfo, System.Reflection.MemberInfo, System.Text.Json.Serialization.JsonConverter, System.Nullable`1<System.Text.Json.Serialization.JsonIgnoreCondition>, Boolean, Boolean)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.CreatePropertyInfo(System.Text.Json.Serialization.Metadata.JsonTypeInfo, System.Type, System.Reflection.MemberInfo, System.Text.Json.JsonSerializerOptions, Boolean, Boolean)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.AddMembersDeclaredBySuperType(System.Text.Json.Serialization.Metadata.JsonTypeInfo, System.Type, Boolean, PropertyHierarchyResolutionState ByRef)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.PopulateProperties(System.Text.Json.Serialization.Metadata.JsonTypeInfo)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.CreateTypeInfoCore(System.Type, System.Text.Json.Serialization.JsonConverter, System.Text.Json.JsonSerializerOptions)
   at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetTypeInfo(System.Type, System.Text.Json.JsonSerializerOptions)
   at System.Text.Json.JsonSerializerOptions.GetTypeInfoNoCaching(System.Type)
   at System.Text.Json.JsonSerializerOptions+CachingContext.CreateCacheEntry(System.Type, CachingContext)
   at System.Collections.Concurrent.ConcurrentDictionary`2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetOrAdd[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon, System.Func`3<System.__Canon,System.__Canon,System.__Canon>, System.__Canon)
   at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(System.Type, Boolean, System.Nullable`1<Boolean>, Boolean, Boolean)
   at System.Text.Json.JsonSerializerOptions.GetTypeInfo(System.Type)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter+<WriteResponseBodyAsync>d__5.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter+<WriteResponseBodyAsync>d__5, Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<WriteResponseBodyAsync>d__5 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter+<WriteResponseBodyAsync>d__5, Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<WriteResponseBodyAsync>d__5 ByRef)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(Microsoft.AspNetCore.Mvc.Formatters.OutputFormatterWriteContext, System.Text.Encoding)
   at Microsoft.AspNetCore.Mvc.Formatters.TextOutputFormatter.WriteAsync(Microsoft.AspNetCore.Mvc.Formatters.OutputFormatterWriteContext)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor.ExecuteAsyncCore(Microsoft.AspNetCore.Mvc.ActionContext, Microsoft.AspNetCore.Mvc.ObjectResult, System.Type, System.Object)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor.ExecuteAsync(Microsoft.AspNetCore.Mvc.ActionContext, Microsoft.AspNetCore.Mvc.ObjectResult)
   at Microsoft.AspNetCore.Mvc.ObjectResult.ExecuteResultAsync(Microsoft.AspNetCore.Mvc.ActionContext)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultAsync(Microsoft.AspNetCore.Mvc.IActionResult)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](State ByRef, Scope ByRef, System.Object ByRef, Boolean ByRef)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeNextResultFilterAsync[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](State ByRef, Scope ByRef, System.Object ByRef, Boolean ByRef)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State ByRef, Scope ByRef, System.Object ByRef, Boolean ByRef)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d, Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<<InvokeAsync>g__Logged|17_1>d ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d, Microsoft.AspNetCore.Mvc.Core, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<<InvokeAsync>g__Logged|17_1>d ByRef)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Mvc.Routing.ControllerRequestDelegateFactory+<>c__DisplayClass12_0.<CreateRequestDelegate>b__0(Microsoft.AspNetCore.Http.HttpContext)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.SetRoutingAndContinue(Microsoft.AspNetCore.Http.HttpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<<Invoke>g__AwaitMatcher|10_0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<<Invoke>g__AwaitMatcher|10_0>d, Microsoft.AspNetCore.Routing, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<<Invoke>g__AwaitMatcher|10_0>d, Microsoft.AspNetCore.Routing, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware+<<Invoke>g__AwaitMatcher|10_0>d, Microsoft.AspNetCore.Routing, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]].ExecuteFromThreadPool(System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
Aborted

This looks like a GC hole.

Tagging subscribers to this area: @mangod9 See info in area-owners.md if you want to be subscribed.

Hey @maf1024 @cw-ametek would you be able to provide a dump of the failure? If this consistently repros on certain platform might be worth trying with DOTNET__HeapVerify=1 which might help diagnose heap corruption.

@mangod9 Buried in my original wall of text was a link to one sample of a crash dump (hosted on my free MSDN credits 😊)

I'll try to attempt with HeapVerify soon and see if it makes the failures more consistent. Would it also possibly yield a more useful crash dump? (I'm unfamiliar with that option but it sounds like it does more exhaustive policing of memory)

@mangod9 DOTNET__HeapVerify=1 doesn't seem to change the randomness of failures on my platform. Here's a batch of dumps with that setting enabled.

Ok, we will investigate. However, just realized that I had a typo in the env. var DOTNET_HeapVerify=1 (single underscore not double :( )

The dumps are not easily diagnosable since this is on a custom distro. Assume you are not able to repro on standard Debian/Ubuntu/Alpine ?

I don't have any other 32-bit ARM hardware to even test it out on.

I'm open to suggestions for virtualized approaches to repro in a more standardized way.

Also for what it's worth, DOTNET_HeapVerify=1 when set correctly doesn't seem to change the randomness of the failures (and I did see a noticeable performance drop, indicating I set it correctly)

We noticed the same issue while porting our .NET applications from Mono to .NET 7/8. Since we could not find a quick solution, we stayed on .NET 6 first, as .NET 6 does not show the same crashes.

Note, our applications run on NXP iMX6 dual/quad cores (Cortex-A9 ARM32) with a custom Yocto-based Linux (kernel 5.15).

We initially observed random seg faults while running "dotnet test" (we run unit tests on our embedded target hardware), but were able to reproduce those seg faults also by running PowerShell Core, for example, a simple command like

while true; do dotnet .pwsh/pwsh.dll -c ls; done

seg faults within a few minutes. Even simple applications like dotnet --info crash, although it happens less often.

But the repo sample posted by @cw-ametek (https://github.com/dotnet/runtime/issues/102396#issuecomment-2129641835) might be easier to debug. We verified it on our hw/os combination; it crashes almost immediately.

We tried different hardware and custom operating system versions in the past. For example, a Raspberry PI 3 with a Ubuntu 32.04 Server Edition, 32bit does not show these crashes. Also other hardware based on Cortex-A15 does not seg fault. Is this related to Cortex-A9 only? We are not sure yet.

Disabling Ready2Run, Tier Compilation, and using different GC settings (server, without concurrent, background GC, etc.) does not change anything. But, reducing the number of CPU cores helps. For example, on a dual-core system the crashes don't happen that often, also, when binding the process via taskset to a single CPU core reduces the probability of these seg faults to almost zero.

That said, would it help if we provide crash dumps? We are also on a custom Linux distro, but we can provide the necessary symbols for debugging those dumps, if needed.

Can we do anything else? As a company we have high interest in getting that fixed.

I'm using dotnet on ARM32 IoT devices in thousands, dotnet process uptime in months, without problems. My CPU is Allwinner H3, 4x ARM A7 on custom Yocto Linux. I have also monitoring of the nodes, so I see if there is process restart or any problem. I migrated from mono to dotnet 5.0 and now I'm using latest 8.0 and kernel 5.15.35. My app is not small, about 1M of lines of code + external libraries.

That said, would it help if we provide crash dumps?

Could you please share stacktrace of the crash to start with?

Sure!

Here are a few managed-only stack traces that we recorded in the past (various applications).

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Collections.Generic.Dictionary`2[[Interop+Error, System.Net.Sockets, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Sockets.SocketError, System.Net.Primitives, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].TryInsert(Error, System.Net.Sockets.SocketError, System.Collections.Generic.InsertionBehavior)
   at System.Net.Sockets.SocketErrorPal.GetSocketErrorForNativeError(Error)
   at System.Net.Sockets.SafeSocketHandle.DoCloseHandle(Boolean)
   at System.Net.Sockets.SafeSocketHandle.CloseHandle(Boolean, Boolean)
   at System.Net.Sockets.SafeSocketHandle.CloseAsIs(Boolean)
   at System.Net.Sockets.Socket.Dispose(Boolean)
   at System.Net.Sockets.Socket.Dispose()
   at NetMQ.Core.Utils.Signaler..ctor()
   at NetMQ.Core.Mailbox..ctor(System.String)
   at NetMQ.Core.Ctx..ctor()
   at NetMQ.NetMQConfig.get_Context()
   at NetMQ.NetMQSocket..ctor(NetMQ.ZmqSocketType, System.String, DefaultAction)
   at NetMQ.Sockets.PairSocket..ctor(System.String)
   at NetMQ.Sockets.PairSocket.CreateSocketPair(NetMQ.Sockets.PairSocket ByRef, NetMQ.Sockets.PairSocket ByRef, System.Action`1<NetMQ.Sockets.PairSocket>, System.Action`1<NetMQ.Sockets.PairSocket>)
   at NetMQ.NetMQQueue`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(Int32)
   at Darwin.Shared.Monitoring.Transport.MonitoringPublisher..ctor(Int32, Int32, System.TimeSpan, System.TimeSpan, PostSharp.Aspects.ConstructorDepth)
   at Darwin.Shared.Monitoring.Transport.MonitoringPublisher..ctor(Int32, PostSharp.Aspects.ConstructorDepth)
   at Darwin.Shared.Monitoring.Transport.MonitoringPublisher..ctor(Int32)
   at Darwin.Shared.Monitoring.MonitoringServer+<StartAsync>d__39.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at Darwin.Shared.Monitoring.MonitoringServer.StartAsync(Boolean, Boolean)
   at Darwin.Fabric.Systeme.Initializers.SystemMonitoringServerInitializer+<_StartServer>d__3.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at Darwin.Fabric.Systeme.Initializers.SystemMonitoringServerInitializer._StartServer(Darwin.Fabric.Runtime.IRuntimeData, Darwin.Shared.Monitoring.IMonitoringServer)
   at Darwin.Fabric.Systeme.Initializers.SystemMonitoringServerInitializer+<>c__DisplayClass2_0+<<_SetupCounters>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at Darwin.Fabric.Systeme.Initializers.SystemMonitoringServerInitializer+<>c__DisplayClass2_0.<_SetupCounters>b__0()
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
   at System.Threading.Tasks.Task+<>c.<.cctor>b__281_0(System.Object)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Runtime.CompilerServices.CastHelpers.IsInstanceOfInterface(Void*, System.Object)
   at System.Linq.Enumerable.ToArray[[Darwin.Fabric.PluginId, Darwin.Fabric, Version=6.1.0.0, Culture=neutral, PublicKeyToken=ab309ed73cde3b9e]](System.Collections.Generic.IEnumerable`1<Darwin.Fabric.PluginId>)
   at System.Linq.Enumerable.ToArray[[Darwin.Fabric.PluginId, Darwin.Fabric, Version=6.1.0.0, Culture=neutral, PublicKeyToken=ab309ed73cde3b9e]](System.Collections.Generic.IEnumerable`1<Darwin.Fabric.PluginId>)
   at System.Linq.Enumerable.ToArray[[Darwin.Fabric.PluginId, Darwin.Fabric, Version=6.1.0.0, Culture=neutral, PublicKeyToken=ab309ed73cde3b9e]](System.Collections.Generic.IEnumerable`1<Darwin.Fabric.PluginId>)
   at System.Linq.Enumerable.ToArray[[Darwin.Fabric.PluginId, Darwin.Fabric, Version=6.1.0.0, Culture=neutral, PublicKeyToken=ab309ed73cde3b9e]](System.Collections.Generic.IEnumerable`1<Darwin.Fabric.PluginId>)
   at System.Linq.Enumerable.ToArray[[Darwin.Fabric.PluginId, Darwin.Fabric, Version=6.1.0.0, Culture=neutral, PublicKeyToken=ab309ed73cde3b9e]](System.Collections.Generic.IEnumerable`1<Darwin.Fabric.PluginId>)
   at System.Linq.Enumerable.ToArray[[Darwin.Fabric.PluginId, Darwin.Fabric, Version=6.1.0.0, Culture=neutral, PublicKeyToken=ab309ed73cde3b9e]](System.Collections
...

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AddIfNotPresent(System.__Canon, Int32 ByRef)
   at System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Add(System.__Canon)
   at Microsoft.DotNet.Cli.Telemetry.TelemetryFilter.get_ParseResultLogRules()
   at Microsoft.DotNet.Cli.Telemetry.TelemetryFilter.get_ParseResultLogRules()
   at Microsoft.DotNet.Cli.Program.ProcessArgs(System.String[], System.TimeSpan, Microsoft.DotNet.Cli.Telemetry.ITelemetry)
   at Microsoft.DotNet.Cli.Program.Main(System.String[])

Process terminated. An exception was not handled in an AsyncLocal<T> notification callback.
   at System.Environment.FailFast(System.String, System.Exception)
   at System.Threading.ExecutionContext.OnValuesChanged(System.Threading.ExecutionContext, System.Threading.ExecutionContext)
   at System.Threading.ExecutionContext.RestoreChangedContextToThread(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ExecutionContext)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
   at System.Threading.Thread+StartHelper.RunWorker()
   at System.Threading.Thread+StartHelper.Run()
   at System.Threading.Thread.StartCallback()
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Threading.ExecutionContext.OnValuesChanged(ExecutionContext previousExecutionCtx, ExecutionContext nextExecutionCtx)

We see very erratic exception patterns. Looks like a kind of memory corruption to me.

Right now, a colleague is trying to get managed+native stack traces via lldb+sos from one of the crash dumps, but we are facing some problems. The thread causing the crash (seg fault) does not show any backtraces. Maybe the dump is corrupted, too. We are working on it and come back to you.

Sidenote: I have to hurry into vacation. My colleagues will take over.

dotnet / runtime