search

HangfireIO / Hangfire

An easy way to perform background job processing in .NET and .NET Core applications. No Windows Service or separate process required
https://www.hangfire.io
Other
9.4k stars 1.7k forks source link

Jobs becoming stuck in processing state #2413

Open selmendorfFrontline opened 5 months ago

selmendorfFrontline commented 5 months ago

We are seeing an issue where jobs are becoming stuck randomly in the processing state. We are not seeing any exceptions, nor have any long-running query or process that would cause the job to remain in the processed state for any considerable length of time, yet the job will sit in the Processing state until manually re-queued.

This issue brings Hangfire to a halt; once all workers have a job in this state, no more work can be done.

We are using Hangfire Pro and Redis. We updated to the latest Hangfire libraries and are running .NET Framework 4.6.1.

Upon re-queueing the job is completes without an issue. We run tens of thousands of jobs daily across a set of 6 servers.

We did want to set the VisibilityTimeout, but we found that it only applies to when the service is reporting an unhealthy state.

What is the debug path to see why Hangfire appears to not get the update for these? What options do we have?

odinserj commented 5 months ago

Can you run the stdump utility when everything is stuck and post the results here? It will dump all the managed stack traces and we'll see the actual methods.

selmendorfFrontline commented 4 months ago 


Found CLR Version: 4.8.4645.0
Found Target Architecture: X64

Following ThreadPool information found:
  CPU Utilization by ThreadPool: 14%
  Worker threads: Active 0, Idle 2, Min 2, Max 32767
  Completion ports: Free 4, Total 4, Min 2, Max 1000
  Type: 

Following AppDomains found:

AppDomain #1
  Address:       28*******408
  Name:          DefaultDomain
  Configuration: w3wp.exe.config
  Directory:     file:///c:/windows/system32/inetsrv/

AppDomain #2
  Address:       28***********456
  Name:          /LM/W3SVC/1/ROOT-1-133619945946975617
  Configuration: web.config
  Directory:     file:///C:/Applications/*************/0.1.0.6816/

Following threads found:

Thread #1
  OS Thread ID:      4712
  AppDomain Address: 2805977711408
  State:             164384

  No managed stack trace found for this thread.

Thread #2
  OS Thread ID:      5148
  AppDomain Address: 2805977711408
  State:             176672
  Role:              Finalizer

  No managed stack trace found for this thread.

Thread #3
  OS Thread ID:      5184
  AppDomain Address: 2805977711408
  State:             131616

  No managed stack trace found for this thread.

Thread #4
  OS Thread ID:      5676
  AppDomain Address: 2805977711408
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame] (System.Threading.Thread.SleepInternal) at mscorlib.dll
   - mscorlib.dll!Unknown at mscorlib.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #5
  OS Thread ID:      5968
  AppDomain Address: 2805977711408
  State:             16949792

  No managed stack trace found for this thread.

Thread #6
  OS Thread ID:      6072
  AppDomain Address: 2805977711408
  State:             135712

  No managed stack trace found for this thread.

Thread #11
  OS Thread ID:      5964
  AppDomain Address: 2805977711408
  State:             16908832

  No managed stack trace found for this thread.

Thread #14
  OS Thread ID:      6000
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(Int32, Boolean) at mscorlib.dll
   - XX.ToolBox.Threading.BasicThread.Execute() at XX.ToolBox.dll
   - XX.ToolBox.Threading.BasicThread.ThreadRunner() at XX.ToolBox.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #16
  OS Thread ID:      5000
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Pro.Redis.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.RedisStorage.ExtendLocks() at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #17
  OS Thread ID:      6136
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitMultiple) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], System.TimeSpan, Boolean) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisStorage.ConnectLoop() at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #18
  OS Thread ID:      6140
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - StackExchange.Redis.SocketManager.WriteAllQueues() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_0(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #19
  OS Thread ID:      6132
  AppDomain Address: 2805997469456
  State:             176672

  Managed stack trace:
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   -  at 
   - System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) at System.dll
   - System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) at System.dll
   - StackExchange.Redis.PhysicalConnection.StackExchange.Redis.ISocketCallback.Read() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_2(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #20
  OS Thread ID:      5324
  AppDomain Address: 2805997469456
  State:             176672

  Managed stack trace:
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   -  at 
   - System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) at System.dll
   - System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) at System.dll
   - StackExchange.Redis.PhysicalConnection.StackExchange.Redis.ISocketCallback.Read() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_2(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #21
  OS Thread ID:      4396
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.WaitAllBlockingCore(System.Collections.Generic.List`1<System.Threading.Tasks.Task>, Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.WaitAll(System.Threading.Tasks.Task[], Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Server.BackgroundServerProcess.WaitForDispatchers(Hangfire.Server.BackgroundServerContext, System.Collections.Generic.IReadOnlyList`1<Hangfire.Processing.IBackgroundDispatcher>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundServerProcess.Execute(System.Guid, Hangfire.Processing.BackgroundExecution, System.Threading.CancellationToken, System.Threading.CancellationToken, System.Threading.CancellationToken) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessingServer.RunServer(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #23
  OS Thread ID:      5296
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Pro.Redis.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.RedisStorage.ExtendLocks() at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #24
  OS Thread ID:      5308
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitMultiple) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], System.TimeSpan, Boolean) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisStorage.ConnectLoop() at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #25
  OS Thread ID:      4752
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - StackExchange.Redis.SocketManager.WriteAllQueues() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_0(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #26
  OS Thread ID:      4336
  AppDomain Address: 2805997469456
  State:             176672

  Managed stack trace:
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   -  at 
   - System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) at System.dll
   - System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) at System.dll
   - StackExchange.Redis.PhysicalConnection.StackExchange.Redis.ISocketCallback.Read() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_2(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #27
  OS Thread ID:      768
  AppDomain Address: 2805997469456
  State:             176672

  Managed stack trace:
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   -  at 
   - System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) at System.dll
   - System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) at System.dll
   - StackExchange.Redis.PhysicalConnection.StackExchange.Redis.ISocketCallback.Read() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_2(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #28
  OS Thread ID:      4364
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.WaitAllBlockingCore(System.Collections.Generic.List`1<System.Threading.Tasks.Task>, Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.WaitAll(System.Threading.Tasks.Task[], Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Server.BackgroundServerProcess.WaitForDispatchers(Hangfire.Server.BackgroundServerContext, System.Collections.Generic.IReadOnlyList`1<Hangfire.Processing.IBackgroundDispatcher>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundServerProcess.Execute(System.Guid, Hangfire.Processing.BackgroundExecution, System.Threading.CancellationToken, System.Threading.CancellationToken, System.Threading.CancellationToken) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessingServer.RunServer(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #29
  OS Thread ID:      944
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerHeartbeatProcess.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #30
  OS Thread ID:      5100
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerHeartbeatProcess.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #31
  OS Thread ID:      964
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerWatchdog.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #32
  OS Thread ID:      5444
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerJobCancellationWatcher.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #33
  OS Thread ID:      2288
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Pro.Redis.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.FetchedJobsWatcher.Execute(System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #34
  OS Thread ID:      5080
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.GetResult() at mscorlib.dll
   - [DebuggerU2MCatchHandlerFrame] at 
   - [HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) at mscorlib.dll
   - Hangfire.Processing.TaskExtensions.GetTaskLikeResult(System.Threading.Tasks.Task, System.Object, System.Type) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeOnTaskScheduler(Hangfire.Server.PerformContext, System.Tuple`3<System.Reflection.MethodInfo,System.Object,System.Object[]>, System.Func`2<System.Object,System.Threading.Tasks.Task>) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(Hangfire.Server.PerformContext, System.Object, System.Object[]) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass10_0.<PerformJobWithFilters>b__0() at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(Hangfire.Server.IServerFilter, Hangfire.Server.PerformingContext, System.Func`1<Hangfire.Server.PerformedContext>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, System.Collections.Generic.IEnumerable`1<Hangfire.Server.IServerFilter>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String, Hangfire.BackgroundJob, System.Collections.Generic.IReadOnlyDictionary`2<System.String,System.Object> ByRef) at Hangfire.Core.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #35
  OS Thread ID:      924
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.GetResult() at mscorlib.dll
   - [DebuggerU2MCatchHandlerFrame] at 
   - [HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) at mscorlib.dll
   - Hangfire.Processing.TaskExtensions.GetTaskLikeResult(System.Threading.Tasks.Task, System.Object, System.Type) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeOnTaskScheduler(Hangfire.Server.PerformContext, System.Tuple`3<System.Reflection.MethodInfo,System.Object,System.Object[]>, System.Func`2<System.Object,System.Threading.Tasks.Task>) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(Hangfire.Server.PerformContext, System.Object, System.Object[]) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass10_0.<PerformJobWithFilters>b__0() at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(Hangfire.Server.IServerFilter, Hangfire.Server.PerformingContext, System.Func`1<Hangfire.Server.PerformedContext>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, System.Collections.Generic.IEnumerable`1<Hangfire.Server.IServerFilter>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String, Hangfire.BackgroundJob, System.Collections.Generic.IReadOnlyDictionary`2<System.String,System.Object> ByRef) at Hangfire.Core.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #36
  OS Thread ID:      5068
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.GetResult() at mscorlib.dll
   - [DebuggerU2MCatchHandlerFrame] at 
   - [HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) at mscorlib.dll
   - Hangfire.Processing.TaskExtensions.GetTaskLikeResult(System.Threading.Tasks.Task, System.Object, System.Type) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeOnTaskScheduler(Hangfire.Server.PerformContext, System.Tuple`3<System.Reflection.MethodInfo,System.Object,System.Object[]>, System.Func`2<System.Object,System.Threading.Tasks.Task>) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(Hangfire.Server.PerformContext, System.Object, System.Object[]) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass10_0.<PerformJobWithFilters>b__0() at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(Hangfire.Server.IServerFilter, Hangfire.Server.PerformingContext, System.Func`1<Hangfire.Server.PerformedContext>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, System.Collections.Generic.IEnumerable`1<Hangfire.Server.IServerFilter>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String, Hangfire.BackgroundJob, System.Collections.Generic.IReadOnlyDictionary`2<System.String,System.Object> ByRef) at Hangfire.Core.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #37
  OS Thread ID:      5056
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.DelayedJobScheduler.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #38
  OS Thread ID:      5076
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.RecurringJobScheduler.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #39
  OS Thread ID:      1080
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerWatchdog.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #40
  OS Thread ID:      1204
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerJobCancellationWatcher.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #41
  OS Thread ID:      5332
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Pro.Redis.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.FetchedJobsWatcher.Execute(System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #42
  OS Thread ID:      5260
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #43
  OS Thread ID:      5236
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #44
  OS Thread ID:      5336
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitMultiple) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], System.TimeSpan, Boolean) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.TrySubscriptionBasedFetch(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #45
  OS Thread ID:      5356
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #46
  OS Thread ID:      5564
  AppDomain Address: 2805997469456
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken) at mscorlib.dll```
odinserj commented 4 months ago

Thank you for the stack traces, but unfortunately the list is not full as it's cut off at the end. May I ask you to post the full stack trace? It shows something, but information is not full.

selmendorfFrontline commented 4 months ago 
Stack Trace Dump 0.4.0

Found CLR Version: 4.8.4645.0
Found Target Architecture: X64

Following ThreadPool information found:
  CPU Utilization by ThreadPool: 98%
  Worker threads: Active 0, Idle 3, Min 2, Max 32767
  Completion ports: Free 5, Total 5, Min 2, Max 1000
  Type: 

Following AppDomains found:

AppDomain #1
  Address:       1735865762624
  Name:          DefaultDomain
  Configuration: w3wp.exe.config
  Directory:     file:///c:/windows/system32/inetsrv/

AppDomain #2
  Address:       1745301424912
  Name:          /LM/W3SVC/1/ROOT-1-133620675257075816
  Configuration: web.config
  Directory:     file:///C:/Applications/*************/0.1.0.6816/

Following threads found:

Thread #1
  OS Thread ID:      5924
  AppDomain Address: 1735865762624
  State:             164384

  No managed stack trace found for this thread.

Thread #2
  OS Thread ID:      6484
  AppDomain Address: 1735865762624
  State:             176672
  Role:              Finalizer

  No managed stack trace found for this thread.

Thread #4
  OS Thread ID:      7244
  AppDomain Address: 1735865762624
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame] (System.Threading.Thread.SleepInternal) at mscorlib.dll
   - mscorlib.dll!Unknown at mscorlib.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #5
  OS Thread ID:      6328
  AppDomain Address: 1735865762624
  State:             16949792

  No managed stack trace found for this thread.

Thread #6
  OS Thread ID:      2720
  AppDomain Address: 1735865762624
  State:             135712

  No managed stack trace found for this thread.

Thread #7
  OS Thread ID:      2184
  AppDomain Address: 1735865762624
  State:             131616

  No managed stack trace found for this thread.

Thread #13
  OS Thread ID:      6740
  AppDomain Address: 1735865762624
  State:             16908832

  No managed stack trace found for this thread.

Thread #14
  OS Thread ID:      8168
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(Int32, Boolean) at mscorlib.dll
   - XX.ToolBox.Threading.BasicThread.Execute() at FL.ToolBox.dll
   - XX.ToolBox.Threading.BasicThread.ThreadRunner() at FL.ToolBox.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #17
  OS Thread ID:      6828
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Pro.Redis.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.RedisStorage.ExtendLocks() at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #18
  OS Thread ID:      1728
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitMultiple) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], System.TimeSpan, Boolean) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisStorage.ConnectLoop() at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #19
  OS Thread ID:      7064
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - StackExchange.Redis.SocketManager.WriteAllQueues() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_0(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #20
  OS Thread ID:      5228
  AppDomain Address: 1745301424912
  State:             176672

  Managed stack trace:
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   -  at 
   - System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) at System.dll
   - System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) at System.dll
   - StackExchange.Redis.PhysicalConnection.StackExchange.Redis.ISocketCallback.Read() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_2(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #21
  OS Thread ID:      6984
  AppDomain Address: 1745301424912
  State:             176672

  Managed stack trace:
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   -  at 
   - System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) at System.dll
   - System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) at System.dll
   - StackExchange.Redis.PhysicalConnection.StackExchange.Redis.ISocketCallback.Read() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_2(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #22
  OS Thread ID:      7100
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.WaitAllBlockingCore(System.Collections.Generic.List`1<System.Threading.Tasks.Task>, Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.WaitAll(System.Threading.Tasks.Task[], Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Server.BackgroundServerProcess.WaitForDispatchers(Hangfire.Server.BackgroundServerContext, System.Collections.Generic.IReadOnlyList`1<Hangfire.Processing.IBackgroundDispatcher>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundServerProcess.Execute(System.Guid, Hangfire.Processing.BackgroundExecution, System.Threading.CancellationToken, System.Threading.CancellationToken, System.Threading.CancellationToken) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessingServer.RunServer(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #24
  OS Thread ID:      7060
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Pro.Redis.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.RedisStorage.ExtendLocks() at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #25
  OS Thread ID:      7788
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitMultiple) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], System.TimeSpan, Boolean) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisStorage.ConnectLoop() at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #26
  OS Thread ID:      7908
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - StackExchange.Redis.SocketManager.WriteAllQueues() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_0(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #27
  OS Thread ID:      8076
  AppDomain Address: 1745301424912
  State:             176672

  Managed stack trace:
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   -  at 
   - System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) at System.dll
   - System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) at System.dll
   - StackExchange.Redis.PhysicalConnection.StackExchange.Redis.ISocketCallback.Read() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_2(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #28
  OS Thread ID:      104
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.WaitAllBlockingCore(System.Collections.Generic.List`1<System.Threading.Tasks.Task>, Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.WaitAll(System.Threading.Tasks.Task[], Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Server.BackgroundServerProcess.WaitForDispatchers(Hangfire.Server.BackgroundServerContext, System.Collections.Generic.IReadOnlyList`1<Hangfire.Processing.IBackgroundDispatcher>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundServerProcess.Execute(System.Guid, Hangfire.Processing.BackgroundExecution, System.Threading.CancellationToken, System.Threading.CancellationToken, System.Threading.CancellationToken) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessingServer.RunServer(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #29
  OS Thread ID:      248
  AppDomain Address: 1745301424912
  State:             176672

  Managed stack trace:
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   - [InlinedCallFrame] (System.Net.UnsafeNclNativeMethods+OSSOCK.recv) at System.dll
   -  at 
   - System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) at System.dll
   - System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) at System.dll
   - StackExchange.Redis.PhysicalConnection.StackExchange.Redis.ISocketCallback.Read() at Hangfire.Pro.Redis.dll
   - StackExchange.Redis.SocketManager+<>c.<.cctor>b__29_2(System.Object) at Hangfire.Pro.Redis.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart(System.Object) at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #30
  OS Thread ID:      6776
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerHeartbeatProcess.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #31
  OS Thread ID:      7444
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerHeartbeatProcess.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #32
  OS Thread ID:      7912
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerWatchdog.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #33
  OS Thread ID:      7568
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerJobCancellationWatcher.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #34
  OS Thread ID:      7408
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Pro.Redis.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.FetchedJobsWatcher.Execute(System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #35
  OS Thread ID:      7412
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.GetResult() at mscorlib.dll
   - [DebuggerU2MCatchHandlerFrame] at 
   - [HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) at mscorlib.dll
   - Hangfire.Processing.TaskExtensions.GetTaskLikeResult(System.Threading.Tasks.Task, System.Object, System.Type) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeOnTaskScheduler(Hangfire.Server.PerformContext, System.Tuple`3<System.Reflection.MethodInfo,System.Object,System.Object[]>, System.Func`2<System.Object,System.Threading.Tasks.Task>) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(Hangfire.Server.PerformContext, System.Object, System.Object[]) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass10_0.<PerformJobWithFilters>b__0() at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(Hangfire.Server.IServerFilter, Hangfire.Server.PerformingContext, System.Func`1<Hangfire.Server.PerformedContext>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, System.Collections.Generic.IEnumerable`1<Hangfire.Server.IServerFilter>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String, Hangfire.BackgroundJob, System.Collections.Generic.IReadOnlyDictionary`2<System.String,System.Object> ByRef) at Hangfire.Core.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #36
  OS Thread ID:      7440
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.GetResult() at mscorlib.dll
   - [DebuggerU2MCatchHandlerFrame] at 
   - [HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) at mscorlib.dll
   - Hangfire.Processing.TaskExtensions.GetTaskLikeResult(System.Threading.Tasks.Task, System.Object, System.Type) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeOnTaskScheduler(Hangfire.Server.PerformContext, System.Tuple`3<System.Reflection.MethodInfo,System.Object,System.Object[]>, System.Func`2<System.Object,System.Threading.Tasks.Task>) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(Hangfire.Server.PerformContext, System.Object, System.Object[]) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass10_0.<PerformJobWithFilters>b__0() at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(Hangfire.Server.IServerFilter, Hangfire.Server.PerformingContext, System.Func`1<Hangfire.Server.PerformedContext>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, System.Collections.Generic.IEnumerable`1<Hangfire.Server.IServerFilter>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String, Hangfire.BackgroundJob, System.Collections.Generic.IReadOnlyDictionary`2<System.String,System.Object> ByRef) at Hangfire.Core.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #37
  OS Thread ID:      7436
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at mscorlib.dll
   - System.Runtime.CompilerServices.TaskAwaiter.GetResult() at mscorlib.dll
   - [DebuggerU2MCatchHandlerFrame] at 
   - [HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) at mscorlib.dll
   - System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) at mscorlib.dll
   - Hangfire.Processing.TaskExtensions.GetTaskLikeResult(System.Threading.Tasks.Task, System.Object, System.Type) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeOnTaskScheduler(Hangfire.Server.PerformContext, System.Tuple`3<System.Reflection.MethodInfo,System.Object,System.Object[]>, System.Func`2<System.Object,System.Threading.Tasks.Task>) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(Hangfire.Server.PerformContext, System.Object, System.Object[]) at Hangfire.Core.dll
   - Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass10_0.<PerformJobWithFilters>b__0() at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(Hangfire.Server.IServerFilter, Hangfire.Server.PerformingContext, System.Func`1<Hangfire.Server.PerformedContext>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, System.Collections.Generic.IEnumerable`1<Hangfire.Server.IServerFilter>) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext) at Hangfire.Core.dll
   - Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String, Hangfire.BackgroundJob, System.Collections.Generic.IReadOnlyDictionary`2<System.String,System.Object> ByRef) at Hangfire.Core.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #38
  OS Thread ID:      2256
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.DelayedJobScheduler.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #39
  OS Thread ID:      2748
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerWatchdog.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #40
  OS Thread ID:      4672
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.ServerJobCancellationWatcher.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #41
  OS Thread ID:      1380
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Pro.Redis.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.FetchedJobsWatcher.Execute(System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #42
  OS Thread ID:      4584
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #43
  OS Thread ID:      1068
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #44
  OS Thread ID:      4364
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #45
  OS Thread ID:      2788
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitMultiple) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], System.TimeSpan, Boolean) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.TrySubscriptionBasedFetch(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #46
  OS Thread ID:      3192
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [GCFrame] at 
   - [HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait) at mscorlib.dll
   - System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken) at mscorlib.dll
   - System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken) at mscorlib.dll
   - Hangfire.Pro.Redis.RedisConnection.FetchNextJob(System.String[], System.Threading.CancellationToken) at Hangfire.Pro.Redis.dll
   - Hangfire.Server.Worker.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #47
  OS Thread ID:      7648
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitMultiple) at mscorlib.dll
   - System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean) at mscorlib.dll
   - System.Net.TimerThread.ThreadProc() at System.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #48
  OS Thread ID:      7644
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.RecurringJobScheduler.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #49
  OS Thread ID:      4208
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.DelayedJobScheduler.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #50
  OS Thread ID:      5852
  AppDomain Address: 1745301424912
  State:             33731104

  Managed stack trace:
   - [HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative) at mscorlib.dll
   - System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at mscorlib.dll
   - System.Threading.WaitHandle.WaitOne(System.TimeSpan) at mscorlib.dll
   - Hangfire.Common.CancellationTokenExtentions.Wait(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Common.CancellationTokenExtentions.WaitOrThrow(System.Threading.CancellationToken, System.TimeSpan) at Hangfire.Core.dll
   - Hangfire.Server.RecurringJobScheduler.Execute(Hangfire.Server.BackgroundProcessContext) at Hangfire.Core.dll
   - Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object) at Hangfire.Core.dll
   - Hangfire.Processing.BackgroundDispatcher.DispatchLoop() at Hangfire.Core.dll
   - System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at mscorlib.dll
   - System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at mscorlib.dll
   - System.Threading.ThreadHelper.ThreadStart() at mscorlib.dll
   - [GCFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 
   - [ContextTransitionFrame] at 
   - [DebuggerU2MCatchHandlerFrame] at 

Thread #74
  OS Thread ID:      3480
  AppDomain Address: 1735865762624
  State:             135712

  No managed stack trace found for this thread.

Thread #75
  OS Thread ID:      3384
  AppDomain Address: 1735865762624
  State:             135712

  No managed stack trace found for this thread.

Thread #77
  OS Thread ID:      6720
  AppDomain Address: 1735865762624
  State:             131616

  No managed stack trace found for this thread.

Thread #76
  OS Thread ID:      5412
  AppDomain Address: 1735865762624
  State:             131616

  No managed stack trace found for this thread.

Thread #16
  OS Thread ID:      3696
  AppDomain Address: 1735865762624
  State:             131616

  No managed stack trace found for this thread.

Thread #68
  OS Thread ID:      7112
  AppDomain Address: 1735865762624
  State:             131616

  No managed stack trace found for this thread.

Thread #12
  OS Thread ID:      3188
  AppDomain Address: 1735865762624
  State:             16945696

  No managed stack trace found for this thread.

Thread #64
  OS Thread ID:      6792
  AppDomain Address: 1735865762624
  State:             16945696

  No managed stack trace found for this thread.

Thread #83
  OS Thread ID:      5836
  AppDomain Address: 1735865762624
  State:             16945696

  No managed stack trace found for this thread.

Thread #91
  OS Thread ID:      532
  AppDomain Address: 1735865762624
  State:             16945696

  No managed stack trace found for this thread.

Thread #85
  OS Thread ID:      1880
  AppDomain Address: 1735865762624
  State:             134386208

  No managed stack trace found for this thread.

Thread #70
  OS Thread ID:      8120
  AppDomain Address: 1735865762624
  State:             131616

  No managed stack trace found for this thread.

Thread #92
  OS Thread ID:      4960
  AppDomain Address: 1735865762624
  State:             134386208

  No managed stack trace found for this thread.

Thread #86
  OS Thread ID:      6212
  AppDomain Address: 1735865762624
  State:             16945696

  No managed stack trace found for this thread.

Thread #88
  OS Thread ID:      420
  AppDomain Address: 1735865762624
  State:             16945696

  No managed stack trace found for this thread.

Thread #65
  OS Thread ID:      4448
  AppDomain Address: 1735865762624
  State:             16945696

  No managed stack trace found for this thread.

Thread #52
  OS Thread ID:      7484
  AppDomain Address: 1735865762624
  State:             134386208

  No managed stack trace found for this thread.

Thread #84
  OS Thread ID:      2060
  AppDomain Address: 1735865762624
  State:             134386208

  No managed stack trace found for this thread.

Thread #53
  OS Thread ID:      1748
  AppDomain Address: 1735865762624
  State:             134386208

  No managed stack trace found for this thread.

Above should be the entire file.

odinserj commented 4 months ago

Thanks for the details. I see a lot of Hangfire's workers are free to receive new tasks, and only 3 of them are busy by doing something on a thread pool (so they are probably async methods). What's strange is that CPU Utilization by ThreadPool: 98%, do you see high CPU usage during these problems?

Unfortunately it's really difficult to get information about queued tasks in the Thread Pool, since walking GC heap is required, and this feature is not implemented yet in stdump. However, it's possible to instruct Hangfire to run async methods in its worker threads directly, without scheduling them to the Thread Pool. In this case, we'll have more chances to understand what methods are actually running, since synchronization context is used in this case.

It's possible to enable this feature by providing a null value for the BackgroundJobServerOptions.TaskScheduler option.

.AddHangfireServer(x => x.TaskScheduler = null)