search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.28k stars 4.73k forks source link

Potential Deadlock When Disposing EventListeners Concurrently #96219

Open kataklinger opened 10 months ago

kataklinger commented 10 months ago

Description

We have number of tests that are creating and disposing System.Diagnostics.Tracing.EventListeners. Since upgrading to .NET Core 8 if the tests are executed concurrently, using dotnet test, test runners started hanging for some of our tests. Attaching a debugger reveals a deadlock occurs in the code that is disposing EventListener.

Here's the stack traces of the two threads involved in the deadlock.

Thread1:

    [Deadlocked, double-click or press enter to view all thread stacks]     Annotated Frame
    [Waiting on lock owned by Thread 23448, double-click or press enter to switch to thread]        Annotated Frame
>   System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventSource.SendCommand(System.Diagnostics.Tracing.EventListener listener, System.Diagnostics.Tracing.EventProviderType eventProviderType, int perEventSourceSessionId, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary<string, string> commandArguments) Line 2569   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventSource.OverrideEventProvider.OnControllerCommand(System.Diagnostics.Tracing.ControllerCommand command, System.Collections.Generic.IDictionary<string, string> arguments, int perEventSourceSessionId) Line 2440  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventProvider.HandleEnableNotification(System.Diagnostics.Tracing.EventProvider target, byte* additionalData, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData) Line 59    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventProviderImpl.ProviderCallback(System.Diagnostics.Tracing.EventProvider target, byte* additionalData, int controlCode, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData) Line 1310  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventProvider.Callback(byte* sourceId, int isEnabled, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData, void* callbackContext) Line 70 C#  Symbols loaded.
    [Native to Managed Transition]      Annotated Frame
    System.Private.CoreLib.dll!00007fff206cb4d5()   Unknown No symbols loaded.
    [Managed to Native Transition]      Annotated Frame
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeInternal.Enable(string outputFile, System.Diagnostics.Tracing.EventPipeSerializationFormat format, uint circularBufferSizeInMB, System.Diagnostics.Tracing.EventPipeProviderConfiguration[] providers = {System.Diagnostics.Tracing.EventPipeProviderConfiguration[1]}) Line 135 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.CommitDispatchConfiguration() Line 106   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords) Line 64    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.Dispose() Line 4083 C#  Symbols loaded.
    Prometheus.NetCore.dll!Prometheus.EventCounterAdapter.Dispose() Line 84 C#  Symbols loaded.
    ZapMiddleware.Shared.WebApi.dll!ZapMiddleware.Shared.WebApi.Metrics.EventCounterMetricsListenerHostedService.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = true) Line 18   C#  Symbols loaded.
    Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.ForeachService<Microsoft.Extensions.Hosting.IHostedService>(System.Collections.Generic.IEnumerable<Microsoft.Extensions.Hosting.IHostedService> services = {System.Linq.Enumerable.ReverseIterator<Microsoft.Extensions.Hosting.IHostedService>}, System.Threading.CancellationToken token = IsCancellationRequested = true, bool concurrent = false, bool abortOnFirstException = false, System.Collections.Generic.List<System.Exception> exceptions = Count = 0, System.Func<Microsoft.Extensions.Hosting.IHostedService, System.Threading.CancellationToken, System.Threading.Tasks.Task> operation = {Method = {System.Reflection.RuntimeMethodInfo}}) Unknown No symbols loaded.
    [Resuming Async Method]     Annotated Frame
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<ForeachService>d__18<Microsoft.Extensions.Hosting.IHostedService>>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.Internal.Host+<ForeachService>d__18`1[Microsoft.Extensions.Hosting.IHostedService]].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<ForeachService>d__18<Microsoft.Extensions.Hosting.IHostedService>>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<ForeachService>d__18<System.__Canon>>.MoveNext() Line 346 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+<StopAsync>d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 273    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() Line 109  C#  Symbols loaded.
    [Completed] Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.BackgroundService.StopAsync(System.Threading.CancellationToken cancellationToken)    Unknown Non-user code. Symbol loading disabled by Include/Exclude setting.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.BackgroundService.<StopAsync>d__6>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+<StopAsync>d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.BackgroundService.<StopAsync>d__6>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.BackgroundService.<StopAsync>d__6>.MoveNext() Line 346   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Id = 3651, Status = RanToCompletion, Method = "{null}", Result = "System.Threading.Tasks.Task`1[System.Object]") Line 273 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 399 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.TwoTaskWhenAnyPromise<System.Threading.Tasks.Task>.Invoke(System.Threading.Tasks.Task completingTask) Line 6494  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3474    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetCanceled(System.Threading.CancellationToken tokenToRecord, object cancellationException) Line 3411 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource<object>.TrySetCanceled(System.Threading.CancellationToken cancellationToken) Line 281    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.SetCanceled() Line 235   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.TimerQueueTimer.Fire(bool isThreadPool) Line 686    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.TimerQueue.FireNextTimers() Line 331    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 918 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 102    C#  Symbols loaded.
    [Native to Managed Transition]      Annotated Frame
    kernel32.dll!00007fffe9487034() Unknown No symbols loaded.
    ntdll.dll!00007fffea29d241()    Unknown No symbols loaded.
    [Async Call Stack]      Annotated Frame
    [Async] Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false)   Unknown No symbols loaded.
    [Async] Microsoft.AspNetCore.Mvc.Testing.dll!Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<ZapMiddleware.Webhooks.Api.Startup>.DisposeAsync()  Unknown No symbols loaded.

Thread 2:

    [Deadlocked, double-click or press enter to view all thread stacks]     Annotated Frame
    [Waiting on lock owned by Thread 48596, double-click or press enter to switch to thread]        Annotated Frame
>   System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener = {Prometheus.EventCounterAdapter.Listener}, System.Diagnostics.Tracing.EventCommand command = Update, bool enable = false, System.Diagnostics.Tracing.EventLevel level = LogAlways, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords = None) Line 52    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.DisableEvents(System.Diagnostics.Tracing.EventSource eventSource) Line 4158 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.CallDisableEventsIfNecessary(System.Diagnostics.Tracing.EventDispatcher eventDispatcher = {System.Diagnostics.Tracing.EventDispatcher}, System.Diagnostics.Tracing.EventSource eventSource = {System.Diagnostics.Tracing.NativeRuntimeEventSource}) Line 4318   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.RemoveReferencesToListenerInEventSources(System.Diagnostics.Tracing.EventListener listenerToRemove = {Prometheus.EventCounterAdapter.Listener}) Line 4360   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.Dispose() Line 4069 C#  Symbols loaded.
    Prometheus.NetCore.dll!Prometheus.EventCounterAdapter.Dispose() Line 84 C#  Symbols loaded.
    ZapMiddleware.Shared.WebApi.dll!ZapMiddleware.Shared.WebApi.Metrics.EventCounterMetricsListenerHostedService.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = true) Line 18   C#  Symbols loaded.
    Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.ForeachService<Microsoft.Extensions.Hosting.IHostedService>(System.Collections.Generic.IEnumerable<Microsoft.Extensions.Hosting.IHostedService> services = {System.Linq.Enumerable.ReverseIterator<Microsoft.Extensions.Hosting.IHostedService>}, System.Threading.CancellationToken token = IsCancellationRequested = true, bool concurrent = false, bool abortOnFirstException = false, System.Collections.Generic.List<System.Exception> exceptions = Count = 0, System.Func<Microsoft.Extensions.Hosting.IHostedService, System.Threading.CancellationToken, System.Threading.Tasks.Task> operation = {Method = {System.Reflection.RuntimeMethodInfo}}) Unknown No symbols loaded.
    [Resuming Async Method]     Annotated Frame
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<ForeachService>d__18<Microsoft.Extensions.Hosting.IHostedService>>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.Internal.Host+<ForeachService>d__18`1[Microsoft.Extensions.Hosting.IHostedService]].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<ForeachService>d__18<Microsoft.Extensions.Hosting.IHostedService>>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<ForeachService>d__18<System.__Canon>>.MoveNext() Line 346 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+<StopAsync>d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 273    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() Line 109  C#  Symbols loaded.
    [Completed] Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.BackgroundService.StopAsync(System.Threading.CancellationToken cancellationToken)    Unknown Non-user code. Symbol loading disabled by Include/Exclude setting.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.BackgroundService.<StopAsync>d__6>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+<StopAsync>d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.BackgroundService.<StopAsync>d__6>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.BackgroundService.<StopAsync>d__6>.MoveNext() Line 346   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Id = 3660, Status = RanToCompletion, Method = "{null}", Result = "System.Threading.Tasks.Task`1[System.Object]") Line 273 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 399 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.TwoTaskWhenAnyPromise<System.Threading.Tasks.Task>.Invoke(System.Threading.Tasks.Task completingTask) Line 6494  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3474    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetCanceled(System.Threading.CancellationToken tokenToRecord, object cancellationException) Line 3411 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource<object>.TrySetCanceled(System.Threading.CancellationToken cancellationToken) Line 281    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.SetCanceled() Line 235   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179  C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983   C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.TimerQueueTimer.Fire(bool isThreadPool) Line 686    C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 918 C#  Symbols loaded.
    System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 102    C#  Symbols loaded.
    [Native to Managed Transition]      Annotated Frame
    kernel32.dll!00007fffe9487034() Unknown No symbols loaded.
    ntdll.dll!00007fffea29d241()    Unknown No symbols loaded.
    [Async Call Stack]      Annotated Frame
    [Async] Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false)   Unknown No symbols loaded.
    [Async] Microsoft.AspNetCore.Mvc.Testing.dll!Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<ZapMiddleware.Webhooks.Api.Startup>.DisposeAsync()  Unknown No symbols loaded.

The summary of stack traces from VS' Parallel Stacks tool shows that EventListener.EventListenersLock and EventPipeEventDispatcher.m_dispatchControlLock locks are taken in the conflicting orders by the different code paths:

image

Taking EventListener.EventListenersLock lock in EventListener.Dispose seems to be against advice given by this comment: 

// We take this lock to synchronize access to the shared session state. It is important to never take the EventSource.EventListenersLock while
// holding this, or we can deadlock. Unfortunately calling in to EventSource at all can take the EventListenersLock in ways that are not obvious,
// so don't call in to EventSource or other EventListeners while holding this unless you are certain it can't take the EventListenersLock.

I could find following issues and PRs relevant to the problem:

Reproduction Steps

Not available at this time.

Expected behavior

One should be able to concurrently and repeatedly dispose EventListeners.

Actual behavior

Deadlock occurs.

Regression?

No response

Known Workarounds

No response

Configuration

.NET Core 8 Windows 10 x64

Other information

No response

ghost commented 10 months ago

Tagging subscribers to this area: @mangod9 See info in area-owners.md if you want to be subscribed.

Issue Details
### Description We have number of tests that are creating and disposing `System.Diagnostics.Tracing.EventListener`s. Since upgrading to .NET Core 8 if the tests are executed concurrently, using `dotnet test`, test runners started hanging for some of our tests. Attaching a debugger reveals a deadlock occurs in the code that is disposing `EventListener`. Here's the stack traces of the two threads involved in the deadlock. Thread1: ``` [Deadlocked, double-click or press enter to view all thread stacks] Annotated Frame [Waiting on lock owned by Thread 23448, double-click or press enter to switch to thread] Annotated Frame > System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventSource.SendCommand(System.Diagnostics.Tracing.EventListener listener, System.Diagnostics.Tracing.EventProviderType eventProviderType, int perEventSourceSessionId, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary commandArguments) Line 2569 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventSource.OverrideEventProvider.OnControllerCommand(System.Diagnostics.Tracing.ControllerCommand command, System.Collections.Generic.IDictionary arguments, int perEventSourceSessionId) Line 2440 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventProvider.HandleEnableNotification(System.Diagnostics.Tracing.EventProvider target, byte* additionalData, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData) Line 59 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventProviderImpl.ProviderCallback(System.Diagnostics.Tracing.EventProvider target, byte* additionalData, int controlCode, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData) Line 1310 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventProvider.Callback(byte* sourceId, int isEnabled, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData, void* callbackContext) Line 70 C# Symbols loaded. [Native to Managed Transition] Annotated Frame System.Private.CoreLib.dll!00007fff206cb4d5() Unknown No symbols loaded. [Managed to Native Transition] Annotated Frame System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeInternal.Enable(string outputFile, System.Diagnostics.Tracing.EventPipeSerializationFormat format, uint circularBufferSizeInMB, System.Diagnostics.Tracing.EventPipeProviderConfiguration[] providers = {System.Diagnostics.Tracing.EventPipeProviderConfiguration[1]}) Line 135 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.CommitDispatchConfiguration() Line 106 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords) Line 64 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.Dispose() Line 4083 C# Symbols loaded. Prometheus.NetCore.dll!Prometheus.EventCounterAdapter.Dispose() Line 84 C# Symbols loaded. ZapMiddleware.Shared.WebApi.dll!ZapMiddleware.Shared.WebApi.Metrics.EventCounterMetricsListenerHostedService.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = true) Line 18 C# Symbols loaded. Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.ForeachService(System.Collections.Generic.IEnumerable services = {System.Linq.Enumerable.ReverseIterator}, System.Threading.CancellationToken token = IsCancellationRequested = true, bool concurrent = false, bool abortOnFirstException = false, System.Collections.Generic.List exceptions = Count = 0, System.Func operation = {Method = {System.Reflection.RuntimeMethodInfo}}) Unknown No symbols loaded. [Resuming Async Method] Annotated Frame System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.Internal.Host+d__18`1[Microsoft.Extensions.Hosting.IHostedService]].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.MoveNext() Line 346 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 273 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() Line 109 C# Symbols loaded. [Completed] Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.BackgroundService.StopAsync(System.Threading.CancellationToken cancellationToken) Unknown Non-user code. Symbol loading disabled by Include/Exclude setting. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext() Line 346 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Id = 3651, Status = RanToCompletion, Method = "{null}", Result = "System.Threading.Tasks.Task`1[System.Object]") Line 273 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetResult(System.__Canon result) Line 399 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TwoTaskWhenAnyPromise.Invoke(System.Threading.Tasks.Task completingTask) Line 6494 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3474 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetCanceled(System.Threading.CancellationToken tokenToRecord, object cancellationException) Line 3411 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource.TrySetCanceled(System.Threading.CancellationToken cancellationToken) Line 281 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource.SetCanceled() Line 235 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.TimerQueueTimer.Fire(bool isThreadPool) Line 686 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.TimerQueue.FireNextTimers() Line 331 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 918 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 102 C# Symbols loaded. [Native to Managed Transition] Annotated Frame kernel32.dll!00007fffe9487034() Unknown No symbols loaded. ntdll.dll!00007fffea29d241() Unknown No symbols loaded. [Async Call Stack] Annotated Frame [Async] Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Unknown No symbols loaded. [Async] Microsoft.AspNetCore.Mvc.Testing.dll!Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory.DisposeAsync() Unknown No symbols loaded. ``` Thread 2: ``` [Deadlocked, double-click or press enter to view all thread stacks] Annotated Frame [Waiting on lock owned by Thread 48596, double-click or press enter to switch to thread] Annotated Frame > System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener = {Prometheus.EventCounterAdapter.Listener}, System.Diagnostics.Tracing.EventCommand command = Update, bool enable = false, System.Diagnostics.Tracing.EventLevel level = LogAlways, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords = None) Line 52 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.DisableEvents(System.Diagnostics.Tracing.EventSource eventSource) Line 4158 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.CallDisableEventsIfNecessary(System.Diagnostics.Tracing.EventDispatcher eventDispatcher = {System.Diagnostics.Tracing.EventDispatcher}, System.Diagnostics.Tracing.EventSource eventSource = {System.Diagnostics.Tracing.NativeRuntimeEventSource}) Line 4318 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.RemoveReferencesToListenerInEventSources(System.Diagnostics.Tracing.EventListener listenerToRemove = {Prometheus.EventCounterAdapter.Listener}) Line 4360 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.Dispose() Line 4069 C# Symbols loaded. Prometheus.NetCore.dll!Prometheus.EventCounterAdapter.Dispose() Line 84 C# Symbols loaded. ZapMiddleware.Shared.WebApi.dll!ZapMiddleware.Shared.WebApi.Metrics.EventCounterMetricsListenerHostedService.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = true) Line 18 C# Symbols loaded. Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.ForeachService(System.Collections.Generic.IEnumerable services = {System.Linq.Enumerable.ReverseIterator}, System.Threading.CancellationToken token = IsCancellationRequested = true, bool concurrent = false, bool abortOnFirstException = false, System.Collections.Generic.List exceptions = Count = 0, System.Func operation = {Method = {System.Reflection.RuntimeMethodInfo}}) Unknown No symbols loaded. [Resuming Async Method] Annotated Frame System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.Internal.Host+d__18`1[Microsoft.Extensions.Hosting.IHostedService]].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.MoveNext() Line 346 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 273 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() Line 109 C# Symbols loaded. [Completed] Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.BackgroundService.StopAsync(System.Threading.CancellationToken cancellationToken) Unknown Non-user code. Symbol loading disabled by Include/Exclude setting. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext() Line 346 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Id = 3660, Status = RanToCompletion, Method = "{null}", Result = "System.Threading.Tasks.Task`1[System.Object]") Line 273 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetResult(System.__Canon result) Line 399 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TwoTaskWhenAnyPromise.Invoke(System.Threading.Tasks.Task completingTask) Line 6494 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3474 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetCanceled(System.Threading.CancellationToken tokenToRecord, object cancellationException) Line 3411 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource.TrySetCanceled(System.Threading.CancellationToken cancellationToken) Line 281 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource.SetCanceled() Line 235 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.TimerQueueTimer.Fire(bool isThreadPool) Line 686 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 918 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 102 C# Symbols loaded. [Native to Managed Transition] Annotated Frame kernel32.dll!00007fffe9487034() Unknown No symbols loaded. ntdll.dll!00007fffea29d241() Unknown No symbols loaded. [Async Call Stack] Annotated Frame [Async] Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Unknown No symbols loaded. [Async] Microsoft.AspNetCore.Mvc.Testing.dll!Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory.DisposeAsync() Unknown No symbols loaded. ``` The summary of stack traces from VS' Parallel Stacks tool shows that `EventListener.EventListenersLock` and `EventPipeEventDispatcher.m_dispatchControlLock` locks are taken in the conflicting orders by the different code paths: ![image](https://github.com/dotnet/runtime/assets/42535720/a87227f1-19ce-41c4-8dc9-2d16d6e34439) Taking `EventListener.EventListenersLock` lock in `EventListener.Dispose` seems to be against advice given by [this comment](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventPipeEventDispatcher.cs#L34-L37): ``` // We take this lock to synchronize access to the shared session state. It is important to never take the EventSource.EventListenersLock while // holding this, or we can deadlock. Unfortunately calling in to EventSource at all can take the EventListenersLock in ways that are not obvious, // so don't call in to EventSource or other EventListeners while holding this unless you are certain it can't take the EventListenersLock. ``` I could find following issues and PRs relevant to the problem: - https://github.com/dotnet/runtime/issues/78233 - https://github.com/dotnet/runtime/pull/79185 - https://github.com/dotnet/runtime/issues/86838 - https://github.com/dotnet/runtime/pull/90053 ### Reproduction Steps Not available at this time. ### Expected behavior One should be able to concurrently and repeatedly dispose `EventListener`s. ### Actual behavior Deadlock occurs. ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration .NET Core 8 Windows 10 x64 ### Other information _No response_
Author: kataklinger
Assignees: -
Labels: `area-System.Threading`, `untriaged`
Milestone: -
ghost commented 10 months ago

Tagging subscribers to this area: @tarekgh, @tommcdon, @pjanotti See info in area-owners.md if you want to be subscribed.

Issue Details
### Description We have number of tests that are creating and disposing `System.Diagnostics.Tracing.EventListener`s. Since upgrading to .NET Core 8 if the tests are executed concurrently, using `dotnet test`, test runners started hanging for some of our tests. Attaching a debugger reveals a deadlock occurs in the code that is disposing `EventListener`. Here's the stack traces of the two threads involved in the deadlock. Thread1: ``` [Deadlocked, double-click or press enter to view all thread stacks] Annotated Frame [Waiting on lock owned by Thread 23448, double-click or press enter to switch to thread] Annotated Frame > System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventSource.SendCommand(System.Diagnostics.Tracing.EventListener listener, System.Diagnostics.Tracing.EventProviderType eventProviderType, int perEventSourceSessionId, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary commandArguments) Line 2569 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventSource.OverrideEventProvider.OnControllerCommand(System.Diagnostics.Tracing.ControllerCommand command, System.Collections.Generic.IDictionary arguments, int perEventSourceSessionId) Line 2440 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventProvider.HandleEnableNotification(System.Diagnostics.Tracing.EventProvider target, byte* additionalData, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData) Line 59 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventProviderImpl.ProviderCallback(System.Diagnostics.Tracing.EventProvider target, byte* additionalData, int controlCode, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData) Line 1310 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventProvider.Callback(byte* sourceId, int isEnabled, byte level, long matchAnyKeywords, long matchAllKeywords, Interop.Advapi32.EVENT_FILTER_DESCRIPTOR* filterData, void* callbackContext) Line 70 C# Symbols loaded. [Native to Managed Transition] Annotated Frame System.Private.CoreLib.dll!00007fff206cb4d5() Unknown No symbols loaded. [Managed to Native Transition] Annotated Frame System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeInternal.Enable(string outputFile, System.Diagnostics.Tracing.EventPipeSerializationFormat format, uint circularBufferSizeInMB, System.Diagnostics.Tracing.EventPipeProviderConfiguration[] providers = {System.Diagnostics.Tracing.EventPipeProviderConfiguration[1]}) Line 135 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.CommitDispatchConfiguration() Line 106 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords) Line 64 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.Dispose() Line 4083 C# Symbols loaded. Prometheus.NetCore.dll!Prometheus.EventCounterAdapter.Dispose() Line 84 C# Symbols loaded. ZapMiddleware.Shared.WebApi.dll!ZapMiddleware.Shared.WebApi.Metrics.EventCounterMetricsListenerHostedService.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = true) Line 18 C# Symbols loaded. Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.ForeachService(System.Collections.Generic.IEnumerable services = {System.Linq.Enumerable.ReverseIterator}, System.Threading.CancellationToken token = IsCancellationRequested = true, bool concurrent = false, bool abortOnFirstException = false, System.Collections.Generic.List exceptions = Count = 0, System.Func operation = {Method = {System.Reflection.RuntimeMethodInfo}}) Unknown No symbols loaded. [Resuming Async Method] Annotated Frame System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.Internal.Host+d__18`1[Microsoft.Extensions.Hosting.IHostedService]].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.MoveNext() Line 346 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 273 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() Line 109 C# Symbols loaded. [Completed] Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.BackgroundService.StopAsync(System.Threading.CancellationToken cancellationToken) Unknown Non-user code. Symbol loading disabled by Include/Exclude setting. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext() Line 346 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Id = 3651, Status = RanToCompletion, Method = "{null}", Result = "System.Threading.Tasks.Task`1[System.Object]") Line 273 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetResult(System.__Canon result) Line 399 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TwoTaskWhenAnyPromise.Invoke(System.Threading.Tasks.Task completingTask) Line 6494 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3474 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetCanceled(System.Threading.CancellationToken tokenToRecord, object cancellationException) Line 3411 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource.TrySetCanceled(System.Threading.CancellationToken cancellationToken) Line 281 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource.SetCanceled() Line 235 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.TimerQueueTimer.Fire(bool isThreadPool) Line 686 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.TimerQueue.FireNextTimers() Line 331 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 918 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 102 C# Symbols loaded. [Native to Managed Transition] Annotated Frame kernel32.dll!00007fffe9487034() Unknown No symbols loaded. ntdll.dll!00007fffea29d241() Unknown No symbols loaded. [Async Call Stack] Annotated Frame [Async] Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Unknown No symbols loaded. [Async] Microsoft.AspNetCore.Mvc.Testing.dll!Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory.DisposeAsync() Unknown No symbols loaded. ``` Thread 2: ``` [Deadlocked, double-click or press enter to view all thread stacks] Annotated Frame [Waiting on lock owned by Thread 48596, double-click or press enter to switch to thread] Annotated Frame > System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener = {Prometheus.EventCounterAdapter.Listener}, System.Diagnostics.Tracing.EventCommand command = Update, bool enable = false, System.Diagnostics.Tracing.EventLevel level = LogAlways, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords = None) Line 52 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.DisableEvents(System.Diagnostics.Tracing.EventSource eventSource) Line 4158 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.CallDisableEventsIfNecessary(System.Diagnostics.Tracing.EventDispatcher eventDispatcher = {System.Diagnostics.Tracing.EventDispatcher}, System.Diagnostics.Tracing.EventSource eventSource = {System.Diagnostics.Tracing.NativeRuntimeEventSource}) Line 4318 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.RemoveReferencesToListenerInEventSources(System.Diagnostics.Tracing.EventListener listenerToRemove = {Prometheus.EventCounterAdapter.Listener}) Line 4360 C# Symbols loaded. System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.Dispose() Line 4069 C# Symbols loaded. Prometheus.NetCore.dll!Prometheus.EventCounterAdapter.Dispose() Line 84 C# Symbols loaded. ZapMiddleware.Shared.WebApi.dll!ZapMiddleware.Shared.WebApi.Metrics.EventCounterMetricsListenerHostedService.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = true) Line 18 C# Symbols loaded. Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.ForeachService(System.Collections.Generic.IEnumerable services = {System.Linq.Enumerable.ReverseIterator}, System.Threading.CancellationToken token = IsCancellationRequested = true, bool concurrent = false, bool abortOnFirstException = false, System.Collections.Generic.List exceptions = Count = 0, System.Func operation = {Method = {System.Reflection.RuntimeMethodInfo}}) Unknown No symbols loaded. [Resuming Async Method] Annotated Frame System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.Internal.Host+d__18`1[Microsoft.Extensions.Hosting.IHostedService]].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__18>.MoveNext() Line 346 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 273 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() Line 109 C# Symbols loaded. [Completed] Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.BackgroundService.StopAsync(System.Threading.CancellationToken cancellationToken) Unknown Non-user code. Symbol loading disabled by Include/Exclude setting. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.ExecutionContextCallback(object s = Evaluation of method System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Extensions.Hosting.BackgroundService+d__6].get_DebuggerDisplay requires calling method System.RuntimeType.GetEnumUnderlyingType, which cannot be called in this context.) Line 289 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext(System.Threading.Thread threadPoolThread = null) Line 368 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBoxd__6>.MoveNext() Line 346 C# Symbols loaded. System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask = Id = 3660, Status = RanToCompletion, Method = "{null}", Result = "System.Threading.Tasks.Task`1[System.Object]") Line 273 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 743 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3462 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetResult(System.__Canon result) Line 399 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TwoTaskWhenAnyPromise.Invoke(System.Threading.Tasks.Task completingTask) Line 6494 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3474 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetCanceled(System.Threading.CancellationToken tokenToRecord, object cancellationException) Line 3411 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource.TrySetCanceled(System.Threading.CancellationToken cancellationToken) Line 281 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource.SetCanceled() Line 235 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 179 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.Invoke(System.Delegate d, object state, System.Threading.CancellationTokenSource source) Line 983 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(bool throwOnFirstException = false) Line 806 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.TimerQueueTimer.Fire(bool isThreadPool) Line 686 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 918 C# Symbols loaded. System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 102 C# Symbols loaded. [Native to Managed Transition] Annotated Frame kernel32.dll!00007fffe9487034() Unknown No symbols loaded. ntdll.dll!00007fffea29d241() Unknown No symbols loaded. [Async Call Stack] Annotated Frame [Async] Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.StopAsync(System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Unknown No symbols loaded. [Async] Microsoft.AspNetCore.Mvc.Testing.dll!Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory.DisposeAsync() Unknown No symbols loaded. ``` The summary of stack traces from VS' Parallel Stacks tool shows that `EventListener.EventListenersLock` and `EventPipeEventDispatcher.m_dispatchControlLock` locks are taken in the conflicting orders by the different code paths: ![image](https://github.com/dotnet/runtime/assets/42535720/a87227f1-19ce-41c4-8dc9-2d16d6e34439) Taking `EventListener.EventListenersLock` lock in `EventListener.Dispose` seems to be against advice given by [this comment](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventPipeEventDispatcher.cs#L34-L37): ``` // We take this lock to synchronize access to the shared session state. It is important to never take the EventSource.EventListenersLock while // holding this, or we can deadlock. Unfortunately calling in to EventSource at all can take the EventListenersLock in ways that are not obvious, // so don't call in to EventSource or other EventListeners while holding this unless you are certain it can't take the EventListenersLock. ``` I could find following issues and PRs relevant to the problem: - https://github.com/dotnet/runtime/issues/78233 - https://github.com/dotnet/runtime/pull/79185 - https://github.com/dotnet/runtime/issues/86838 - https://github.com/dotnet/runtime/pull/90053 ### Reproduction Steps Not available at this time. ### Expected behavior One should be able to concurrently and repeatedly dispose `EventListener`s. ### Actual behavior Deadlock occurs. ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration .NET Core 8 Windows 10 x64 ### Other information _No response_
Author: kataklinger
Assignees: -
Labels: `area-System.Diagnostics.Tracing`, `area-System.Threading`, `untriaged`
Milestone: -
tommcdon commented 10 months ago

@kataklinger thanks for the excellent analysis! Please feel free to submit a PR if you are interested.

mdh1418 commented 1 week ago

Hi @kataklinger, recently there have been several PRs merged that addressed similar deadlocks in EventSource/EventListener, so this may have been fixed. Do you have a simple repro for the scenario where you're running into the deadlock, or would you be able to try out .NET 9.0 (https://dotnet.microsoft.com/en-us/download/dotnet) to see if the problem persists?

dotnet-policy-service[bot] commented 1 week ago

This issue has been marked needs-author-action and may be missing some important information.

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.