madskristensen / BundlerMinifier

Visual Studio extension
Other
615 stars 172 forks source link

Avoid overlapping timers to avoid threadpool starvation #535

Closed davkean closed 3 years ago

davkean commented 3 years ago

This fix avoids starting another callback until the current one has finished to avoid thread pool starvation.

We've automatically captured dumps where overlapping timers are all blocked on the same thread:

  ntdll.dll!_ZwWaitForMultipleObjects@20() Line 825 Unknown
  KERNELBASE.dll!WaitForMultipleObjectsEx(unsigned long nCount, void * const * lpHandles, int bWaitAll, unsigned long dwMilliseconds, int bAlertable) Line 1551 C
  [Managed to Native Transition]
  mscorlib.dll!System.Collections.Concurrent.ConcurrentDictionary<EnvDTE.Project, System.IO.FileSystemWatcher>.AcquireLocks(int fromInclusive, int toExclusive, ref int locksAcquired) Unknown
  mscorlib.dll!System.Collections.Concurrent.ConcurrentDictionary<EnvDTE.Project, System.IO.FileSystemWatcher>.AcquireAllLocks(ref int locksAcquired) Unknown
  mscorlib.dll!System.Collections.Concurrent.ConcurrentDictionary<EnvDTE.Project, System.IO.FileSystemWatcher>.GetValues() Unknown
> BundlerMinifierVsix.dll!BundlerMinifierVsix.Commands.ProjectEventCommand.TimerElapsed(object state) Unknown
  mscorlib.dll!System.Threading.TimerQueueTimer.CallCallbackInContext(object state) Unknown
  mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
  mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
  mscorlib.dll!System.Threading.TimerQueueTimer.CallCallback() Unknown
  mscorlib.dll!System.Threading.TimerQueueTimer.Fire() Unknown
  mscorlib.dll!System.Threading.TimerQueue.FireQueuedTimerCompletion(object state) Unknown
  mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Unknown
  mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Unknown
  mscorlib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() Unknown

In some dumps we saw over 1800 threads with the above stack.

Failure Score Size 90th (MB) Size 50th (MB) AvgSize (MB)
Thd_ThreadStack_bundlerminifiervsix!BundlerMinifierVsix.Commands.ProjectEventCommand.TimerElapsed 590 1961.89 1616.9 1789.39

Internal bug for this is: 1279025.

davkean commented 3 years ago

tag @madskristensen

madskristensen commented 3 years ago

Thanks @davkean