search

Azure / azure-amqp

AMQP C# library
Other
94 stars 72 forks source link

ThreadPool usage #190

Closed JoshLove-msft closed 3 years ago

JoshLove-msft commented 3 years ago

While attempting to track down a bug in the Service Bus SDK involving occasional lock renewal exceptions when using a processor with concurrency of 20 or higher. I tracked the issue down to the fact that the message ends up getting delivered to the Service Bus receiver after the lock has already expired. When OnReceiveMessage is called, the lock is still valid, but there is a long delay before the receive operation is signaled as having completed (and the ServiceBusReceiver gets the message).

The code that is signaling the completion is this:

ActionItem.Schedule(o => {
  var w = (ReceiveAsyncResult)o; w.Signal(false);
  Console.WriteLine($"{DateTimeOffset.UtcNow.ToString("o")}: callback executed: {lockTokenGuid}"); 
  }, waiter);

If I change this code to:

Task.Run(() => waiter.Signal(false));

I can no longer reproduce the issue. ActionItem.Schedule calls into ThreadPool.QueueUserWorkItem. Perhaps using the Task library is better at optimizing the threadpool usage than using the ThreadPool API directly.

If I use the original code but I call ThreadPool.SetMinThreads(100, 100), it also works, further indicating that the issue stems from the thread pool thread allocation.

JoshLove-msft commented 3 years ago

Root cause was blocking that was occurring in the user callback.