IndexOutOfRangeException in DisconnectedPerformanceMonitor.PulseAsync()

[saguiitay]

I'm periodically calling DisconnectedPerformanceMonitor.PulseAsync() to collect statistics on my TaskHubs.

Every now and then I encounter an IndexOutOfRange exception with the following stack:

ClientCommandRequest: Details=System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at DurableTask.AzureStorage.Monitoring.DisconnectedPerformanceMonitor.QueueMetricHistory.Add(Int32 value) in /_/src/DurableTask.AzureStorage/Monitoring/DisconnectedPerformanceMonitor.cs:line 558
   at DurableTask.AzureStorage.Monitoring.DisconnectedPerformanceMonitor.<UpdateQueueMetrics>d__29.MoveNext() in /_/src/DurableTask.AzureStorage/Monitoring/DisconnectedPerformanceMonitor.cs:line 188
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DurableTask.AzureStorage.Monitoring.DisconnectedPerformanceMonitor.<PulseAsync>d__28.MoveNext() in /_/src/DurableTask.AzureStorage/Monitoring/DisconnectedPerformanceMonitor.cs:line 138
--- End of stack trace from previous location where exception was thrown ---
...

This looks like a race condition somewhere.

[cgillum]

@nytiannn since you've been looking at this code lately, would you be able to take a look and see if there is a bug or race condition that might cause this error in the current implementation?

[saguiitay]

@nytian any update on this? It keeps happening sporadically.