TestKit exception: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state.

[tjaskula]

I'm not sure if the real problem is in TestKit but this exception I had while testing in Vlingo.Wire. Needs to be investigated. @zpbappi if you have some suggestions I would be grateful here.

consoleLogger: Inbound stream listening: for 'mock'

                                                   Unhandled Exception:Unhandled Exception:Unhandled Exception:   
                                                                                                                  Unhandled Exception:
                                                                                                                                       Unhandled Exception: 
                                                                                                                                                            Unhandled Exception: 
                                                                                                                                                                                 Unhandled Exception: 

                                                                                                                                                                                                      Unhandled Exception:Unhandled Exception:  System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                       at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
                                                                                                                                  at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
                                                                                                                                                                                                                at Vlingo.Actors.TestKit.TestWorld.Track(IMessage message)
                                          at Vlingo.Actors.Plugin.Mailbox.TestKit.TestMailbox.Send(IMessage message)
                                                                                                                       at Vlingo.Actors.Scheduled__Proxy.IntervalSignal(IScheduled scheduled, Object data)
                                                                                                                                                                                                             at Vlingo.Common.Scheduler.SchedulerTask.Tick(Object data)
                                       at System.Threading.TimerQueueTimer.<>c.<.cctor>b__22_0(Object state)
                                                                                                               at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
       --- End of stack trace from previous location where exception was thrown ---
                                                                                      at System.Threading.ExeSystem.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                                                                                                                       at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
                                                                                                                                                                                        at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
                     at Vlingo.Actors.TestKit.TestWorld.Track(IMessage message)
                                                                                  at Vlingo.Actors.Plugin.Mailbox.TestKit.TestMailbox.Send(IMessage message)
                                                                                                                                                               at Vlingo.Actors.Scheduled__Proxy.IntervalSignal(IScheduled scheduled, Object data)
                  at Vlingo.Common.Scheduler.SchedulerTask.Tick(Object data)
                                                                               at System.Threading.TimerQueueTimer.<>c.<.cctor>b__22_0(Object state)
                                                                                                                                                       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
                                               --- End of stack trace from previous location where exception was thrown ---
                                                                                                                              at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
                         at System.Threading.TimerQueueTimer.CallCallback()
                                                                              at System.Threading.TimerQueueTimer.Fire()
                                                                                                                           at System.Threading.TimerQueue.FireNextTimers()
                                                                                                                                                                             at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
                                                                                                                              at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
                                                                                                                                                                                                            at Vlingo.Actors.TestKit.TestWorld.Track(IMessage message)
                                      at Vlingo.Actors.Plugin.Mailbox.TestKit.TestMailbox.Send(IMessage message)
                                                                                                                   at Vlingo.Actors.Scheduled__Proxy.IntervalSignal(IScheduled scheduled, Object data)
                                                                                                                                                                                                         at Vlingo.Common.Scheduler.SchedulerTask.Tick(Object data)
                                   at System.Threading.TimerQueueTimer.<System.NullReferenceException: Object reference not set to an instance of an object.
                                                                                                                                                               at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
                                                                                                                                                                                                                                at System.Collections.Generic.Dictionary`2.ContainsKey(TKey key)
                                                                at Vlingo.Actors.TestKit.TestWorld.Track(IMessage message)
                                                                                                                             at Vlingo.Actors.Plugin.Mailbox.TestKit.TestMailbox.Send(IMessage message)
                                                                                                                                                                                                          at Vlingo.Actors.Scheduled__Proxy.IntervalSignal(IScheduled scheduled, Object data)
                                                             at Vlingo.Common.Scheduler.SchedulerTask.Tick(Object data)
                                                                                                                          at System.Threading.TimerQueueTimer.<>c.<.cctor>b__22_0(Object state)
                                                                                                                                                                                                  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
                                                                                          --- End of stack trace from previous location where exception was thrown ---
                                                                                                                                                                         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
                                                                    at System.Threading.TimerQueueTimer.CallCallback()
                                                                                                                         at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()
                                                     at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                                                                                                                              at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
                                                                                                                                                                                               at System.Collections.Generic.Dictionary`2.ContainsKey(TKey key)
                               at Vlingo.Actors.TestKit.TestWorld.Track(IMessage message)
                                                                                            at Vlingo.Actors.Plugin.Mailbox.TestKit.TestMailbox.Send(IMessage message)
                                                                                                                                                                         at Vlingo.Actors.Scheduled__Proxy.IntervalSignal(IScheduled scheduled, Object data)
                            at Vlingo.Common.Scheduler.SchedulerTask.Tick(Object data)
                                                                                         at System.Threading.TimerQueueTimer.<>c.<.cctor>b__22_0(Object state)
                                                                                                                                                                 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
                                                         --- End of stack trace from previous location where exception was thrown ---
                                                                                                                                        at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
                                   at System.Threading.TimerQueueTimer.CallCallback()
                                                                                        at System.Threading.TimerQueueTimer.Fire()
                                                                                                                                     at System.Threading.TimerQueueTimer.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
                                                                                                                                                                                                                                  at System.Threading.ThreadPoolWorkQueue.Dispatch()
                                                    at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
                                                                                                                     System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                                                                                                                               at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
       at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
                                                                                     at Vlingo.Actors.TestKit.TestWorld.Track(IMessage message)
                                                                                                                                                  at Vlingo.Actors.Plugin.Mailbox.TestKit.Te>c.<.cctor>b__22_0(Object state)
                                                                                                                                                                                                                               at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
                                                                                                                       --- End of stack trace from previous location where exception was thrown ---
                                                                                                                                                                                                      at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
                                                                                                 at System.Threading.TimerQueueTimer.CallCallback()
                                                                                                                                                      at System.Threading.TimerQueueTimer.Fire()
                                                                                                                                                                                                   at System.Threading.TimerQueue.FireNextTimers()
                  at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)
                                                                                 System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                                                                                           at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
                                                                                                                                                            at System.Collections.Generic.Dictionary`2.ContainsKey(TKey key)
                                                                                                                                                                                                                               at Vlingo.Actors.TestKit.TestWorld.Track(IMessage message)

[VaughnVernon]

Wouldn't changing the collection type to concurrent be the fix?

[tjaskula]

This may be a fix, but I wanted to check first if something else is not a cause of this exception and if multiple threads here is a normal behavior.

[VaughnVernon]

The idea behind TestWorld is that actors created directly through it are wrapped by TestActor and have synchronous delivery. The TestMailbox records (tracks) all messages delivered through it into a HashMap, which is not a concurrent capable collection. Thus I would say that it should be "impossible" to get concurrent access here, but I don't know the semantics of your TestActor instances and whether there are threads/tasks effecting async access around the TestActors/TestMailbox. To do so is officially incorrect use of these facilities, but again your socket readers might cause this. I don't know.

[zpbappi]

@tjaskula will look into it and give you an explanation of why it is happening or why it shouldn't happen.

[tjaskula]

@zpbappi thanks, I'll have a look this evening as well.

[tjaskula]

@zpbappi this seems to me like it was a side effect of the code I had. It's hard to explain but now when the underlying code is fixed I cannot reproduce this error. I consider this is not a valid bug anymore. But at least I have an explanation from @VaughnVernon what was the idea behind TestWorld