{key} is missing from previous value

[hefajj]

I have no idea how to debug exception of type 'DynamicData.MissingKeyException' - "{key} is missing from previous value". It is happening on on .AddOrUpdate action. Record with this {key} already exist in the SourceCash, so this is on Update.

[RolandPheasant]

This will be happening downstream, or at least I think it is the same an a problem which I recently fixed.

If you get the latest beta from nuget.org, I am optimistic the problem will go away.

Let me know

[RolandPheasant]

If not could you copy the stack trace

[RolandPheasant]

A clue as well is this exception is only thrown in the group operator

[hefajj]

Upgraded to Beta. Now getting: {"4_HUF is missing from previous value on update. Object type RxDynamicData1.CurPositionPerClient, Key type System.String, Group key type System.Int64"}

StackTrace: at DynamicData.Kernel.Continuation1.Then(Action1 onComplete, Action1 onError) in C:\projects\dynamicdata-dpbpa\DynamicData\Kernel\Continuation.cs:line 40 at DynamicData.ObservableCache2.UpdateFromIntermediate(Action1 updateAction, Action1 errorHandler) in C:\projects\dynamicdata-dpbpa\DynamicData\Cache\ObservableCache.cs:line 72 at DynamicData.Internal.Grouper3.<>cDisplayClass7_0.b_2(IGrouping2 group) in C:\projects\dynamicdata-dpbpa\DynamicData\Cache\Internal\Grouper.cs:line 110 at DynamicData.Kernel.EnumerableEx.ForEach[T](IEnumerable1 source, Action1 action) in C:\projects\dynamicdata-dpbpa\DynamicData\Kernel\EnumerableEx.cs:line 98 at DynamicData.Internal.Grouper3.HandleUpdates(IEnumerable1 changes, Boolean isRegrouping) in C:\projects\dynamicdata-dpbpa\DynamicData\Cache\Internal\Grouper.cs:line 103 at DynamicData.Internal.Grouper3.Update(IChangeSet2 updates) in C:\projects\dynamicdata-dpbpa\DynamicData\Cache\Internal\Grouper.cs:line 84 at System.Reactive.Linq.ObservableImpl.Select2..OnNext(TSource value) --- End of stack trace from previous location where exception was thrown --- at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) at System.Reactive.Stubs.<.cctor>b1(Exception ex) at System.Reactive.AnonymousObserver1.OnErrorCore(Exception error) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.AutoDetachObserver1.OnErrorCore(Exception exception) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.Linq.ObservableImpl.Do1._.OnError(Exception error) --- End of stack trace from previous location where exception was thrown --- at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) at System.Reactive.Stubs.<.cctor>b__1(Exception ex) at System.Reactive.AnonymousObserver1.OnErrorCore(Exception error) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.AutoDetachObserver1.OnErrorCore(Exception exception) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.Observer1.OnError(Exception error) at System.Reactive.Subjects.Subject1.OnError(Exception error) at System.Reactive.Linq.ObservableImpl.AsObservable1..OnError(Exception error) at System.Reactive.AutoDetachObserver1.OnErrorCore(Exception exception) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.Observer1.OnError(Exception error) at System.Reactive.Subjects.Subject1.OnError(Exception error) at System.Reactive.Linq.ObservableImpl.AsObservable`1..OnError(Exception error) at System.Reactive.Linq.ObservableImpl.Where1._.OnError(Exception error) at System.Reactive.Linq.ObservableImpl.Select2..OnNext(TSource value) at System.Reactive.AutoDetachObserver1.OnNextCore(T value) at System.Reactive.ObserverBase1.OnNext(T value) at System.Reactive.Linq.ObservableImpl.Where`1..OnNext(TSource value) at System.Reactive.Linq.ObservableImpl.Select2._.OnNext(TSource value) at System.Reactive.AutoDetachObserver1.OnNextCore(T value) at System.Reactive.ObserverBase1.OnNext(T value) at System.Reactive.AnonymousObserver1.OnNextCore(T value) at System.Reactive.ObserverBase1.OnNext(T value) at System.Reactive.Observer1.OnNext(T value) at System.Reactive.Subjects.Subject1.OnNext(T value) at DynamicData.ObservableCache2.InvokeNext(IChangeSet2 changes) in C:\projects\dynamicdata-dpbpa\DynamicData\Cache\ObservableCache.cs:line 96 --- End of stack trace from previous location where exception was thrown --- at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) at System.Reactive.Stubs.<.cctor>b__1(Exception ex) at System.Reactive.AnonymousObserver1.OnErrorCore(Exception error) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.AutoDetachObserver1.OnErrorCore(Exception exception) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.Linq.ObservableImpl.Do1._.OnError(Exception error) --- End of stack trace from previous location where exception was thrown --- at System.Reactive.PlatformServices.ExceptionServicesImpl.Rethrow(Exception exception) at System.Reactive.Stubs.<.cctor>b1(Exception ex) at System.Reactive.AnonymousObserver1.OnErrorCore(Exception error) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.AutoDetachObserver1.OnErrorCore(Exception exception) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.Observer1.OnError(Exception error) at System.Reactive.Subjects.Subject1.OnError(Exception error) at System.Reactive.Linq.ObservableImpl.AsObservable1._.OnError(Exception error) at System.Reactive.AutoDetachObserver1.OnErrorCore(Exception exception) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.Observer1.OnError(Exception error) at System.Reactive.Subjects.Subject1.OnError(Exception error) at System.Reactive.Linq.ObservableImpl.AsObservable1..OnError(Exception error) at System.Reactive.Linq.ObservableImpl.Where`1..OnError(Exception error) at System.Reactive.Linq.ObservableImpl.Select2._.OnError(Exception error) at System.Reactive.AutoDetachObserver1.OnErrorCore(Exception exception) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.Linq.ObservableImpl.Where1..OnError(Exception error) at System.Reactive.Linq.ObservableImpl.Select`2..OnError(Exception error) at System.Reactive.AutoDetachObserver1.OnErrorCore(Exception exception) at System.Reactive.ObserverBase1.OnError(Exception error) at DynamicData.DynamicDataEx.<>cDisplayClass0_11.<FinallySafe>b__1(Exception ex) in C:\projects\dynamicdata-dpbpa\DynamicData\DynamicDataEx.cs:line 50 at System.Reactive.AnonymousObserver1.OnErrorCore(Exception error) at System.Reactive.ObserverBase1.OnError(Exception error) at System.Reactive.Subjects.Subject1.OnError(Exception error) at DynamicData.ObservableCache2.InvokeNext(IChangeSet2 changes) in C:\projects\dynamicdata-dpbpa\DynamicData\Cache\ObservableCache.cs:line 103 at DynamicData.Kernel.Continuation1.Then(Action1 onComplete, Action1 onError) in C:\projects\dynamicdata-dpbpa\DynamicData\Kernel\Continuation.cs:line 45 at DynamicData.ObservableCache2.UpdateFromSource(Action1 updateAction, Action1 errorHandler) in C:\projects\dynamicdata-dpbpa\DynamicData\Cache\ObservableCache.cs:line 83 at DynamicData.SourceCache2.Edit(Action1 updateAction, Action1 errorHandler) in C:\projects\dynamicdata-dpbpa\DynamicData\Cache\SourceCache.cs:line 39 at DynamicData.ObservableCacheEx.AddOrUpdate[TObject,TKey](ISourceCache2 source, IEnumerable1 items) in C:\projects\dynamicdata-dpbpa\DynamicData\ObservableCacheEx.cs:line 399 at RxDynamicData1.Program.<>c.<Main>b__12_27(IQuery2 query) in C:\OneDrive\DEV\margin trading\Dynamic data\Excel2\Program.cs:line 563 at System.Reactive.Linq.ObservableImpl.Select2._.OnNext(TSource value)

[RolandPheasant]

Is the key value and the group key value immutable or at least invariant per Item?

Also is it possible to replicate this in a stand alone project or in some test code?

[RolandPheasant]

Also do you have multiple threads updating the cache? It should be thread safe but good to check.

[hefajj]

I had put together example where I try to AddOrUpdate the same thing in many places (Will Dynamic Data manage that for me? I tried to play with Synchronize, but no positive results.

https://github.com/hefajj/DynamicData-example

[RolandPheasant]

Thanks for that. It will make it much easier to trace the problem.

I will take a look this evening, after my day job

[RolandPheasant]

@hefajj I have got the example but have only managed to replicate the problem a couple of times. Is there a way to produce it more regularly?

However I have my suspicions. .Edit() uses a monitor lock which is re-entrant and some of the code seems to be writing back to the same services which produced the original update. I suspect it may be a case of shifting threads i.e. use ObserveOn. Also enumerating within an edit and writing back can cause issues, so applying a ToArray() before enumerating could help.

One other point worth mentioning but this may not relate to the problem is your domain objects should implement a hashcode and equality members (this is generally recommended when ToString() is overridden). Normally I produce a hashcode on ID (and version or timestamp if there is one). Resharper can do this for you.

Also there are also some operators which already handle some of what you are trying to do. I suggest I check it in to GitHub and change the code so you can see. I will if you are happy with that as I do not want to publish your code without your permission.

In the mean-time I will try and change the code to replicate the error more often

[RolandPheasant]

Also I just noticed this code

 item.AmountInBase = item.Amount * latestAskPrice;
updater.AddOrUpdate(item);   

I always recommend immutability (and not just for dynamic data and particularly for rx).

If would be better to construct a new object

//perhaps use a copy constructor like this
var  newItem = new CurPairPositionPerClient(item,item.Amount * latestAskPrice );
updater.AddOrUpdate(newItem );   

[RolandPheasant]

I experimented and applied a sensible hash code and managed to reduce the problem to CurrencyPositionPerClientUpdaterService.UpdateAllCurrenciesPositions. It is certainly a circular update. The problem can be designed out by splitting the objects into multiple objects. Currency pair with price is one data source, as price. Then by introducing a new object which takes the input streams and creates a derived stream the circular update will not exist. Dynamic data provides many options for joining these streams together.

This may sound complicated but the code will become much simpler, clearer and reliable.

Btw I am experienced with Fx trading apps so I understand the problem domain

[hefajj]

Thank you Roland for your contribution. I have just published this code on GitHub to share this example with everybody: https://github.com/hefajj/DynamicData-example I will try to do my best to leverage your superb library.

Feel free to commit there anything you think that is good for learning purposes or reuse it in any other place you want.

In CurrencyPositionPerClientUpdaterService.UpdateAllCurrenciesPositions I wanted to replicate what you have done in Dynamic.Trader. You don't have separate object, but you are updating tradeService with new price trades.ForEach(t=>t.SetMarketPrice(price) directly via groupedData.Cache.Items and then producing event _marketPriceChangedSubject.OnNext(marketPrice);

I thought that AddOrUpdate is doing more or less the same?

[RolandPheasant]

Thanks for publishing that. I will definitely contribute to the project so I can iron out the problems. Hopefully I will get a chance to do so next week.

[hefajj]

Deleting AddOrUpdate of the same ISourceCash in different places is solving the problem. I will try to split the objects into multiple objects as you recommended.