Occasionally the data node just stops adding blocks.
On investigation it was found that there's a deadlock possible when
The ingestion goroutine is trying to notify that there's some new MarketDepth data
The observer attached to the MarketDepth service takes a mutex in order to iterate over it's list of subscribers
Because MarketDepth was using a blocking notification scheme, it pushes a value onto a channel and waits for it to be read
Meanwhile, perhaps due to an closed client, an request to unsubscribe comes in and tries to take the same mutex but can't
And we have a deadlock
I propose to remove the concept of a blocking notifier (which I implemented just to try and replicate what was happening in the old market depth service, but I think it might have been an unintentional 'feature' in the first place)
Further I noticed that for all the rest of the 'non blocking' notifiers, if the buffer was full when trying to notify it would silently drop the events. I would like to change it so that the channel is closed and the subscription is dropped if it fails to send; that way at least clients will know they are missing data.
Occasionally the data node just stops adding blocks.
On investigation it was found that there's a deadlock possible when
observerattached to the MarketDepth service takes a mutex in order to iterate over it's list of subscribersI propose to remove the concept of a blocking notifier (which I implemented just to try and replicate what was happening in the old market depth service, but I think it might have been an unintentional 'feature' in the first place)
Further I noticed that for all the rest of the 'non blocking' notifiers, if the buffer was full when trying to notify it would silently drop the events. I would like to change it so that the channel is closed and the subscription is dropped if it fails to send; that way at least clients will know they are missing data.