Closed pashchuk closed 6 years ago
hmm I wonder how much of the blocking collection issue is related to https://github.com/dotnet/corefx/issues/30781 . They say they fixed that issue in 2.1.5 release. But they never released 2.1.5 based on their website.. hmm
I say the issue looks related because, based on our InboundMessageQueue.WaitMessage
code, it would catch InvalidOperationException
and return a null
if this.messageQueues[(int)type].Take();
throws a InvalidOperationException
, which sounds like the path your exception log is hitting. And this is very similar to the issue described in https://github.com/dotnet/corefx/issues/30781
Maybe upgrade to corefx 2.2 would fix this issue.
Regarding ObjectDisposedException it really minor but a bit annoying because why the silo is trying to dispose the service provider again?
I never see this before in our tests. Maybe it has something to do with running inside Kubernetus . Do you have a repro of this?
The ObjectDisposedException should be much less common on 2.1.0. It occurs because worker threads are waiting for work, but the queue they wait on is disposed from under them. In 2.1.0 we have a new scheduler which doesn't operate in the same way and so the exceptions should not occur.
I believe we had some code to avoid those exceptions by pumping the queue once after cancellation, but I cannot see it anymore.
I can reproduce ArgumentNullException
on my local machine.
I propose either 1) suppressing this kind of exception or 2) canceling the blocking thread before destroying in-used objects (e.g. message center ->inbound/outbound queue -> BlockingCollection -> SemaphoreSlim ).
These strange exceptions scary newbie (like me).
When the silo is starting to shut down gracefully, at some point strange exceptions has been thrown. The most important ones:
and
and
Less important
the silo is deployed to the Kubernetus v1.10.6-gke.2 with 3 replicas as a StatefulSet using AdoNet clustering provider Orleans v2.1.0-beta1 dotnet sdk v2.1.401 dotnet runtime v2.1.3
Looks like there is something bad happened in the corefx blocking collection. Regarding ObjectDisposedException it really minor but a bit annoying because why the silo is trying to dispose the service provider again? Does anyone know anything about this?