shayhatsor
commented
8 years ago @peter-perot, we have three issues here:
.NET Runtime Platform stalled- Orleans uses a timer to try to assert whether the underlying .net framework is stalled, a 1 second delay is considered a stall. Which sometimes means that GC is running. In your case, I'm not sure, as the warning says "We are now using total of 17MB memory. gc=273, 56, 4" (@sergeybykov might know more). And as you have noted, you haven't experienced any poor user experience. Are you running the silos on VMs ? Sometimes the clock on a VM runs super-fast or super-slow. Try to look at the logs of a silo that gets no traffic. If the .NET Runtime is actually stalled that frequently, some other evil is going on which probably doesn't stem from Orleans.ConnectionLossException- An error that occurs when the ZK server fails to respond in a timely manner. In Orleans, I've hardcoded the timeout to 2 seconds. I could have made it configurable, but I've found that it's never the problem, just a symptom of a real problem. i.e. network issues or ZK server configuration issues. I suggest:- Make sure the zookeeper cluster is continuously accessible from the silo machines.
- You're running with default ZK server configuration. Which includes a 10 concurrent connections per IP. It's alarming that it isn't enough, since Orleans should have only one connection to the ZK cluster at any give time. You can try setting it to
maxClientCnxns=0in all ZK serverszoo.cfgfile, which disables this limit. Before doing that, you can consult the ZK servers logs at the times Orleans got the exception. If it's a server issue, you'd probably find some helpful correlated error there.
- IIS weird exception - Well, I wrote that bit of code, so I think you've found a bug. It seems that the process doesn't have the credentials to create the log file and I haven't taken that into consideration. An easy quick fix for this is to add
ZooKeeper.LogToFile = false;before starting the silo. Or adding the necessary credentials for the app pool to create the file. cool bug :smile:
Keep me posted.
peter-perot
I have a strange issue with Zookeeper membership provider. We have different production deployments on different servers for different clients, but the issue is all the same. At some point in time (here the trouble starts at 19:31 GMT when exceptions are caught) silos start logging the following:
On the client side (a WebAPI running with Katana/Owin) the following is logged:
At first I don't know if the "stalled" warnings have something to do with this issue. I have configured garbage collection according to the manuals:
Concerning Zookeeper it has been deployed out of the box with no further configuration (only the log path has been adjusted).
It seems that these exceptions have no real impact on the stability experienced by the user unless we install our WebAPI in IIS. In this case the AppPool shuts down after some hours and the event viewer shows some weird exception from org.apache namespace:
Any idea what is going on here?