Closed jptaylor226 closed 9 years ago
I spot the following
2015-04-06T20:03:47 PID[38708] Error Error opening trace file Client-2015-04-06-20.03.46.894Z.log -- Using DefaultTraceListener instead -- Exception=System.UnauthorizedAccessException: Access to the path 'D:\Windows\system32\Client-2015-04-06-20.03.46.894Z.log' is denied. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
Could that trigger the first exception and consequently disallow connecting?
Good catch, @veikkoeeva!
@jptaylor226 Try to specify a full path in TraceToFile.
@jptaylor226, @sergeybykov A quick one before sauna. I wonder if this has also something to with that the API is on a website and the silos in a Worker Role, so in a different subnet and no possibility to have even a VNET in between (I recall one couldn't attach web sites to a VNET, but that might have changed already). I remember there's a lengthy discussion about that somewhere in Codeplex, but I could only find this. In any event, I slap it here in case it's useful on itself.
I made an effort to remove the Trace configurations from both the client and worker role and redeployed to azure, but the behavior remains unchanged-- still getting the "Target silo is unavailable" message. Based on the codeplex discussion, it seems the error is mostly in my mental model. If I understand correctly now, the worker role needs to be configured as a silo client, and one of many appropriate designs might be to have the worker role subscribe to the an azure queue or topic, and then send messages to grains.
@jptaylor226 Are you deploying silos and client as part of the same hosted service? You mentioned Azure web site, so I suspect that's not the case. This is not supported due to the lack of direct TCP connectivity from outside of a hosted service. You would need a virtual network configured across them, but I'm not sure if somebody tried that.
If I understand correctly now, the worker role needs to be configured as a silo client
In a typical topology, a worker role is used for hosting silos and a web role (IIS) or another worker role (non-IIS HTTP server) hosting clients (frontends). Only the frontends need to be configured as silo clients.
Thanks for the quick help, Sergey-- I'm making the transition from local test environment to closer to production, so I have some work to do. Moving on to making another worker in the same service at this point, will post an update on this thread.
OK-- so went ahead and used this walkthrough (http://www.asp.net/web-api/overview/hosting-aspnet-web-api/host-aspnet-web-api-in-an-azure-worker-role) to make a Web Api role in the azure cloud service, and things seem to work.
Great!
Using vnet with silos and web clients in separate deployments worked for me.
Project Orleans gurus,
My group at work is considering getting heavily invested with the implementation of our next gen system, but I keep running into the seemingly common exception: [System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansException: Target silo is unavailable]
I'm currently running 1.0.5 (glad to be rid of the SDK for codegen). The silos are hosted in a worker role of an azure service; the client is a Web Api, hosted as an azure web site. It looks like the client resolves the location of the silos, but is unable to connect.
My current set of (sanitized) configs are as follows: Azure Service: ServiceConfiguration.Cloud.cscfg:
<?xml version="1.0" encoding="utf-8"?>
<ServiceConfiguration serviceName="HanoiCloud" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="4" osVersion="*" schemaVersion="2014-06.2.4">
<Role name="HanoiWorker">
<Instances count="2" />
<ConfigurationSettings>
<Setting name="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName=[foo];AccountKey=[foo Key]" />
</ConfigurationSettings>
</Role>
</ServiceConfiguration>
ServiceDefinition.csdef:
<?xml version="1.0" encoding="utf-8"?>
<ServiceDefinition name="HanoiCloud" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2014-06.2.4">
<WorkerRole name="HanoiWorker" vmsize="Medium">
<ConfigurationSettings>
<Setting name="DataConnectionString" />
</ConfigurationSettings>
<LocalResources>
<LocalStorage name="LocalStoreDirectory" cleanOnRoleRecycle="false" />
</LocalResources>
<Endpoints>
<InternalEndpoint name="OrleansSiloEndpoint" protocol="tcp" port="11111" />
<InternalEndpoint name="OrleansProxyEndpoint" protocol="tcp" port="30000" />
</Endpoints>
</WorkerRole>
</ServiceDefinition>
Worker Role: OrleansConfiguration.xml (tagged with 'Copy always')
<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
<Globals>
<Liveness LivenessType ="AzureTable" />
<SystemStore DeploymentId="51d7..." DataConnectionString="DefaultEndpointsProtocol=https;AccountName=[foo];AccountKey=[foo Key]"/>
</Globals>
<Defaults>
<Networking Address="" Port="11111" />
<ProxyingGateway Address="" Port="30000" />
<Tracing DefaultTraceLevel="Info" TraceToConsole="true" TraceToFile="{0}-{1}.log" />
</Defaults>
</OrleansConfiguration>
Client code-- which is a Web Api, deployed in Azure:
DevTestAzureConfiguration.xml
<?xml version="1.0" encoding="utf-8" ?>
<!--
This is a sample client configuration file.
For a detailed reference, see "Orleans Configuration Reference.html".
-->
<ClientConfiguration xmlns="urn:orleans">
<SystemStore DeploymentId="51d7..."
DataConnectionString="DefaultEndpointsProtocol=https;AccountName=[foo];AccountKey=[foo Key]" />
<Tracing DefaultTraceLevel="Verbose" TraceToConsole="true" TraceToFile="{0}-{1}.log">
<TraceLevelOverride LogPrefix="HanoiClientApi" TraceLevel="Verbose" />
</Tracing>
</ClientConfiguration>
HGController.cs sample:
[HttpGet]
[Route("moves")]
public string GetMoves()
{
var result = "Hello world.";
try
{
if (!Orleans.GrainClient.IsInitialized)
{
var file = HostingEnvironment.MapPath("~") + "/DevTestAzureConfiguration.xml";
Orleans.GrainClient.Initialize(file);
}
var monk = MonkFactory.GetGrain(Guid.NewGuid());
result = result + monk.GetMoves().Result;
}
catch (Exception ex)
{
result = ex.ToString();
Trace.Write(ex);
}
return result;
}
Full exception text when attempt to call this code from a browser: System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansException: Target silo is unavailable --- End of inner exception stack trace --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.GrainReference.d0d 0
1.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task
1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task1.get_Result() at Orleans.OutsideRuntimeClient.StartInternal() at Orleans.OutsideRuntimeClient.Start() at Orleans.GrainClient.DoInternalInitialize(ClientConfiguration config, OutsideRuntimeClient runtimeClient) at Orleans.GrainClient.<>c__DisplayClass1.<InternalInitialize>b__0(Object state) --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task
1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task`1.get_Result() at Orleans.GrainClient.InternalInitialize(ClientConfiguration config, OutsideRuntimeClient runtimeClient) at Orleans.GrainClient.Initialize(FileInfo configFile) at HanoiWebApi.Controllers.HGController.GetMoves() ---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansException: Target silo is unavailable --- End of inner exception stack trace --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.GrainReference.1.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task
1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task1.get_Result() at Orleans.OutsideRuntimeClient.StartInternal() at Orleans.OutsideRuntimeClient.Start() at Orleans.GrainClient.DoInternalInitialize(ClientConfiguration config, OutsideRuntimeClient runtimeClient) at Orleans.GrainClient.<>c__DisplayClass1.<InternalInitialize>b__0(Object state) ---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> Orleans.Runtime.OrleansException: Target silo is unavailable --- End of inner exception stack trace --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.GrainReference.<InvokeMethodAsync>d__0
1.MoveNext() ---> (Inner Exception #0) Orleans.Runtime.OrleansException: Target silo is unavailable<--- <--- <---A typical log from the client includes: 2015-04-06T20:03:47 PID[38708] Error Error opening trace file Client-2015-04-06-20.03.46.894Z.log -- Using DefaultTraceListener instead -- Exception=System.UnauthorizedAccessException: Access to the path 'D:\Windows\system32\Client-2015-04-06-20.03.46.894Z.log' is denied. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append) at System.IO.FileInfo.CreateText() at Orleans.Runtime.LogWriterToFile..ctor(FileInfo logFile) at Orleans.Runtime.TraceLogger.Initialize(ITraceConfiguration config, Boolean configChange) 2015-04-06T20:03:47 PID[38708] Information [2015-04-06 20:03:47.644 GMT 46 INFO 100000 AssemblyLoader.Client ] Recursively searching for assemblies in D:\local\Temporary ASP.NET Files\root\3d2b4354\66ec5cba\assembly\dl3\bbd4e5d8\f7980aad_c868d001...
2015-04-06T20:03:47 PID[38708] Information [2015-04-06 20:03:47.676 GMT 46 INFO 100000 AssemblyLoader.Client ] User assembly ignored: D:\local\Temporary ASP.NET Files\root\3d2b4354\66ec5cba\assembly\dl3\bbd4e5d8\f7980aad_c868d001\Orleans.DLL * Assembly filename is excluded.
2015-04-06T20:03:47 PID[38708] Information [2015-04-06 20:03:47.676 GMT 46 INFO 100000 AssemblyLoader.Client ] 0 assemblies loaded.
2015-04-06T20:03:47 PID[38708] Information [2015-04-06 20:03:47.769 GMT 46 INFO 100313 OutsideRuntimeClient ] ---------- Initializing OutsideRuntimeClient on RD00155DFC427E at 100.79.212.22 Client Id = cli/a18ba188 ----------
2015-04-06T20:03:47 PID[38708] Information [2015-04-06 20:03:47.801 GMT 46 INFO 100314 OutsideRuntimeClient ] ---------- Starting OutsideRuntimeClient with runtime Version='1.0.0.0 IsRelease.' Config= Platform version info: .NET version: 4.0.30319.36213 Is .NET 4.5=True OS version: Microsoft Windows NT 6.2.9200.0 GC Type=Server GCLatencyMode=Batch Host: RD00155DFC427E Processor Count: 8 Client Configuration: Config File Name: D:\home\site\wwwroot\DevTestAzureConfiguration.xml Start time: 2015-04-06 20:03:47.785 GMT Gateway Provider: AzureTable Gateways: Unspecified Preferred Gateway Index: -1 GatewayListRefreshPeriod: 00:01:00 Azure: DeploymentId: 51d70ba47f1142ffb1d5bf786350dd7f DataConnectionString: DefaultEndpointsProtocol=https;AccountName=skaterwormhanoi;AccountKey=<--SNIP--> Preferred Address Family: InterNetwork DNS Host Name: RD00155DFC427E Client Name: Client Tracing: Default Trace Level: Info TraceLevelOverrides: HanoiClientApi ==> Info Trace to Console: True Trace File Name: D:\Windows\system32\Client-2015-04-06-20.03.46.894Z.log Write Messaging Traces: False LargeMessageWarningThreshold: 85000 PropagateActivityId: False BulkMessageLimit: 5 Statistics: MetricsTableWriteInterval: 00:00:30 PerfCounterWriteInterval: -00:00:00.0010000 LogWriteInterval: 00:05:00 WriteLogStatisticsToTable: True StatisticsCollectionLevel: Info Messaging: Response timeout: 00:00:30 Maximum resend count: 0 Resend On Timeout: False Maximum Socket Age: 10675199.02:48:05.4775807 Drop Expired Messages: True Client Sender Buckets: 8192 Use standard (.NET) serializer: False [NOTE: This *MUST match the setting on the server or nothing will work!] Buffer Pool Buffer Size: 4096 Buffer Pool Max Size: 10000 Buffer Pool Preallocation Size: 250 Use Message Batching: False Max Message Batching Size: 10 Providers: No providers configured.
2015-04-06T20:03:48 PID[38708] Information [2015-04-06 20:03:48.457 GMT 7 INFO 100801 OrleansSiloInstanceManager ] Attached to Azure storage table OrleansSiloInstances
2015-04-06T20:03:48 PID[38708] Information [2015-04-06 20:03:48.941 GMT 7 INFO 100836 OrleansSiloInstanceManager ] Initialized schema for Azure storage table OrleansSiloInstances 2015-04-06T20:03:48 PID[38708] Information [2015-04-06 20:03:48.957 GMT 46 INFO 100277 OrleansSiloInstanceManager ] Searching for active gateway silos for deployment 51d70ba47f1142ffb1d5bf786350dd7f ...
2015-04-06T20:03:49 PID[38708] Information [2015-04-06 20:03:49.473 GMT 46 INFO 100278 OrleansSiloInstanceManager ] Found 2 active Gateway Silos.
2015-04-06T20:03:49 PID[38708] Information [2015-04-06 20:03:49.473 GMT 46 INFO 101309 Messaging.GatewayManager ] Found 2 knownGateways from Gateway listProvider [gwy.tcp://10.79.160.121:30000/165789566, gwy.tcp://10.79.220.91:30000/165789786]
2015-04-06T20:03:49 PID[38708] Warning [2015-04-06 20:03:49.504 GMT 46 WARNING 100719 RuntimeStatisticsGroup 100.79.212.22:0] Error initializing CPU & Memory perf counters - you need to repair Windows perf counter config on this machine with 'lodctr /r' command 2015-04-06T20:03:49 PID[38708] Warning [2015-04-06 20:03:49.535 GMT 46 WARNING 100708 RuntimeStatisticsGroup 100.79.212.22:0] CPU & Memory perf counters did not initialize correctly - try repairing Windows perf counter config on this machine with 'lodctr /r' command
2015-04-06T20:03:49 PID[38708] Information [2015-04-06 20:03:49.551 GMT 48 INFO 100801 ClientMetricsTableDataManager 100.79.212.22:0] Attached to Azure storage table OrleansClientMetrics
2015-04-06T20:03:49 PID[38708] Information [2015-04-06 20:03:49.879 GMT 48 INFO 100836 ClientMetricsTableDataManager 100.79.212.22:0] Initialized schema for Azure storage table OrleansClientMetrics 2015-04-06T20:03:49 PID[38708] Information [2015-04-06 20:03:49.879 GMT 48 INFO 100714 ClientTableStatistics 100.79.212.22:0] Starting performance metrics reporting with reportFrequency=00:00:30
2015-04-06T20:03:49 PID[38708] Information [2015-04-06 20:03:49.910 GMT 47 INFO 100801 StatsTableDataManager 100.79.212.22:0] Attached to Azure storage table OrleansClientStatistics 2015-04-06T20:03:49 PID[38708] Information [2015-04-06 20:03:49.957 GMT 47 INFO 100836 StatsTableDataManager 100.79.212.22:0] Initialized schema for Azure storage table OrleansClientStatistics
2015-04-06T20:03:50 PID[38708] Warning [2015-04-06 20:03:50.035 GMT 46 WARNING 100178 Messaging.GatewayConnection.GatewayClientSender_gwy.tcp://10.79.160.121:30000/165789566/1 100.79.212.22:0] Unable to connect to gateway at address gwy.tcp://10.79.160.121:30000/165789566 on trial 0. 2015-04-06T20:03:50 PID[38708] Warning [2015-04-06 20:03:50.051 GMT 46 WARNING 100913 Messaging.GatewayConnection.GatewayClientSender_gwy.tcp://10.79.160.121:30000/165789566/1 100.79.212.22:0] Marking gateway at address gwy.tcp://10.79.160.121:30000/165789566 as Disconnected
2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.051 GMT 14 INFO 100277 OrleansSiloInstanceManager 100.79.212.22:0] Searching for active gateway silos for deployment 51d70ba47f1142ffb1d5bf786350dd7f ...
2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.066 GMT 14 INFO 100278 OrleansSiloInstanceManager 100.79.212.22:0] Found 2 active Gateway Silos.
2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.082 GMT 14 INFO 101309 Messaging.GatewayManager 100.79.212.22:0] Refreshed the live GateWay list. Found 2 gateways from Gateway listProvider: [gwy.tcp://10.79.160.121:30000/165789566, gwy.tcp://10.79.220.91:30000/165789786]. Picked only known live out of them. Now has 2 live Gateways: [gwy.tcp://10.79.160.121:30000/165789566, gwy.tcp://10.79.220.91:30000/165789786]. Previous refresh time was = 4/6/2015 8:03:49 PM 2015-04-06T20:03:50 PID[38708] Warning [2015-04-06 20:03:50.160 GMT 46 WARNING 100178 Messaging.GatewayConnection.GatewayClientSender_gwy.tcp://10.79.160.121:30000/165789566/1 100.79.212.22:0] Unable to connect to gateway at address gwy.tcp://10.79.160.121:30000/165789566 on trial 1. 2015-04-06T20:03:50 PID[38708] Warning [2015-04-06 20:03:50.160 GMT 46 WARNING 100913 Messaging.GatewayConnection.GatewayClientSender_gwy.tcp://10.79.160.121:30000/165789566/1 100.79.212.22:0] Marking gateway at address gwy.tcp://10.79.160.121:30000/165789566 as Disconnected
2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.160 GMT 42 INFO 100277 OrleansSiloInstanceManager 100.79.212.22:0] Searching for active gateway silos for deployment 51d70ba47f1142ffb1d5bf786350dd7f ...
2015-04-06T20:03:50 PID[38708] Warning [2015-04-06 20:03:50.160 GMT 46 WARNING 100912 Messaging.GatewayConnection.GatewayClientSender_gwy.tcp://10.79.160.121:30000/165789566/1 100.79.212.22:0] Marking gateway at address gwy.tcp://10.79.160.121:30000/165789566 as Dead in my client local gateway list. 2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.176 GMT 42 INFO 100278 OrleansSiloInstanceManager 100.79.212.22:0] Found 2 active Gateway Silos.
2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.176 GMT 42 INFO 101309 Messaging.GatewayManager 100.79.212.22:0] Refreshed the live GateWay list. Found 2 gateways from Gateway listProvider: [gwy.tcp://10.79.160.121:30000/165789566, gwy.tcp://10.79.220.91:30000/165789786]. Picked only known live out of them. Now has 1 live Gateways: [gwy.tcp://10.79.220.91:30000/165789786]. Previous refresh time was = 4/6/2015 8:03:50 PM
2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.191 GMT 46 INFO 100000 OutsideRuntimeClient 100.79.212.22:0] OutsideRuntimeClient.Reset(): client Id *cli/a18ba188
2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.254 GMT 46 INFO 100704 ClientLogStatistics 100.79.212.22:0] Statistics: ^^^ App.Requests.Latency.Average.Millis=158.000 App.Requests.LatencyHistogram.Millis=[104.8576:209.7151]=1, App.Requests.TimedOut.Current=0 App.Requests.Total.Requests.Current=1 Client.ConnectedGatewayCount=0 Messaging.Expired.AtDispatch.Current=0 Messaging.Expired.AtInvoke.Current=0 Messaging.Expired.AtReceive.Current=0 Messaging.Expired.AtRespond.Current=0 Messaging.Expired.AtSend.Current=0 Messaging.Received.BatchSize.PerSocketDirection.ClientToGateway=0.000 Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.ClientToGateway= Messaging.Received.Bytes.Header.Current=0 Messaging.Received.Bytes.Total.Current=0 Messaging.Received.Direction.OneWay.Current=0 Messaging.Received.Direction.Request.Current=0 Messaging.Received.Direction.Response.Current=0 Messaging.Received.Messages.Total.Current=0 Messaging.Received.MessageSizeHistogram.Bytes= Messaging.Rejected.Request.Current=1 Messaging.Rerouted.OneWay.Current=0 Messaging.Rerouted.Request.Current=0 Messaging.Rerouted.Response.Current=0 Messaging.Sent.BatchSize.PerSocketDirection.ClientToGateway=0.000 Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.ClientToGateway= Messaging.Sent.Bytes.Header.Current=0 Messaging.Sent.Bytes.Total.Current=0 Messaging.Sent.Direction.OneWay.Current=0 Messaging.Sent.Direction.Request.Current=0 Messaging.Sent.Direction.Response.Current=0 Messaging.Sent.Messages.Total.Current=0 Messaging.Sent.MessageSizeHistogram.Bytes= Networking.Sockets.ClientToGateway.Duplex.Closed.Current=2 Networking.Sockets.ClientToGateway.Duplex.Opened.Current=0 Serialization.BufferPool.AllocatedBuffers.Current=250 Serialization.BufferPool.BuffersInPool=250 Serialization.BufferPool.CheckedInBuffers.Current=250 Serialization.BufferPool.CheckedInDroppedBuffers.Current=0 Serialization.BufferPool.CheckedOutBuffers.Current=250 Serialization.BufferPool.InUse.AllocatedAndNotInPool_Buffers=0 Serialization.BufferPool.InUse.CheckedOutAndNotCheckedIn_Buffers=0 Storage.Azure.Table.ServerBusy.Current=0
2015-04-06T20:03:50 PID[38708] Information [2015-04-06 20:03:50.426 GMT 46 INFO 100000 OutsideRuntimeClient 100.79.212.22:0] OutsideRuntimeClient.ConstructorReset(): client Id *cli/a18ba188