search

dotnet / orleans

Cloud Native application framework for .NET
https://docs.microsoft.com/dotnet/orleans
MIT License
10.04k stars 2.02k forks source link

AzureSilo.Start() fails #987

Closed dinoo closed 8 years ago

dinoo commented 8 years ago

Hey guys,

Today, I decided to pick up were I left off a few month ago. I thought, before I get started I should upgrade to 1.0.10, which seemed a good idea. However, when I start my worker role, it now gives me the exception you can see below. The project is using the latest version of Azure Storage library (6.1.0) and I have included below the OrleansConfiguration for my AzureSilo.

I checked my local storage (emulator) and It have a deployment registered in the OrleansSiloInstances table.

OrleansConfiguration

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <StorageProviders>
      <Provider Type="Orleans.Storage.MemoryStorage" Name="MemoryStore" />
      <Provider Type="Orleans.Storage.MemoryStorage" Name="PubSubStore" NumStorageGrains="1"/>
    </StorageProviders>
    <StreamProviders>
      <Provider Type="Orleans.Providers.Streams.SimpleMessageStream.SimpleMessageStreamProvider" Name="SMSProvider" FireAndForgetDelivery="true"/>
    </StreamProviders>
    <SeedNode Address="localhost" Port="11111" />
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11111" />
    <ProxyingGateway Address="localhost" Port="30000" />
    <Tracing DefaultTraceLevel="Info" TraceToConsole="true" TraceToFile="{0}-{1}.log" BulkMessageLimit="1000">
      <TraceLevelOverride LogPrefix="Application" TraceLevel="Info" />
    </Tracing>
    <Statistics MetricsTableWriteInterval="30s" PerfCounterWriteInterval="30s" LogWriteInterval="300s" WriteLogStatisticsToTable="true" StatisticsCollectionLevel="Info"/>
  </Defaults>
</OrleansConfiguration>

Exception

WaWorkerHost.exe Error: 0 : [2015-11-04 09:03:37.723 GMT     5  ERROR   100439  Silo    127.0.0.1:11111]    !!!!!!!!!! Exception during Silo.Start  
Exc level 0: System.AggregateException: One or more errors occurred.
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(TimeSpan timeout)
   at Orleans.OrleansTaskExtentions.WaitWithThrow(Task task, TimeSpan timeout)
   at Orleans.Runtime.Silo.DoStart()
   at Orleans.Runtime.Silo.Start()
Exc level 1: System.MissingMethodException: Method not found: 'Microsoft.WindowsAzure.Storage.Table.TableOperation Microsoft.WindowsAzure.Storage.Table.TableOperation.Retrieve(System.String, System.String)'.
   at Orleans.AzureUtils.AzureTableDataManager`1.<ReadSingleTableEntryAsync>d__27.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Orleans.AzureUtils.AzureTableDataManager`1.ReadSingleTableEntryAsync(String partitionKey, String rowKey)
   at Orleans.AzureUtils.OrleansSiloInstanceManager.<TryCreateTableVersionEntryAsync>d__2b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Orleans.Runtime.MembershipService.AzureBasedMembershipTable.<InitializeMembershipTable>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClassa.<<QueueTask>b__8>d__c.MoveNext()
WaWorkerHost.exe Error: 0 : [2015-11-04 09:03:37.736 GMT     5  ERROR   100105  OrleansSiloHost 127.0.0.1:11111]    !!!!!!!!!! ERROR starting Orleans silo name=Services.SimulationEngineWorkerRole_IN_0 Exception=
Exc level 0: System.AggregateException: One or more errors occurred.
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(TimeSpan timeout)
   at Orleans.OrleansTaskExtentions.WaitWithThrow(Task task, TimeSpan timeout)
   at Orleans.Runtime.Silo.DoStart()
   at Orleans.Runtime.Silo.Start()
   at Orleans.Runtime.Host.SiloHost.StartOrleansSilo()
Exc level 1: System.MissingMethodException: Method not found: 'Microsoft.WindowsAzure.Storage.Table.TableOperation Microsoft.WindowsAzure.Storage.Table.TableOperation.Retrieve(System.String, System.String)'.
   at Orleans.AzureUtils.AzureTableDataManager`1.<ReadSingleTableEntryAsync>d__27.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Orleans.AzureUtils.AzureTableDataManager`1.ReadSingleTableEntryAsync(String partitionKey, String rowKey)
   at Orleans.AzureUtils.OrleansSiloInstanceManager.<TryCreateTableVersionEntryAsync>d__2b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Orleans.Runtime.MembershipService.AzureBasedMembershipTable.<InitializeMembershipTable>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClassa.<<QueueTask>b__8>d__c.MoveNext() 
Exc level 0: System.AggregateException: One or more errors occurred.
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(TimeSpan timeout)
   at Orleans.OrleansTaskExtentions.WaitWithThrow(Task task, TimeSpan timeout)
   at Orleans.Runtime.Silo.DoStart()
   at Orleans.Runtime.Silo.Start()
   at Orleans.Runtime.Host.SiloHost.StartOrleansSilo()
Exc level 1: System.MissingMethodException: Method not found: 'Microsoft.WindowsAzure.Storage.Table.TableOperation Microsoft.WindowsAzure.Storage.Table.TableOperation.Retrieve(System.String, System.String)'.
   at Orleans.AzureUtils.AzureTableDataManager`1.<ReadSingleTableEntryAsync>d__27.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Orleans.AzureUtils.AzureTableDataManager`1.ReadSingleTableEntryAsync(String partitionKey, String rowKey)
   at Orleans.AzureUtils.OrleansSiloInstanceManager.<TryCreateTableVersionEntryAsync>d__2b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Orleans.Runtime.MembershipService.AzureBasedMembershipTable.<InitializeMembershipTable>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClassa.<<QueueTask>b__8>d__c.MoveNext()
WaWorkerHost.exe Error: 0 : [2015-11-04 09:03:37.739 GMT     5  ERROR   100285  OrleansAzureSilo    127.0.0.1:11111]    !!!!!!!!!! Failed to start Orleans silo 'Services.SimulationEngineWorkerRole_IN_0' as a Secondary node.
gabikliot commented 8 years ago

We have not tested with Azure Storage library 6.1.0 and not using this version (we are using 4.2 https://github.com/dotnet/orleans/blob/master/src/OrleansAzureUtils/packages.config#L9). There might be some version mismatch.

dinoo commented 8 years ago

Do you know if 4.2 Storage library is compatible with the v4.2 Storage Emulator from 2.7.1 Azure SDK?

gabikliot commented 8 years ago

I do not. But others (@sergeybykov , @jthelin , @jason-bragg) might know.

dinoo commented 8 years ago

Resolved

Based on your comment @gabikliot, I decided to try and downgrade my WorkerRole project to use Microsoft.WindowsAzure.Storage 4.2.0.0 and Microsoft.WindowsAzure.Configuration 2.0.0.0 instead of the latest versions I used before. However, I have library that I include in the project that uses the latest versions of both libraries.

I tried adding the following to my app.config

<dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.ConfigurationManager" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.0.0" newVersion="2.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.Storage" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.2.0.0" />
      </dependentAssembly>

but Assembly Binding Log Viewer tells me

The operation failed.
Bind result: hr = 0x80131040. No description available.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable  <path>\Services\csx\Debug\roles\WorkerRoleA\base\x64\WaWorkerHost.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: DisplayName = Microsoft.WindowsAzure.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///<path>/Services/csx/Debug/roles/WorkerRoleA/approot
LOG: Initial PrivatePath = <path>\Services\csx\Debug\roles\WorkerRoleA\approot
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = RoleManager
Calling assembly : WorkerRoleA, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: <path>\Services\csx\Debug\roles\WorkerRoleA\approot\WorkerRoleA.dll.config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: Microsoft.WindowsAzure.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///<path>/Services/csx/Debug/roles/WorkerRoleA/approot/Microsoft.WindowsAzure.Configuration.DLL.
LOG: Assembly download was successful. Attempting setup of file: <path>\Services\csx\Debug\roles\WorkerRoleA\approot\Microsoft.WindowsAzure.Configuration.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: Microsoft.WindowsAzure.Configuration, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
WRN: Comparing the assembly name resulted in the mismatch: Major Version
ERR: The assembly reference did not match the assembly definition found.
ERR: Run-from-source setup phase failed with hr = 0x80131040.
ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

Can I somehow use the old ones for my WorkerRole while my included library uses the newer ones?

Update

I added the older versions of Azure libraries used by the AzureUtils package to my project and set both to "Copy always" and I configured my app.config as follows:

<dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.ConfigurationManager" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="3.0.0.0-3.0.0.0" newVersion="3.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.Storage" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="6.1.0.0-6.1.0.0" newVersion="6.1.0.0" />
      </dependentAssembly>
<dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.ConfigurationManager" publicKeyToken="31bf3856ad364e35" />
        <codeBase version="2.0.0.0" href="azure4.2\Microsoft.WindowsAzure.Configuration.dll" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.Storage" publicKeyToken="31bf3856ad364e35" />
        <codeBase version="4.2.0.0" href="azure4.2\Microsoft.WindowsAzure.Storage.dll" />
      </dependentAssembly>
gabikliot commented 8 years ago

This may help: http://dotnet.github.io/orleans/Deployment-and-Versioning-of-Dependencies

sergeybykov commented 8 years ago

@dinoo You don't have to go all the way down to 4.2. We are running in production with 5.0.

If there is indeed an issue with 6.1, we need to understand what it is and look at fixing it.

dinoo commented 8 years ago

Thanks for the help, guys.

Let me know if you think 6.1.0 should work normally with the a future version of Project Orleans.

galvesribeiro commented 8 years ago

I faced same exception here as well... I've started having issues on my current system due to this change. We are using ARM (Azure Resource Manager) to make some operations at storage and the ARM models are not available on v5.0.2 on Microsoft.WindowsAzure.Storage so, we must have a way to get things upgraded.

Since 6.0, Azure SDK with the introduction of ARM does not do any fancy operations anymore on the client (previously it has a different implementation to each service and it rely on old Service Management APIs which was based on Powershell). Now they created a fully http API with swagger on it which makes things less breakable IMHO. I believe once we move to 6+, this kind of condition is not easily match anymore. Even Azure Powershell CLI is now based on this same SDK. The idea behind this is avoid this kind of situation.

Will do a test with it and check what are the problems...

sergeybykov commented 8 years ago

This is being addressed by https://github.com/dotnet/orleans/pull/1024.