search

dotnet / orleans

Cloud Native application framework for .NET
https://docs.microsoft.com/dotnet/orleans
MIT License
10.1k stars 2.03k forks source link

Error on type 'MessageCenter'. The requested address is not valid in its context #4524

Closed ifle closed 6 years ago

ifle commented 6 years ago

After moving to the V2.0 from v2-rc3, several developers get the "The requested address is not valid in its context" error. At several devs all works as expected. We use the ado.net clustering. From this message is not clear where is a problem

2018-04-27 16:30:33.777 +03:00 [WRN] No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
2018-04-27 16:30:33.848 +03:00 [INF] Starting OrleansTaskScheduler with 8 Max Active application Threads and 1 system thread.
2018-04-27 16:30:34.455 +03:00 [INF] Silo starting with GC settings: ServerGC=false GCLatencyMode=Interactive
2018-04-27 16:30:34.455 +03:00 [WRN] Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
2018-04-27 16:30:34.458 +03:00 [WRN] Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
2018-04-27 16:30:34.464 +03:00 [INF] -------------- Initializing silo on host RIfle MachineName RIfle at 169.254.45.213:11111, gen 262531833 --------------
2018-04-27 16:30:34.467 +03:00 [INF] Starting silo Silo_e5667
2018-04-27 16:31:47.657 +03:00 [ERR] An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = SiloWrapper (ReflectionActivator), Services = [Orleans.Hosting.ISiloHost], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = Silo (ReflectionActivator), Services = [Orleans.Runtime.Silo], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, System.IServiceProvider)' on type 'Silo'. ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = MessageCenter (ReflectionActivator), Services = [Orleans.Runtime.Messaging.MessageCenter], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.EndpointOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.SiloMessagingOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.NetworkingOptions], Orleans.Serialization.SerializationManager, Orleans.Runtime.MessageFactory, Orleans.Factory`2[Orleans.Runtime.Messaging.MessageCenter,Orleans.Runtime.Messaging.Gateway], Orleans.Runtime.ExecutorService, Microsoft.Extensions.Logging.ILoggerFactory)' on type 'MessageCenter'. ---> The requested address is not valid in its context (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) (See inner exception for details.)
Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. 
See the inner exception for details. Registration: Activator = SiloWrapper (ReflectionActivator), 
Services = [Orleans.Hosting.ISiloHost], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, 
Ownership = OwnedByLifetimeScope ---> An error occurred during the activation of a particular registration. 
See the inner exception for details. Registration: Activator = Silo (ReflectionActivator), Services = [Orleans.Runtime.Silo], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> 
An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, System.IServiceProvider)' on type 'Silo'. 
---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = MessageCenter (ReflectionActivator), Services = [Orleans.Runtime.Messaging.MessageCenter], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope 
---> An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.EndpointOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.SiloMessagingOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.NetworkingOptions], Orleans.Serialization.SerializationManager, Orleans.Runtime.MessageFactory, Orleans.Factory`2[Orleans.Runtime.Messaging.MessageCenter,Orleans.Runtime.Messaging.Gateway], Orleans.Runtime.ExecutorService, Microsoft.Extensions.Logging.ILoggerFactory)' 
Error on type 'MessageCenter'. 
---> The requested address is not valid in its context (See inner exception for details.) 
(See inner exception for details.) (See inner exception for details.) (See inner exception for details.) 
(See inner exception for details.) 
---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = Silo (ReflectionActivator), Services = [Orleans.Runtime.Silo], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, System.IServiceProvider)' on type 'Silo'. ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = MessageCenter (ReflectionActivator), Services = [Orleans.Runtime.Messaging.MessageCenter], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.EndpointOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.SiloMessagingOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.NetworkingOptions], Orleans.Serialization.SerializationManager, Orleans.Runtime.MessageFactory, Orleans.Factory`2[Orleans.Runtime.Messaging.MessageCenter,Orleans.Runtime.Messaging.Gateway], Orleans.Runtime.ExecutorService, Microsoft.Extensions.Logging.ILoggerFactory)' on type 'MessageCenter'. ---> The requested address is not valid in its context (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) ---> Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, System.IServiceProvider)' on type 'Silo'. ---> An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = MessageCenter (ReflectionActivator), Services = [Orleans.Runtime.Messaging.MessageCenter], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.EndpointOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.SiloMessagingOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.NetworkingOptions], Orleans.Serialization.SerializationManager, Orleans.Runtime.MessageFactory, Orleans.Factory`2[Orleans.Runtime.Messaging.MessageCenter,Orleans.Runtime.Messaging.Gateway], Orleans.Runtime.ExecutorService, Microsoft.Extensions.Logging.ILoggerFactory)' on type 'MessageCenter'. ---> The requested address is not valid in its context (See inner exception for details.) (See inner exception for details.) (See inner exception for details.) ---> Autofac.Core.DependencyResolutionException: An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = MessageCenter (ReflectionActivator), Services = [Orleans.Runtime.Messaging.MessageCenter], Lifetime = Autofac.Core.Lifetime.RootScopeLifetime, Sharing = Shared, Ownership = OwnedByLifetimeScope ---> An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.EndpointOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.SiloMessagingOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.NetworkingOptions], Orleans.Serialization.SerializationManager, Orleans.Runtime.MessageFactory, Orleans.Factory`2[Orleans.Runtime.Messaging.MessageCenter,Orleans.Runtime.Messaging.Gateway], Orleans.Runtime.ExecutorService, Microsoft.Extensions.Logging.ILoggerFactory)' on type 'MessageCenter'. ---> The requested address is not valid in its context (See inner exception for details.) (See inner exception for details.) ---> Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor 'Void .ctor(Orleans.Runtime.ILocalSiloDetails, Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.EndpointOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.SiloMessagingOptions], Microsoft.Extensions.Options.IOptions`1[Orleans.Configuration.NetworkingOptions], Orleans.Serialization.SerializationManager, Orleans.Runtime.MessageFactory, Orleans.Factory`2[Orleans.Runtime.Messaging.MessageCenter,Orleans.Runtime.Messaging.Gateway], Orleans.Runtime.ExecutorService, Microsoft.Extensions.Logging.ILoggerFactory)' on type 'MessageCenter'. ---> The requested address is not valid in its context (See inner exception for details.) ---> System.Net.Sockets.SocketException: The requested address is not valid in its context
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Orleans.Runtime.SocketManager.GetAcceptingSocketForEndpoint(IPEndPoint address)
   at Orleans.Runtime.Messaging.IncomingMessageAcceptor..ctor(MessageCenter msgCtr, IPEndPoint here, SocketDirection socketDirection, MessageFactory messageFactory, SerializationManager serializationManager, ExecutorService executorService, ILoggerFactory loggerFactory)
   at Orleans.Runtime.Messaging.MessageCenter.Initialize(IOptions`1 endpointOptions, IOptions`1 messagingOptions, IOptions`1 networkingOptions)
   at Orleans.Runtime.Messaging.MessageCenter..ctor(ILocalSiloDetails siloDetails, IOptions`1 endpointOptions, IOptions`1 messagingOptions, IOptions`1 networkingOptions, SerializationManager serializationManager, MessageFactory messageFactory, Factory`2 gatewayFactory, ExecutorService executorService, ILoggerFactory loggerFactory)
   at lambda_method(Closure , Object[] )
   at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
   --- End of inner exception stack trace ---
   at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
   at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   --- End of inner exception stack trace ---
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Orleans.Runtime.Silo..ctor(ILocalSiloDetails siloDetails, IServiceProvider services)
   at lambda_method(Closure , Object[] )
   at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
   --- End of inner exception stack trace ---
   at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
   at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   --- End of inner exception stack trace ---
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
   at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   --- End of inner exception stack trace ---
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters)
   at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at PApp.Server.Host.Program.BuildSilo()
   at PApp.Server.Host.Program.Main(String[] args)
2018-04-27 16:31:47.724 +03:00 [INF] Silo Stopping...
2018-04-27 16:31:47.725 +03:00 [INF] Silo Stopped
2018-04-27 16:32:45.710 +03:00 [WRN] No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
2018-04-27 16:32:45.789 +03:00 [INF] Starting OrleansTaskScheduler with 8 Max Active application Threads and 1 system thread.
2018-04-27 16:32:46.290 +03:00 [INF] Silo starting with GC settings: ServerGC=false GCLatencyMode=Interactive
2018-04-27 16:32:46.292 +03:00 [WRN] Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
2018-04-27 16:32:46.293 +03:00 [WRN] Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
2018-04-27 16:32:46.299 +03:00 [INF] -------------- Initializing silo on host RIfle MachineName RIfle at 169.254.45.213:11111, gen 262531965 --------------
2018-04-27 16:32:46.301 +03:00 [INF] Starting silo Silo_e8a96
benjaminpetit commented 6 years ago

Could you post the logged options at startup?

ifle commented 6 years ago

We use serilog

services.AddLogging(builder => builder.ConfigureLogging(appSettings, appSettings.ApplicationInsightsServer, "server"));

public static void ConfigureLogging(this ILoggingBuilder loggingBuilder, AppSettings settings, ApplicationInsightsSettings applicationInsightsSettings)
{
    ConfigureLogger(settings, applicationInsightsSettings);
    loggingBuilder.AddSerilog(dispose: true);
}

public static void ConfigureLogger(AppSettings settings, ApplicationInsightsSettings applicationInsightsSettings)
{
    var loggerConfiguration = new LoggerConfiguration()
        .Enrich.FromLogContext();

    if (settings.WriteToConsole)
    {
        loggerConfiguration.WriteTo.Console(LogEventLevel.Information);
    }

    if (applicationInsightsSettings?.Enabled == true)
    {
        loggerConfiguration.WriteTo.ApplicationInsightsTraces(applicationInsightsSettings.InstrumentationKey, applicationInsightsSettings.MinimalLogLevel);
    }

    Log.Logger = loggerConfiguration.CreateLogger();        
}
benjaminpetit commented 6 years ago

When Orleans starts, it will print the values of the options used. Could you copy them here? I suspect that there is an issue with the IP that Orleans tries to use. It may work on some machine but not on other depending on their network configuration.

ifle commented 6 years ago 
2018-04-27 16:30:33.777 +03:00 [WRN] No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
2018-04-27 16:30:33.848 +03:00 [INF] Starting OrleansTaskScheduler with 8 Max Active application Threads and 1 system thread.
2018-04-27 16:30:34.455 +03:00 [INF] Silo starting with GC settings: ServerGC=false GCLatencyMode=Interactive
2018-04-27 16:30:34.455 +03:00 [WRN] Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
2018-04-27 16:30:34.458 +03:00 [WRN] Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
2018-04-27 16:30:34.464 +03:00 [INF] -------------- Initializing silo on host RIfle MachineName RIfle at 169.254.45.213:11111, gen 262531833 --------------
2018-04-27 16:30:34.467 +03:00 [INF] Starting silo Silo_e5667
ifle commented 6 years ago

What is interesting, that at home I tested on PC and on laptop. On PC I have not error, when on laptop I have it

benjaminpetit commented 6 years ago

The interesting logs should happen after the "starting silo", something like this:

[2018-04-24 21:22:29.597 GMT 15 Information 0   Orleans.Runtime.SiloOptionsLogger]  Configuration Orleans.Configuration.EndpointOptions: 
AdvertisedIPAddress: 127.0.0.1
SiloPort: 27450
GatewayPort: 48095
SiloListeningEndpoint: 
GatewayListeningEndpoint:

What is interesting, that at home I tested on PC and on leptop. On PC I have not error, when on laptop I have it

I assume you have two network interface on your laptop? Could you try when you disable one?

ifle commented 6 years ago

Silo has been failed before writing this info I have hyper-V virtual ethernet. Disabling it, is not helped.

What is helped, the setting listenOnAnyHostAddress to true

siloHostBuilder.ConfigureEndpoints(11111, 30000, listenOnAnyHostAddress: true);

I used to default parameters that cause to this error

siloHostBuilder.ConfigureEndpoints(11111, 30000);
ifle commented 6 years ago

When I set listenOnAnyHostAddress to true in silo does not failed, but the client unable to connect it

benjaminpetit commented 6 years ago

That pretty much confirm my suspicions: when you set listenOnAnyHostAddress to true, Orleans will bind to the socket 0.0.0.0, but will publish another IP address in the membership table. Can you try to look at what IP is written in this table?

ifle commented 6 years ago

Yes this is IP is not localhost 127.0.0.1, like 169.254.xx.xx

ifle commented 6 years ago

Before the v2.0 we haven't this error

benjaminpetit commented 6 years ago

If I am not mistaken this is a link-local ipv4 address, right? What kind of network interface do you have on your machine?

ifle commented 6 years ago

Yes this is ipv4 address. I have wireless and cable adapters

ifle commented 6 years ago

Also hyper-v switch

ifle commented 6 years ago

Only devs with leptops have this error

benjaminpetit commented 6 years ago

To what interface this IP belongs to? It seems that it is a link-local ipv4 address, I am pretty sure it is useless. Can you try to force the IP address by configuring manually the AdvertisedIP in EndpointOptions?

benjaminpetit commented 6 years ago

By default, Orleans will try to take the first non-localhost IP available on the machine that is hosting the silo. If there is more than one, we sort them and we take the first one. I am pretty sure here it fails because here you have a ip4 link-local address https://en.wikipedia.org/wiki/Link-local_address

We could add some filtering in our algorithm that choose an IP address, but I don't think having a link-local address is something expected...

ifle commented 6 years ago

How that works prior of 2.0 release?

veikkoeeva commented 6 years ago

@ifle I think it's been luck. This seems similar as https://github.com/dotnet/orleans/issues/2538#issuecomment-270174447. I.e. I've had this kind of problems with virtual switch with static routes. Related, I think, is https://github.com/dotnet/orleans/issues/2054 and other linked from these issues.

lfzm commented 6 years ago

@benjaminpetit Is there any solution? We also encountered this problem