Aaronontheweb
commented
8 years ago can you post the configuration information for your web service also?
Closed voltcode closed 8 years ago
Aaronontheweb
commented
8 years ago can you post the configuration information for your web service also?
voltcode
commented
8 years ago It's basically the same, apart from port=0 and role=[webapp]
Analogously for [wcfservice]
Aaronontheweb
commented
8 years ago Recapping what we discussed in Gitter:
your issue is using port 0 for the services those services are going to rejoin on new ports so they look like different nodes and since the cluster thinks a node is missing it won't let new nodes join, because the old nodes have not been downed
voltcode
commented
8 years ago I changed ports to hardcoded ([webapp] to 6001 and wcfservice to [7001]) and I still experience the same erroneous outcome when [webapp] is restarted.
voltcode
commented
8 years ago New log from [mainservice]:
Configuration Result:
[Success] Name AkkaPoC.Grid.Service
[Success] ServiceName AkkaPoC.Grid.Service
Topshelf v3.3.154.0, .NET Framework v4.0.30319.34209
[INFO][2016-01-27 08:25:03][Thread 0001][remoting] Starting remoting
[INFO][2016-01-27 08:25:03][Thread 0001][remoting] Remoting started; listening on addresses : [akka.tcp://akkapocsystem@localhost:4053]
[INFO][2016-01-27 08:25:03][Thread 0001][remoting] Remoting now listens on addresses: [akka.tcp://akkapocsystem@localhost:4053]
[INFO][2016-01-27 08:25:03][Thread 0001][Cluster] Cluster Node [akka.tcp://akkapocsystem@localhost:4053] - Starting up...
The AkkaPoC.Grid.Service service is now running, press Control+C to exit.
[INFO][2016-01-27 08:25:03][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Node [akka.tcp://akkapocsystem@localhost:4053] is JOINING, roles [mainservice]
[INFO][2016-01-27 08:25:04][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:4053] to [Up]
[INFO][2016-01-27 08:25:27][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Node [akka.tcp://akkapocsystem@localhost:6001] is JOINING, roles [webapp]
[INFO][2016-01-27 08:25:27][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
!!!>>>HERE THE event is passed from mainservice to [webapp] CORRECTLY<<!!!
!!!>>> AFTERWARDS I KILL the [webapp] <<<!!!
[WARNING][2016-01-27 08:26:00][Thread 0010][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-2]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.OnReceive(Object message)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:26:00][Thread 0010][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-2] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.OnReceive(Object message)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:26:00][Thread 0007][akka://akkapocsystem/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fakkapocsystem%40127.0.0.1%3a65316-2] Message DisassociateUnderlying from akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-2/endpointWriter to akka://akkapocsystem/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fakkapocsystem%40127.0.0.1%3a65316-2 was not delivered. 1 dead letters encountered.
[INFO][2016-01-27 08:26:00][Thread 0007][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-2] Message DeathWatchNotification from akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-2 to akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-2 was not delivered. 2 dead letters encountered.
[INFO][2016-01-27 08:26:01][Thread 0007][akka://akkapocsystem/deadLetters] Message GossipStatus from akka://akkapocsystem/system/cluster/core/daemon to akka://akkapocsystem/deadLetters was not delivered. 3 dead letters encountered.
[INFO][2016-01-27 08:26:01][Thread 0006][akka://akkapocsystem/deadLetters] Message Heartbeat from akka://akkapocsystem/system/cluster/core/daemon/heartbeatSender to akka://akkapocsystem/deadLetters was not delivered. 4 dead letters encountered.
[INFO][2016-01-27 08:26:02][Thread 0024][akka://akkapocsystem/deadLetters] Message GossipStatus from akka://akkapocsystem/system/cluster/core/daemon to akka://akkapocsystem/deadLetters was not delivered. 5 dead letters encountered.
[INFO][2016-01-27 08:26:02][Thread 0024][akka://akkapocsystem/deadLetters] Message Heartbeat from akka://akkapocsystem/system/cluster/core/daemon/heartbeatSender to akka://akkapocsystem/deadLetters was not delivered. 6 dead letters encountered.
[INFO][2016-01-27 08:26:03][Thread 0024][akka://akkapocsystem/deadLetters] Message GossipStatus from akka://akkapocsystem/system/cluster/core/daemon to akka://akkapocsystem/deadLetters was not delivered. 7 dead letters encountered.
[INFO][2016-01-27 08:26:03][Thread 0021][akka://akkapocsystem/deadLetters] Message Heartbeat from akka://akkapocsystem/system/cluster/core/daemon/heartbeatSender to akka://akkapocsystem/deadLetters was not delivered. 8 dead letters encountered.
[INFO][2016-01-27 08:26:04][Thread 0021][akka://akkapocsystem/deadLetters] Message GossipStatus from akka://akkapocsystem/system/cluster/core/daemon to akka://akkapocsystem/deadLetters was not delivered. 9 dead letters encountered.
[WARNING][2016-01-27 08:26:06][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Cluster Node [akka.tcp://akkapocsystem@localhost:4053] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://akkapocsystem@localhost:6001, status = Up]
[WARNING][2016-01-27 08:26:08][Thread 0008][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-3/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:6001: Error [Invalid address: akka.tcp://akkapocsystem@localhost:6001] []
[WARNING][2016-01-27 08:26:08][Thread 0010][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:6001]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:6001] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Helios.Net.Connections.TcpConnection.Open()
--- End of inner exception stack trace ---
at Helios.Net.Connections.TcpConnection.Open()
at Akka.Remote.Transport.Helios.HeliosHelpers.Open()
at Akka.Remote.Transport.Helios.CommonHandlers.Open()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()
--- End of inner exception stack trace ---]
[ERROR][2016-01-27 08:26:08][Thread 0010][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-3] Invalid address: akka.tcp://akkapocsystem@localhost:6001
Cause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:6001 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Helios.Net.Connections.TcpConnection.Open()
--- End of inner exception stack trace ---
at Helios.Net.Connections.TcpConnection.Open()
at Akka.Remote.Transport.Helios.HeliosHelpers.Open()
at Akka.Remote.Transport.Helios.CommonHandlers.Open()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)
at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )
at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.OnReceive(Object message)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[WARNING][2016-01-27 08:26:15][Thread 0008][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-4/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:6001: Error [Invalid address: akka.tcp://akkapocsystem@localhost:6001] []
[WARNING][2016-01-27 08:26:15][Thread 0011][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:6001]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:6001] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Helios.Net.Connections.TcpConnection.Open()
--- End of inner exception stack trace ---
at Helios.Net.Connections.TcpConnection.Open()
at Akka.Remote.Transport.Helios.HeliosHelpers.Open()
at Akka.Remote.Transport.Helios.CommonHandlers.Open()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()
--- End of inner exception stack trace ---]
[ERROR][2016-01-27 08:26:15][Thread 0011][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-4] Invalid address: akka.tcp://akkapocsystem@localhost:6001
Cause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:6001 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Helios.Net.Connections.TcpConnection.Open()
--- End of inner exception stack trace ---
at Helios.Net.Connections.TcpConnection.Open()
at Akka.Remote.Transport.Helios.HeliosHelpers.Open()
at Akka.Remote.Transport.Helios.CommonHandlers.Open()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)
at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )
at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.OnReceive(Object message)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[WARNING][2016-01-27 08:26:22][Thread 0011][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-5/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:6001: Error [Invalid address: akka.tcp://akkapocsystem@localhost:6001] []
[WARNING][2016-01-27 08:26:22][Thread 0011][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:6001]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:6001] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Helios.Net.Connections.TcpConnection.Open()
--- End of inner exception stack trace ---
at Helios.Net.Connections.TcpConnection.Open()
at Akka.Remote.Transport.Helios.HeliosHelpers.Open()
at Akka.Remote.Transport.Helios.CommonHandlers.Open()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()
--- End of inner exception stack trace ---]
[ERROR][2016-01-27 08:26:22][Thread 0011][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-5] Invalid address: akka.tcp://akkapocsystem@localhost:6001
Cause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:6001 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:6001
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Helios.Net.Connections.TcpConnection.Open()
--- End of inner exception stack trace ---
at Helios.Net.Connections.TcpConnection.Open()
at Akka.Remote.Transport.Helios.HeliosHelpers.Open()
at Akka.Remote.Transport.Helios.CommonHandlers.Open()
at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)
at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)
at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )
at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.OnReceive(Object message)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:26:25][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader can currently not perform its duties, reachability status: [akka.tcp://akkapocsystem@localhost:4053 -> UniqueAddress: (akka.tcp://akkapocsystem@localhost:6001, 1094617602): Unreachable [Unreachable] (1), ], member status: [$akka.tcp://akkapocsystem@localhost:4053 $Up seen=$True, $akka.tcp://akkapocsystem@localhost:6001 $Up seen=$False]
[INFO][2016-01-27 08:26:28][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] New incarnation of existing member [UniqueAddress: (akka.tcp://akkapocsystem@localhost:6001, 43043012)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
[INFO][2016-01-27 08:26:28][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Marking unreachable node [akka.tcp://akkapocsystem@localhost:6001] as Down
[INFO][2016-01-27 08:26:29][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader can perform its duties again
[INFO][2016-01-27 08:26:29][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is removing unreachable node [akka.tcp://akkapocsystem@localhost:6001]
[WARNING][2016-01-27 08:26:29][Thread 0010][remoting] Association to [akka.tcp://akkapocsystem@localhost:6001] having UID [1094617602] is irrecoverably failed. UID is now quarantined and all messages to this UID will be delivered to dead letters. Remote actorsystem must be restarted to recover from this situation.
[WARNING][2016-01-27 08:26:39][Thread 0011][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-7]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:26:39][Thread 0010][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-7] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:26:39][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Node [akka.tcp://akkapocsystem@localhost:6001] is JOINING, roles [webapp]
[INFO][2016-01-27 08:26:40][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
Report 02. Calendar Heat Map - Kopia, was shared to you
[INFO][2016-01-27 08:26:41][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:42][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:43][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:44][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:45][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[WARNING][2016-01-27 08:26:45][Thread 0009][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-8]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:26:45][Thread 0008][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-8] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:26:46][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:47][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:48][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:49][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:50][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[WARNING][2016-01-27 08:26:50][Thread 0010][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-9]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)]
[INFO][2016-01-27 08:26:50][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Existing member [UniqueAddress: (akka.tcp://akkapocsystem@localhost:6001, 43043012)] is joining again.
[ERROR][2016-01-27 08:26:50][Thread 0010][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-9] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:26:51][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:52][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
!!!>>> HERE I RESPAWN the [webapp] <<<!!!
!!!>>> HERE new mainservice sends new message to [webapp] but it is not received <<<!!!
[INFO][2016-01-27 08:26:53][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:54][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:55][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:56][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[WARNING][2016-01-27 08:26:56][Thread 0009][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-10]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:26:56][Thread 0008][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-10] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:26:57][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:58][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:26:59][Thread 0015][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:00][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:01][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[WARNING][2016-01-27 08:27:01][Thread 0011][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-11]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:27:01][Thread 0008][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-11] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:27:01][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Existing member [UniqueAddress: (akka.tcp://akkapocsystem@localhost:6001, 43043012)] is joining again.
[INFO][2016-01-27 08:27:02][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:03][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:04][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:05][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:06][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:07][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[WARNING][2016-01-27 08:27:07][Thread 0008][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-12]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:27:07][Thread 0011][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-12] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:27:08][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:09][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
Report #18866 sencond labels error, was shared to you
[INFO][2016-01-27 08:27:11][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[INFO][2016-01-27 08:27:12][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[WARNING][2016-01-27 08:27:12][Thread 0009][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-13]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)]
[INFO][2016-01-27 08:27:12][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Existing member [UniqueAddress: (akka.tcp://akkapocsystem@localhost:6001, 43043012)] is joining again.
[ERROR][2016-01-27 08:27:12][Thread 0010][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-13] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[INFO][2016-01-27 08:27:13][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:6001] to [Up]
[WARNING][2016-01-27 08:27:17][Thread 0010][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-14]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:27:17][Thread 0009][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-14] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[WARNING][2016-01-27 08:27:23][Thread 0009][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-15]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:27:23][Thread 0008][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-15] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
Report 04. Wyniki, was shared to you
[WARNING][2016-01-27 08:27:35][Thread 0008][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-16]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:27:35][Thread 0010][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-16] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[WARNING][2016-01-27 08:27:40][Thread 0011][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-17]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:27:40][Thread 0009][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-17] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)
[WARNING][2016-01-27 08:27:45][Thread 0011][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-18]] Association with remote system akka.tcp://akkapocsystem@localhost:6001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()]
[ERROR][2016-01-27 08:27:45][Thread 0011][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a6001-18] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.Unhandled(Object message)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)My hocon from [webapp]
<hocon>
<![CDATA[
akka {
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
}
remote {
log-remote-lifecycle-events = DEBUG
log-received-messages = on
log-sent-messages = on
helios.tcp {
transport-class = "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote"
hostname = localhost
port = 6001
}
}
cluster {
seed-nodes = ["akka.tcp://akkapocsystem@localhost:4053"]
roles = [webapp]
}
}
]]>
</hocon>I tried waiting 1min before restarting webapp and now in such case it works fine. I suppose I must find the right config setting and set it to a value that's smaller than usual IIS app pool recycle time....
However when I look at the logs I still see lots of Akka.Remote.EndpointDisassociatedException: Disassociated ERROR messages in the logs even after the restart and successful message passing.
I would be good too know to get rid of those errors after rejoin, so that the logs present the real cluster state and ignore some old messages after a while.
cgstevens
commented
8 years ago @Aaronontheweb Correct me if I am wrong or help me understand :)
Talking with @voltcode it seems like he has ran into the same thing which not sure if what I did is a work around.
Just calling Cluster.Terminate doesn't have the member leave the cluster which then causes it to be in a stuck state.
In all of my systems I do the following which basically just does a Cluster.Leave(SelfAddress) and the wait for 5 seconds then terminates. This does fix the issue where when restarting or shutting down the services it actually does leave to allow the new one to rejoin.
protected void Application_Stop()
{
SystemActors.ClusterHelper.Tell(new ClusterHelper.RemoveMember());
Thread.Sleep(5000); // Give the Remove time to actually remove...
ClusterSystem.Terminate();
}I can validate this in my Cluster Monitor example by remove that RemoveMember call.
voltcode
commented
8 years ago As per gitter discussion, this issue still persists and I have created a minimal reproduction sample. The sample is located here: https://github.com/voltcode/AkkaSample
The scenario to replicate the bug is as follows (assuming the solution is open in VS):
1.mark the service project as startup, launch the service (best with CTRL+F5 so it stays outside debugger).
Please consider reopening this issue @Aaronontheweb . @cgstevens do you mind having a look as well, and see if this minimal scenario correlates with your case?
voltcode
commented
8 years ago Another experiment - I changed the sample app to use pure remoting without cluster, connection gets re-established properly, so I believe the problem is in the Cluster code not in Remote and the network stack below.
Scenario with Remote was scheduling repeatedly a message from a service actor to web actor, and killing the web application after mesages went through (verified with a breakpoint in web code). I can push the Remote sample to repo if it helps?
voltcode
commented
8 years ago I have created a branch of my sample https://github.com/voltcode/AkkaSample/tree/AkkaSampleRemote to illustrate the "remote" version without clustering working fine.
voltcode
commented
8 years ago Possibly related https://github.com/akka/akka/issues/16224
Aaronontheweb
commented
8 years ago I'll reopen this and see what could be possibly going on here
voltcode
commented
8 years ago I began debugging the issue. I found that at one point during reconnection attempt, the EndpointManager of the webapp seems to try ungating the connection but its ungate message is never handled.
message Ungate is sent by pass.Endpoint.Tell(new ReliableDeliverySupervisor.Ungate());
logs confirm that it's never handled - I wonder if ignoring ungate is the desired option.
private void HandleInboundAssociation(InboundAssociation ia, bool writerIsIdle)
{
var readonlyEndpoint = _endpoints.ReadOnlyEndpointFor(ia.Association.RemoteAddress);
var handle = ((AkkaProtocolHandle)ia.Association);
if (readonlyEndpoint != null)
{
var endpoint = readonlyEndpoint.Item1;
if (_pendingReadHandoffs.ContainsKey(endpoint)) _pendingReadHandoffs[endpoint].Disassociate();
_pendingReadHandoffs.AddOrSet(endpoint, handle);
endpoint.Tell(new EndpointWriter.TakeOver(handle, Self));
_endpoints.WritableEndpointWithPolicyFor(handle.RemoteAddress).Match()
.With<Pass>(pass =>
{
pass.Endpoint.Tell(new ReliableDeliverySupervisor.Ungate());
});
}More clarification, the first time Ungate is sent (and not handled) during reconnect attempt is actually in a different place of the same method as above:
}
else if (pass != null) // has a UID value
{
if (handle.HandshakeInfo.Uid == pass.Uid)
{
_pendingReadHandoffs.GetOrElse(pass.Endpoint, null)?.Disassociate();
_pendingReadHandoffs.AddOrSet(pass.Endpoint, handle);
pass.Endpoint.Tell(new EndpointWriter.StopReading(pass.Endpoint, Self));
pass.Endpoint.Tell(new ReliableDeliverySupervisor.Ungate());
}
elseit doesnt get handled, consecutive "ungates" are sent from the place in the previous comment.
voltcode
commented
8 years ago exact log message (proof of Ungate unhanlded):
2016-02-20 00:00:43,714 [11] DEBUG Akka.Actor.LocalActorRef [(null)] - Unhandled message from akka://voltcode/system/endpointManager : Akka.Remote.ReliableDeliverySupervisor+Ungate
voltcode
commented
8 years ago a longer excerpt from the logs:
2016-02-20 00:01:30,100 [6] DEBUG Akka.Actor.LocalActorRef [(null)] - Unhandled message from akka://voltcode/system/endpointManager : Akka.Remote.ReliableDeliverySupervisor+Ungate
2016-02-20 00:01:30,100 [6] DEBUG Akka.Remote.EndpointReader [(null)] - Stopped
2016-02-20 00:01:30,101 [6] DEBUG Akka.Remote.Transport.ProtocolStateActor [(null)] - Stopped
2016-02-20 00:01:30,101 [6] INFO Akka.Actor.LocalActorRef [(null)] - Message Disassociated from NoSender to akka://voltcode/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fvoltcode%40127.0.0.1%3a59345-9 was not delivered. 8 dead letters encountered.
2016-02-20 00:01:30,101 [6] DEBUG Akka.Remote.EndpointWriter [(null)] - received AutoReceiveMessage <Terminated>: [akka://voltcode/system/endpointManager/endpointWriter-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-2/endpointReader-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-8] - ExistenceConfirmed=True
2016-02-20 00:01:30,102 [13] DEBUG Akka.Remote.EndpointReader [(null)] - Started (Akka.Remote.EndpointReader)
2016-02-20 00:01:30,103 [13] DEBUG Akka.Remote.EndpointReader [(null)] - now watched by [akka://voltcode/system/endpointManager/endpointWriter-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-2]
2016-02-20 00:01:30,103 [13] DEBUG Akka.Remote.EndpointWriter [(null)] - now supervising akka://voltcode/system/endpointManager/endpointWriter-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-2/endpointReader-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-9
2016-02-20 00:01:30,112 [27] DEBUG Akka.Cluster.ClusterCoreDaemon [(null)] - [Uninitialized] Received Akka.Cluster.GossipEnvelope
2016-02-20 00:01:35,119 [6] DEBUG Akka.Remote.Transport.AkkaProtocolManager [(null)] - now supervising akka://voltcode/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fvoltcode%40127.0.0.1%3a59348-11
2016-02-20 00:01:35,120 [6] DEBUG Akka.Remote.Transport.ProtocolStateActor [(null)] - Started (Akka.Remote.Transport.ProtocolStateActor)
2016-02-20 00:01:35,120 [6] DEBUG Akka.Remote.EndpointReader [(null)] - Stopped
2016-02-20 00:01:35,121 [6] DEBUG Akka.Remote.EndpointWriter [(null)] - received AutoReceiveMessage <Terminated>: [akka://voltcode/system/endpointManager/endpointWriter-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-2/endpointReader-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-9] - ExistenceConfirmed=True
2016-02-20 00:01:35,121 [6] DEBUG Akka.Remote.Transport.ProtocolStateActor [(null)] - Stopped
2016-02-20 00:01:35,122 [6] DEBUG Akka.Actor.LocalActorRef [(null)] - Unhandled message from akka://voltcode/system/endpointManager : Akka.Remote.ReliableDeliverySupervisor+Ungate
2016-02-20 00:01:35,122 [6] INFO Akka.Actor.LocalActorRef [(null)] - Message Disassociated from NoSender to akka://voltcode/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fvoltcode%40127.0.0.1%3a59347-10 was not delivered. 9 dead letters encountered.
2016-02-20 00:01:35,131 [14] DEBUG Akka.Remote.EndpointReader [(null)] - Started (Akka.Remote.EndpointReader)
2016-02-20 00:01:35,132 [14] DEBUG Akka.Remote.EndpointWriter [(null)] - now supervising akka://voltcode/system/endpointManager/endpointWriter-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-2/endpointReader-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-10
2016-02-20 00:01:35,132 [14] DEBUG Akka.Remote.EndpointReader [(null)] - now watched by [akka://voltcode/system/endpointManager/endpointWriter-akka.tcp%3a%2f%2fvoltcode%40localhost%3a4056-2]
2016-02-20 00:01:35,147 [26] DEBUG Akka.Cluster.ClusterCoreDaemon [(null)] - [Uninitialized] Received Akka.Cluster.GossipEnvelope
2016-02-20 00:01:41,159 [6] DEBUG Akka.Remote.Transport.AkkaProtocolManager [(null)] - now supervising akka://voltcode/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fvoltcode%40127.0.0.1%3a59349-12
2016-02-20 00:01:41,159 [6] DEBUG Akka.Remote.Transport.ProtocolStateActor [(null)] - Started (Akka.Remote.Transport.ProtocolStateActor)
2016-02-20 00:01:41,160 [6] DEBUG Akka.Actor.LocalActorRef [(null)] - Unhandled message from akka://voltcode/system/endpointManager : Akka.Remote.ReliableDeliverySupervisor+Ungate
2016-02-20 00:01:41,160 [6] DEBUG Akka.Remote.EndpointReader [(null)] - Stopped
2016-02-20 00:01:41,160 [6] DEBUG Akka.Remote.Transport.ProtocolStateActor [(null)] - Stoppedcorresponding log from the other application:
2016-02-20 00:01:29,081 [6] INFO Akka.Cluster.ClusterCoreDaemon [(null)] - Leader is moving node [akka.tcp://voltcode@localhost:5001] to [Up]
2016-02-20 00:01:30,089 [6] INFO Akka.Cluster.ClusterCoreDaemon [(null)] - Leader is moving node [akka.tcp://voltcode@localhost:5001] to [Up]
2016-02-20 00:01:30,105 [6] WARN Akka.Remote.ReliableDeliverySupervisor [(null)] - Association with remote system akka.tcp://voltcode@localhost:5001 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level) in C:\src\akkanet\core\akka.net\src\core\Akka.Remote\Endpoint.cs:line 1028
at Akka.Remote.EndpointWriter.Unhandled(Object message) in C:\src\akkanet\core\akka.net\src\core\Akka.Remote\Endpoint.cs:line 966
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ReceiveActor.cs:line 70
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ReceiveActor.cs:line 80
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.cs:line 194
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorBase.cs:line 155
at Akka.Actor.ActorCell.ReceiveMessage(Object message) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 145
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DeathWatch.cs:line 55
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 115
at Akka.Actor.ActorCell.Invoke(Envelope envelope) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 60
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.FaultHandling.cs:line 398
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 177]
2016-02-20 00:01:30,112 [6] ERROR Akka.Actor.OneForOneStrategy [(null)] - Disassociated
Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level) in C:\src\akkanet\core\akka.net\src\core\Akka.Remote\Endpoint.cs:line 1028
at Akka.Remote.EndpointWriter.Unhandled(Object message) in C:\src\akkanet\core\akka.net\src\core\Akka.Remote\Endpoint.cs:line 966
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ReceiveActor.cs:line 70
at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ReceiveActor.cs:line 80
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.cs:line 194
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorBase.cs:line 155
at Akka.Actor.ActorCell.ReceiveMessage(Object message) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 145
at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DeathWatch.cs:line 55
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 115
at Akka.Actor.ActorCell.Invoke(Envelope envelope) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 60
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.FaultHandling.cs:line 398
at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope) in C:\src\akkanet\core\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 177
2016-02-20 00:01:31,089 [6] INFO Akka.Cluster.ClusterCoreDaemon [(null)] - Leader is moving node [akka.tcp://voltcode@localhost:5001] to [Up]
2016-02-20 00:01:32,091 [6] INFO Akka.Cluster.ClusterCoreDaemon [(null)] - Leader is moving node [akka.tcp://voltcode@localhost:5001] to [Up]
2016-02-20 00:01:33,092 [6] INFO Akka.Cluster.ClusterCoreDaemon [(null)] - Leader is moving node [akka.tcp://voltcode@localhost:5001] to [Up]
2adding full logs from both sides AkkaSample-console.zip AkkaSample-SERVICE.zip
voltcode
commented
8 years ago I found that the unexpected "Ungate" is sent when ReliableDeliverySupervisor (recipient) is in Receiving state.
voltcode
commented
8 years ago another observation: in initial (successful) join the sequence of remoting messages on the joining node is OutboundUnassociated (calls sendAssociate) and then OutboundUnderlyingAssociated.
on second (failed) join after process restart, the sequence is: OutboundUnassociated (calls sendAssociate), then InboundUnassociated.
at the same time, the message on the seed node is on failed join is AssociateUnderlyingRefuseUid with refuseUid = null. Thats because the handle inside EndpointWriter is null on the failed join (it's not null on the initial join)
voltcode
commented
8 years ago I have created a new version of sample project for bug replication - the non-seed node that's trying to join is now a console project - much quicker to reproduce than restarting web app
https://github.com/voltcode/AkkaSample/tree/AkkaSampleConsole
to reproduce - run Service first, Console second. kill Console. start Console again
annymsMthd
commented
8 years ago voltcode
commented
8 years ago I just wanted to say that I retested the scenario on nightly build 1.0.9.238
<package id="Akka.Cluster" version="1.0.9.238-beta" targetFramework="net452" />
and the issue no longer occurs - thank you very much! Keeping my fingers crossed for official 1.1 build - I will retest when it's fully cooked.
cgstevens
commented
8 years ago Hooray!!!
Sent from my iPhone, so if there are any misspellings I blame it.
On Jul 6, 2016, at 8:55 AM, voltcode notifications@github.com<mailto:notifications@github.com> wrote:
I just wanted to say that I retested the scenario on nightly build 1.0.9.238
and the issue no longer occurs - thank you very much! Keeping my fingers crossed for official 1.1 build - I will retest when it's fully cooked.
You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/akkadotnet/akka.net/issues/1670#issuecomment-230762726, or mute the threadhttps://github.com/notifications/unsubscribe/AKqYaPAcrKNBcJ2ZyrT1Bmc5VM1IdgaEks5qS6W_gaJpZM4HMk7u.
My setup:
a [wcfservice] role talks to [mainservice] role (a windows service) that talks to [webapp] role (an mvc app)
1.I start up [mainservice], [wcfservice], [webapp](in this order)
all looks good in the logs, cluster is formed.
2.message is send from wcfservice to mainservice. mainservice processes it and sends a message to webapp. All works as expected
3.I kill [webapp]. gossip failures show up in [mainservice] logs (as expected).
4.I restart webapp, there is no message confirming that it rejoined the cluster. Sending new message as in p.2. does not work.
I would expect that the webapp would rejoin the cluster automatically on startup. If I kill the [mainservice] together with [webapp], and start them both up (mainservice first) then cluster is formed correctly (as in situation from p.1).
akka std out from [mainservice] logs edited to show when p.2,3,4 occur (look for !!!>>>)
[mainservice] hocon