petabridge / petabridge.cmd-issues

Issue tracker for Petabridge.Cmd
https://cmd.petabridge.com/
2 stars 0 forks source link

Petabridge.Cmd `actor hierarchy` prints different results each time. #36

Closed kimbyungeun closed 3 years ago

kimbyungeun commented 4 years ago

https://github.com/akkadotnet/akka.net/issues/4585

Environment

Question

our service

image

actor hierarchy -d 8 1st run

image


### Why is the output different every time the `actor hierarchy` command?

@seunghalee91
kimbyungeun commented 4 years ago

The reasons why the results are in different orders each time is because the actor hierarchy scan is done recursively in parallel down in the actor hierarchy using wildcard ActorSelections and the completion order of those parallel tasks is arbitrary. We're able to guarantee that a parent doesn't complete its scan until its children have completed theirs and so on, all the way until we reach the specified level of depth. But we don't guarantee the completion order of siblings - i.e. the order in which two children of the /user actor complete their scans.

Using akka 1.3.16 will print all actors.

pbm api log

[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c3
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c2
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3064
/user/petabridge.cmd/127.0.0.1%3A3064/actor
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler/$h
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler/$h/$a
/user/petabridge.cmd-log-memorizer

Using akka 1.4.10 will print some actors.

pbm lighthouse log

[127.0.0.1:9115] pbm> cluster show
akka.tcp://mls-cluster-system@127.0.0.1:3507 | [api] | up |
akka.tcp://mls-cluster-system@127.0.0.1:3511 | [role1] | up |
akka.tcp://mls-cluster-system@127.0.0.1:4062 | [lighthouse] | up |
akka.tcp://mls-cluster-system@127.0.0.1:8112 | [role2] | up |
akka.tcp://mls-cluster-system@127.0.0.1:8113 | [role3] | up |
Count: 5 nodes
[127.0.0.1:9115] pbm>

pbm api log

petabridge.cmd (0.8.0.0) Copyright 2017 - 2020, Petabridge®.

successfully connected to [::ffff:127.0.0.1]:9111 Commands downloaded from server. type help to see what's available [127.0.0.1:9111] pbm> actor hierarchy -d 10 /user /user/ApiMasterActor /user/ApiMasterActor/$a /user/ApiMasterActor/ClusterSingleton /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor /user/pbm-uptime /user/petabridge.cmd /user/petabridge.cmd/127.0.0.1%3A3943 /user/petabridge.cmd/127.0.0.1%3A3943/actor /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b /user/petabridge.cmd-log-memorizer [127.0.0.1:9111] pbm> actor hierarchy -d 10 /user /user/ApiMasterActor /user/ApiMasterActor/$a /user/ApiMasterActor/ClusterSingleton /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor /user/pbm-uptime /user/petabridge.cmd /user/petabridge.cmd/127.0.0.1%3A3943 /user/petabridge.cmd/127.0.0.1%3A3943/actor /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b /user/petabridge.cmd-log-memorizer [127.0.0.1:9111] pbm> actor hierarchy -d 10 /user /user/ApiMasterActor /user/ApiMasterActor/$a /user/ApiMasterActor/ClusterSingleton /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor /user/pbm-uptime /user/petabridge.cmd /user/petabridge.cmd/127.0.0.1%3A3943 /user/petabridge.cmd/127.0.0.1%3A3943/actor /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b /user/petabridge.cmd-log-memorizer [127.0.0.1:9111] pbm> actor hierarchy -d 10 /user /user/ApiMasterActor /user/ApiMasterActor/$a /user/ApiMasterActor/ClusterSingleton /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor /user/pbm-uptime /user/petabridge.cmd /user/petabridge.cmd/127.0.0.1%3A3943 /user/petabridge.cmd/127.0.0.1%3A3943/actor /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b /user/petabridge.cmd-log-memorizer [127.0.0.1:9111] pbm>


* The following `Received unexpected ActorIdentity` warning log is displayed in the [api] role.

[2020/10/22 19:50:48.525][ 27][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] Received petabridge.cmd connection from client [127.0.0.1:3943] [2020/10/22 19:50:48.525][ 27][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending palettes to client... [2020/10/22 19:50:57.714][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]] [2020/10/22 19:50:57.776][ 32][DEBUG]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b[Akka.Actor.LocalActorRef] Unhandled message from akka.tcp://mls-cluster-system@127.0.0.1:3927/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 : : [akka.tcp://mls-cluster-system@127.0.0.1:3927/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#1878102528] - MessageId=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor [2020/10/22 19:50:57.805][ 10][DEBUG]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b[Akka.Actor.LocalActorRef] Unhandled message from akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1 : : [akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#585265792] - MessageId=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user to client [15 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor to client [30 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/$a to client [33 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton to client [47 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor to client [74 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor to client [74 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor to client [74 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/pbm-uptime to client [26 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd to client [30 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943 to client [47 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor to client [53 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler to client [61 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a to client [64 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b to client [67 bytes] [2020/10/22 19:50:58.302][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd-log-memorizer to client [44 bytes] [2020/10/22 19:50:58.309][ 30][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=True]: N/A to client [15 bytes] [2020/10/22 19:51:01.446][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]] [2020/10/22 19:51:01.508][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#585265792]] [2020/10/22 19:51:02.100][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user to client [15 bytes] [2020/10/22 19:51:02.100][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor to client [30 bytes] [2020/10/22 19:51:02.100][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/$a to client [33 bytes] [2020/10/22 19:51:02.100][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton to client [47 bytes] [2020/10/22 19:51:02.100][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor to client [74 bytes] [2020/10/22 19:51:02.109][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 to client [77 bytes] [2020/10/22 19:51:02.109][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor to client [74 bytes] [2020/10/22 19:51:02.109][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor to client [74 bytes] [2020/10/22 19:51:02.109][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/pbm-uptime to client [26 bytes] [2020/10/22 19:51:02.109][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd to client [30 bytes] [2020/10/22 19:51:02.109][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943 to client [47 bytes] [2020/10/22 19:51:02.109][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor to client [53 bytes] [2020/10/22 19:51:02.109][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler to client [61 bytes] [2020/10/22 19:51:02.126][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b to client [64 bytes] [2020/10/22 19:51:02.126][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b to client [67 bytes] [2020/10/22 19:51:02.126][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd-log-memorizer to client [44 bytes] [2020/10/22 19:51:02.126][ 5][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=True]: N/A to client [15 bytes] [2020/10/22 19:51:04.114][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]] [2020/10/22 19:51:04.131][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#585265792]] [2020/10/22 19:51:04.721][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user to client [15 bytes] [2020/10/22 19:51:04.721][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor to client [30 bytes] [2020/10/22 19:51:04.721][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/$a to client [33 bytes] [2020/10/22 19:51:04.721][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton to client [47 bytes] [2020/10/22 19:51:04.731][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor to client [74 bytes] [2020/10/22 19:51:04.731][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 to client [77 bytes] [2020/10/22 19:51:04.731][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor to client [74 bytes] [2020/10/22 19:51:04.731][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor to client [74 bytes] [2020/10/22 19:51:04.731][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/pbm-uptime to client [26 bytes] [2020/10/22 19:51:04.731][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd to client [30 bytes] [2020/10/22 19:51:04.731][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943 to client [47 bytes] [2020/10/22 19:51:04.731][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor to client [53 bytes] [2020/10/22 19:51:04.748][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler to client [61 bytes] [2020/10/22 19:51:04.748][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c to client [64 bytes] [2020/10/22 19:51:04.774][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b to client [67 bytes] [2020/10/22 19:51:04.774][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd-log-memorizer to client [44 bytes] [2020/10/22 19:51:04.774][ 26][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=True]: N/A to client [15 bytes] [2020/10/22 19:51:06.455][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]] [2020/10/22 19:51:06.517][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#585265792]] [2020/10/22 19:51:06.581][ 6][DEBUG]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b[Akka.Actor.LocalActorRef] Unhandled message from akka.tcp://mls-cluster-system@127.0.0.1:3927/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 : : [akka.tcp://mls-cluster-system@127.0.0.1:3927/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#1878102528] - MessageId=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor [2020/10/22 19:51:07.066][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user to client [15 bytes] [2020/10/22 19:51:07.066][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor to client [30 bytes] [2020/10/22 19:51:07.066][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/$a to client [33 bytes] [2020/10/22 19:51:07.066][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton to client [47 bytes] [2020/10/22 19:51:07.066][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor to client [74 bytes] [2020/10/22 19:51:07.066][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor to client [74 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor to client [74 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/pbm-uptime to client [26 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd to client [30 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943 to client [47 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor to client [53 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler to client [61 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d to client [64 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b to client [67 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=False]: /user/petabridge.cmd-log-memorizer to client [44 bytes] [2020/10/22 19:51:07.076][ 10][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending [RESPONSE][FINAL=True]: N/A to client [15 bytes]

### [api] role hocon

petabridge.cmd { host = "0.0.0.0" port = 9111 log-palettes-on-startup = on }

akka { service.name = "TestClusterApi" service.description = "TestClusterApi" system.name = "mls-cluster-system"

stdout-loglevel = DEBUG
loglevel = DEBUG
log-config-on-start = on
loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]

actor
{
    provider = cluster
    guardian-supervisor-strategy = "TestClusterApi.ActorService.UserSupervisorStrategy, TestClusterApi"

    debug
    {
        receive = off         
        autoreceive = off     
        lifecycle = off       
        event-stream = off    
        unhandled = on        
    }

    deployment 
    {
        /ApiMasterActor 
        {

        }

        "/ApiMasterActor/*/TestClusterRole1HelloActor"
        {
            router = round-robin-pool
            nr-of-instances = 100

            cluster 
            {
                enabled = on
                max-nr-of-instances-per-node = 1
                allow-local-routees = off
                use-role = role1
            }
        }

        "/ApiMasterActor/*/TestClusterRole2HelloActor"
        {
            router = round-robin-pool
            nr-of-instances = 100

            cluster 
            {
                enabled = on
                max-nr-of-instances-per-node = 1
                allow-local-routees = off
                use-role = role2
            }
        }

        "/ApiMasterActor/*/TestClusterRole3HelloActor"
        {
            router = round-robin-pool
            nr-of-instances = 100

            cluster 
            {
                enabled = on
                max-nr-of-instances-per-node = 1
                allow-local-routees = off
                use-role = role3
            }
        }
    }
}

remote
{
    dot-netty.tcp 
    {
        public-hostname = 127.0.0.1
        port = 0

        send-buffer-size    = 2097152b  # 2MB
        receive-buffer-size = 2097152b  # 2MB
        maximum-frame-size  = 1048576b  # 1MB
    }

    transport-failure-detector
    {
        heartbeat-interval = 5s                
        acceptable-heartbeat-pause = 50s       
    }
}

extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]

cluster 
{
    seed-nodes = ["akka.tcp://mls-cluster-system@127.0.0.1:4062"]
    roles = [api]

    auto-down-unreachable-after = 10s 

    gossip-interval = 5s 
    gossip-time-to-live = 30s 

    failure-detector 
    {
        heartbeat-interval = 5s
        acceptable-heartbeat-pause = 50s
    }

    singleton
    {
        singleton-name = "ClusterSingleton"
        role = "role1"
        hand-over-retry-interval = 1s
        min-number-of-hand-over-retries = 10
    }
}

}

Aaronontheweb commented 3 years ago

we'll take a look at this - wonder if we made any changes to ActorSelection in the meantime.

IgorFedchenko commented 3 years ago

@kimbyungeun I am going to look into this. From the logs in your last message, it seems like under Akka 1.4+ not all actors are displayed, right? But all c1/c2/c3 child actors under TestClusterRole1HelloActor are running all the time, right (I mean, it is not that they are shutting down and starting from time to time)?

Do you have a sample project you used to generate this logs? If you do, could you share in as a zip file or Github repo to help me with reproduction? That would be extremely useful.

kimbyungeun commented 3 years ago

@IgorFedchenko

Q1

I am going to look into this. From the logs in your last message, it seems like under Akka 1.4+ not all actors are displayed, right?

  • YES

    Q2

    But all c1/c2/c3 child actors under TestClusterRole1HelloActor are running all the time, right (I mean, it is not that they are shutting down and starting from time to time)?

  • YES

    Lighthouse.log

    
    [2020-10-23 09:18:22.1925][ INFO][P:13308][T:  1]  Configuration Result:
    [Success] Name Mirero.MLS.TestLightHouse4061
    [Success] Description Mirero.MLS.TestLightHouse.4061
    [Success] ServiceName Mirero.MLS.TestLightHouse4061 
    [2020-10-23 09:18:22.2125][ INFO][P:13308][T:  1]  Topshelf v4.2.1.215, .NET Framework v4.0.30319.42000 
    [2020-10-23 09:18:22.2355][DEBUG][P:13308][T:  1]  Running as a console application, creating the console host. 
    [2020-10-23 09:18:22.2355][DEBUG][P:13308][T:  1]  Starting up as a console application 
    [2020-10-23 09:18:22.5005][DEBUG][P:13308][T:  5] EventStream(mls-cluster-system) Logger log1-NLogLogger [NLogLogger] started 
    [2020-10-23 09:18:22.5005][DEBUG][P:13308][T:  5] EventStream(mls-cluster-system) StandardOutLogger being removed 
    [2020-10-23 09:18:22.5005][DEBUG][P:13308][T:  5] EventStream(mls-cluster-system) Default Loggers started 
    [2020-10-23 09:18:22.5135][ INFO][P:13308][T:  8] remoting (akka://mls-cluster-system) Starting remoting 
    [2020-10-23 09:18:22.5285][DEBUG][P:13308][T:  8] remoting (akka://mls-cluster-system) Starting prune timer for endpoint manager... 
    [2020-10-23 09:18:22.7185][ INFO][P:13308][T:  5] remoting (akka://mls-cluster-system) Remoting started; listening on addresses : [akka.tcp://mls-cluster-system@127.0.0.1:4062] 
    [2020-10-23 09:18:22.7185][ INFO][P:13308][T:  5] remoting (akka://mls-cluster-system) Remoting now listens on addresses: [akka.tcp://mls-cluster-system@127.0.0.1:4062] 
    [2020-10-23 09:18:22.7515][ INFO][P:13308][T:  4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Starting up... 
    [2020-10-23 09:18:22.7755][ INFO][P:13308][T:  6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Started up successfully 
    [2020-10-23 09:18:22.8015][ INFO][P:13308][T: 19] ActorSystem(mls-cluster-system)   akka : {
    version : "0.0.1 Akka"
    home : 
    loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
    loggers-dispatcher : akka.actor.default-dispatcher
    logger-startup-timeout : 5s
    logger-async-start : false
    loglevel : DEBUG
    suppress-json-serializer-warning : on
    stdout-loglevel : DEBUG
    log-config-on-start : on
    log-dead-letters : 10
    log-dead-letters-during-shutdown : off
    log-dead-letters-suspend-duration : "5 minutes"
    extensions : ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]
    daemonic : off
    actor : {
    provider : cluster
    guardian-supervisor-strategy : Akka.Actor.DefaultSupervisorStrategy
    creation-timeout : 20s
    reaper-interval : 5
    serialize-messages : off
    serialize-creators : off
    unstarted-push-timeout : 10s
    ask-timeout : infinite
    typed : {
    timeout : 5
    }
    inbox : {
    inbox-size : 1000
    default-timeout : 5s
    }
    router : {
    type-mapping : {
    from-code : Akka.Routing.NoRouter
    round-robin-pool : Akka.Routing.RoundRobinPool
    round-robin-group : Akka.Routing.RoundRobinGroup
    random-pool : Akka.Routing.RandomPool
    random-group : Akka.Routing.RandomGroup
    smallest-mailbox-pool : Akka.Routing.SmallestMailboxPool
    broadcast-pool : Akka.Routing.BroadcastPool
    broadcast-group : Akka.Routing.BroadcastGroup
    scatter-gather-pool : Akka.Routing.ScatterGatherFirstCompletedPool
    scatter-gather-group : Akka.Routing.ScatterGatherFirstCompletedGroup
    consistent-hashing-pool : Akka.Routing.ConsistentHashingPool
    consistent-hashing-group : Akka.Routing.ConsistentHashingGroup
    tail-chopping-pool : Akka.Routing.TailChoppingPool
    tail-chopping-group : Akka.Routing.TailChoppingGroup
    }
    }
    deployment : {
    default : {
    dispatcher : 
    mailbox : 
    router : from-code
    nr-of-instances : 1
    within : "5 s"
    virtual-nodes-factor : 10
    routees : {
    paths : []
    }
    resizer : {
    enabled : off
    lower-bound : 1
    upper-bound : 10
    pressure-threshold : 1
    rampup-rate : 0.2
    backoff-threshold : 0.3
    backoff-rate : 0.1
    messages-per-resize : 10
    }
    remote : 
    target : {
    nodes : []
    }
    metrics-selector : mix
    cluster : {
    enabled : off
    max-nr-of-instances-per-node : 1
    max-total-nr-of-instances : 10000
    allow-local-routees : on
    use-role : 
    }
    }
    }
    synchronized-dispatcher : {
    type : SynchronizedDispatcher
    executor : current-context-executor
    throughput : 10
    }
    task-dispatcher : {
    type : TaskDispatcher
    executor : task-executor
    throughput : 30
    }
    default-fork-join-dispatcher : {
    type : ForkJoinDispatcher
    executor : fork-join-executor
    throughput : 30
    dedicated-thread-pool : {
    thread-count : 3
    threadtype : background
    }
    }
    default-dispatcher : {
    type : Dispatcher
    executor : default-executor
    default-executor : {
    }
    thread-pool-executor : {
    }
    fork-join-executor : {
    dedicated-thread-pool : {
    thread-count : 3
    threadtype : background
    }
    }
    current-context-executor : {
    }
    shutdown-timeout : 1s
    throughput : 30
    throughput-deadline-time : 0ms
    attempt-teamwork : on
    mailbox-requirement : 
    }
    default-mailbox : {
    mailbox-type : Akka.Dispatch.UnboundedMailbox
    mailbox-capacity : 1000
    mailbox-push-timeout-time : 10s
    stash-capacity : -1
    }
    mailbox : {
    requirements : {
    Akka.Dispatch.IUnboundedMessageQueueSemantics : akka.actor.mailbox.unbounded-queue-based
    Akka.Dispatch.IBoundedMessageQueueSemantics : akka.actor.mailbox.bounded-queue-based
    Akka.Dispatch.IDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based
    Akka.Dispatch.IUnboundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based
    Akka.Dispatch.IBoundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.bounded-deque-based
    Akka.Dispatch.IMultipleConsumerSemantics : akka.actor.mailbox.unbounded-queue-based
    Akka.Event.ILoggerMessageQueueSemantics : akka.actor.mailbox.logger-queue
    }
    unbounded-queue-based : {
    mailbox-type : Akka.Dispatch.UnboundedMailbox
    }
    bounded-queue-based : {
    mailbox-type : Akka.Dispatch.BoundedMailbox
    }
    unbounded-deque-based : {
    mailbox-type : Akka.Dispatch.UnboundedDequeBasedMailbox
    }
    bounded-deque-based : {
    mailbox-type : Akka.Dispatch.BoundedDequeBasedMailbox
    }
    logger-queue : {
    mailbox-type : Akka.Event.LoggerMailboxType
    }
    }
    debug : {
    receive : off
    autoreceive : off
    lifecycle : off
    fsm : off
    event-stream : off
    unhandled : on
    router-misconfiguration : off
    }
    serializers : {
    json : "Akka.Serialization.NewtonSoftJsonSerializer, Akka"
    bytes : "Akka.Serialization.ByteArraySerializer, Akka"
    akka-containers : "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote"
    akka-misc : "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote"
    primitive : "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote"
    proto : "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote"
    daemon-create : "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote"
    akka-system-msg : "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote"
    akka-cluster : "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster"
    akka-cluster-client : "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools"
    akka-pubsub : "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools"
    }
    serialization-bindings : {
    System.Byte[] : bytes
    System.Object : json
    "Akka.Actor.ActorSelectionMessage, Akka" : akka-containers
    "Akka.Remote.DaemonMsgCreate, Akka.Remote" : daemon-create
    "Google.Protobuf.IMessage, Google.Protobuf" : proto
    "Akka.Actor.Identify, Akka" : akka-misc
    "Akka.Actor.ActorIdentity, Akka" : akka-misc
    "Akka.Actor.IActorRef, Akka" : akka-misc
    "Akka.Actor.PoisonPill, Akka" : akka-misc
    "Akka.Actor.Kill, Akka" : akka-misc
    "Akka.Actor.RemoteScope, Akka" : akka-misc
    "Akka.Routing.FromConfig, Akka" : akka-misc
    "Akka.Routing.DefaultResizer, Akka" : akka-misc
    "Akka.Routing.RoundRobinPool, Akka" : akka-misc
    "Akka.Routing.BroadcastPool, Akka" : akka-misc
    "Akka.Routing.RandomPool, Akka" : akka-misc
    "Akka.Routing.ScatterGatherFirstCompletedPool, Akka" : akka-misc
    "Akka.Routing.TailChoppingPool, Akka" : akka-misc
    "Akka.Routing.ConsistentHashingPool, Akka" : akka-misc
    "Akka.Configuration.Config, Akka" : akka-misc
    "Akka.Remote.RemoteWatcher+Heartbeat, Akka.Remote" : akka-misc
    "Akka.Remote.RemoteWatcher+HeartbeatRsp, Akka.Remote" : akka-misc
    "Akka.Remote.Routing.RemoteRouterConfig, Akka.Remote" : akka-misc
    "Akka.Dispatch.SysMsg.SystemMessage, Akka" : akka-system-msg
    System.String : primitive
    System.Int32 : primitive
    System.Int64 : primitive
    "Akka.Cluster.IClusterMessage, Akka.Cluster" : akka-cluster
    "Akka.Cluster.Routing.ClusterRouterPool, Akka.Cluster" : akka-cluster
    "Akka.Cluster.Tools.Client.IClusterClientMessage, Akka.Cluster.Tools" : akka-cluster-client
    "Akka.Cluster.Tools.PublishSubscribe.IDistributedPubSubMessage, Akka.Cluster.Tools" : akka-pubsub
    "Akka.Cluster.Tools.PublishSubscribe.Internal.SendToOneSubscriber, Akka.Cluster.Tools" : akka-pubsub
    }
    serialization-identifiers : {
    "Akka.Serialization.ByteArraySerializer, Akka" : 4
    "Akka.Serialization.NewtonSoftJsonSerializer, Akka" : 1
    "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" : 2
    "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" : 3
    "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" : 6
    "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" : 16
    "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" : 17
    "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" : 22
    "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" : 5
    "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" : 15
    "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" : 9
    }
    serialization-settings : {
    }
    }
    scheduler : {
    tick-duration : 10ms
    ticks-per-wheel : 512
    implementation : Akka.Actor.HashedWheelTimerScheduler
    shutdown-timeout : 5s
    }
    io : {
    pinned-dispatcher : {
    type : PinnedDispatcher
    executor : fork-join-executor
    }
    tcp : {
    direct-buffer-pool : {
    class : "Akka.IO.Buffers.DirectBufferPool, Akka"
    buffer-size : 512
    buffers-per-segment : 500
    initial-segments : 1
    buffer-pool-limit : 1024
    }
    disabled-buffer-pool : {
    class : "Akka.IO.Buffers.DisabledBufferPool, Akka"
    buffer-size : 512
    }
    buffer-pool : akka.io.tcp.disabled-buffer-pool
    max-channels : 256000
    selector-association-retries : 10
    batch-accept-limit : 10
    register-timeout : 5s
    max-received-message-size : unlimited
    trace-logging : off
    selector-dispatcher : akka.io.pinned-dispatcher
    worker-dispatcher : akka.actor.default-dispatcher
    management-dispatcher : akka.actor.default-dispatcher
    file-io-dispatcher : akka.actor.default-dispatcher
    file-io-transferTo-limit : 524288
    finish-connect-retries : 5
    windows-connection-abort-workaround-enabled : off
    outgoing-socket-force-ipv4 : false
    }
    udp : {
    direct-buffer-pool : {
    class : "Akka.IO.Buffers.DirectBufferPool, Akka"
    buffer-size : 512
    buffers-per-segment : 500
    initial-segments : 1
    buffer-pool-limit : 1024
    }
    buffer-pool : akka.io.udp.direct-buffer-pool
    nr-of-socket-async-event-args : 32
    max-channels : 4096
    select-timeout : infinite
    selector-association-retries : 10
    receive-throughput : 3
    direct-buffer-size : 18432
    direct-buffer-pool-limit : 1000
    received-message-size-limit : unlimited
    trace-logging : off
    selector-dispatcher : akka.io.pinned-dispatcher
    worker-dispatcher : akka.actor.default-dispatcher
    management-dispatcher : akka.actor.default-dispatcher
    }
    udp-connected : {
    direct-buffer-pool : {
    class : "Akka.IO.Buffers.DirectBufferPool, Akka"
    buffer-size : 512
    buffers-per-segment : 500
    initial-segments : 1
    buffer-pool-limit : 1024
    }
    buffer-pool : akka.io.udp-connected.direct-buffer-pool
    nr-of-socket-async-event-args : 32
    max-channels : 4096
    select-timeout : infinite
    selector-association-retries : 10
    receive-throughput : 3
    direct-buffer-size : 18432
    direct-buffer-pool-limit : 1000
    received-message-size-limit : unlimited
    trace-logging : off
    selector-dispatcher : akka.io.pinned-dispatcher
    worker-dispatcher : akka.actor.default-dispatcher
    management-dispatcher : akka.actor.default-dispatcher
    }
    dns : {
    dispatcher : akka.actor.default-dispatcher
    resolver : inet-address
    inet-address : {
    provider-object : Akka.IO.InetAddressDnsProvider
    positive-ttl : 30s
    negative-ttl : 10s
    cache-cleanup-interval : 120s
    use-ipv6 : true
    }
    }
    }
    coordinated-shutdown : {
    default-phase-timeout : "5 s"
    terminate-actor-system : on
    exit-clr : off
    run-by-clr-shutdown-hook : on
    run-by-actor-system-terminate : on
    phases : {
    before-service-unbind : {
    }
    service-unbind : {
    depends-on : [before-service-unbind]
    }
    service-requests-done : {
    depends-on : [service-unbind]
    }
    service-stop : {
    depends-on : [service-requests-done]
    }
    before-cluster-shutdown : {
    depends-on : [service-stop]
    }
    cluster-sharding-shutdown-region : {
    timeout : "10 s"
    depends-on : [before-cluster-shutdown]
    }
    cluster-leave : {
    depends-on : [cluster-sharding-shutdown-region]
    }
    cluster-exiting : {
    timeout : "10 s"
    depends-on : [cluster-leave]
    }
    cluster-exiting-done : {
    depends-on : [cluster-exiting]
    }
    cluster-shutdown : {
    depends-on : [cluster-exiting-done]
    }
    before-actor-system-terminate : {
    depends-on : [cluster-shutdown]
    }
    actor-system-terminate : {
    timeout : "10 s"
    depends-on : [before-actor-system-terminate]
    }
    }
    }
    remote : {
    startup-timeout : "10 s"
    shutdown-timeout : "10 s"
    flush-wait-on-shutdown : "2 s"
    use-passive-connections : on
    backoff-interval : "0.05 s"
    command-ack-timeout : "30 s"
    handshake-timeout : "15 s"
    use-dispatcher : akka.remote.default-remote-dispatcher
    untrusted-mode : off
    trusted-selection-paths : []
    require-cookie : off
    secure-cookie : 
    log-received-messages : off
    log-sent-messages : off
    log-remote-lifecycle-events : DEBUG
    log-frame-size-exceeding : off
    log-buffer-size-exceeding : 50000
    transport-failure-detector : {
    implementation-class : Akka.Remote.DeadlineFailureDetector,Akka.Remote
    heartbeat-interval : 5s
    acceptable-heartbeat-pause : 50s
    }
    watch-failure-detector : {
    implementation-class : Akka.Remote.PhiAccrualFailureDetector,Akka.Remote
    heartbeat-interval : "1 s"
    threshold : 10.0
    max-sample-size : 200
    min-std-deviation : "100 ms"
    acceptable-heartbeat-pause : "10 s"
    unreachable-nodes-reaper-interval : 1s
    expected-response-after : "1 s"
    }
    retry-gate-closed-for : "5 s"
    prune-quarantine-marker-after : "5 d"
    quarantine-after-silence : "2 d"
    system-message-buffer-size : 20000
    system-message-ack-piggyback-timeout : "0.3 s"
    resend-interval : "2 s"
    resend-limit : 200
    initial-system-message-delivery-timeout : "3 m"
    enabled-transports : [akka.remote.dot-netty.tcp]
    adapters : {
    gremlin : Akka.Remote.Transport.FailureInjectorProvider,Akka.Remote
    trttl : Akka.Remote.Transport.ThrottlerProvider,Akka.Remote
    }
    helios : {
    tcp : {
    transport-class : "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote.Transport.Helios"
    }
    }
    dot-netty : {
    tcp : {
    transport-class : Akka.Remote.Transport.DotNetty.TcpTransport,Akka.Remote
    applied-adapters : []
    transport-protocol : tcp
    byte-order : little-endian
    port : 4062
    public-port : 0
    hostname : 
    public-hostname : 127.0.0.1
    dns-use-ipv6 : false
    enforce-ip-family : false
    enable-ssl : false
    enable-backwards-compatibility : false
    connection-timeout : "15 s"
    batching : {
    enabled : true
    max-pending-writes : 30
    max-pending-bytes : 16k
    flush-interval : 40ms
    }
    use-dispatcher-for-io : 
    write-buffer-high-water-mark : 0b
    write-buffer-low-water-mark : 0b
    send-buffer-size : 256000b
    receive-buffer-size : 256000b
    maximum-frame-size : 128000b
    backlog : 4096
    tcp-nodelay : on
    tcp-keepalive : on
    tcp-reuse-addr : off-for-windows
    server-socket-worker-pool : {
    pool-size-min : 2
    pool-size-factor : 1.0
    pool-size-max : 2
    }
    client-socket-worker-pool : {
    pool-size-min : 2
    pool-size-factor : 1.0
    pool-size-max : 2
    }
    }
    udp : {
    transport-protocol : udp
    }
    ssl : {
    certificate : {
    path : 
    password : 
    use-thumprint-over-file : false
    thumpbrint : 
    store-name : 
    store-location : current-user
    }
    suppress-validation : false
    }
    }
    gremlin : {
    debug : off
    }
    default-remote-dispatcher : {
    type : ForkJoinDispatcher
    executor : fork-join-executor
    dedicated-thread-pool : {
    thread-count : 4
    }
    }
    backoff-remote-dispatcher : {
    type : ForkJoinDispatcher
    executor : fork-join-executor
    dedicated-thread-pool : {
    thread-count : 4
    }
    }
    }
    cluster : {
    seed-nodes : ["akka.tcp://mls-cluster-system@127.0.0.1:4062"]
    seed-node-timeout : 5s
    retry-unsuccessful-join-after : 10s
    shutdown-after-unsuccessful-join-seed-nodes : off
    auto-down-unreachable-after : 10s
    down-removal-margin : off
    downing-provider-class : 
    allow-weakly-up-members : on
    roles : [lighthouse]
    run-coordinated-shutdown-when-down : on
    role : {
    }
    min-nr-of-members : 1
    log-info : on
    log-info-verbose : off
    periodic-tasks-initial-delay : 1s
    gossip-interval : 5s
    gossip-time-to-live : 30s
    leader-actions-interval : 1s
    unreachable-nodes-reaper-interval : 1s
    publish-stats-interval : off
    use-dispatcher : 
    gossip-different-view-probability : 0.8
    reduce-gossip-different-view-probability : 400
    failure-detector : {
    implementation-class : "Akka.Remote.PhiAccrualFailureDetector, Akka.Remote"
    heartbeat-interval : 5s
    threshold : 8.0
    max-sample-size : 1000
    min-std-deviation : "100 ms"
    acceptable-heartbeat-pause : 50s
    monitored-by-nr-of-members : 9
    expected-response-after : "1 s"
    }
    split-brain-resolver : {
    active-strategy : off
    stable-after : 20s
    static-quorum : {
    quorum-size : undefined
    role : 
    }
    keep-majority : {
    role : 
    }
    keep-oldest : {
    down-if-alone : on
    role : 
    }
    keep-refree : {
    address : 
    down-all-if-less-than-nodes : 1
    }
    }
    scheduler : {
    tick-duration : 33ms
    ticks-per-wheel : 512
    }
    debug : {
    verbose-heartbeat-logging : off
    verbose-receive-gossip-logging : off
    }
    client : {
    receptionist : {
    name : receptionist
    role : lighthouse
    number-of-contacts : 3
    response-tunnel-receive-timeout : 1d
    use-dispatcher : 
    heartbeat-interval : 2s
    acceptable-heartbeat-pause : 13s
    failure-detection-interval : 2s
    }
    initial-contacts : []
    establishing-get-contacts-interval : 3s
    refresh-contacts-interval : 60s
    heartbeat-interval : 2s
    acceptable-heartbeat-pause : 13s
    buffer-size : 1000
    reconnect-timeout : off
    }
    pub-sub : {
    name : distributedPubSubMediator
    role : 
    routing-logic : random
    gossip-interval : 1s
    removed-time-to-live : 120s
    max-delta-elements : 3000
    use-dispatcher : 
    }
    }
    service : {
    name : Mirero.MLS.TestLightHouse4061
    description : Mirero.MLS.TestLightHouse.4061
    }
    system : {
    name : mls-cluster-testlighthouse-4061
    }
    }
    lighthouse : {
    actorsystem : mls-cluster-system
    }
    petabridge : {
    cmd : {
    host : 0.0.0.0
    port : 9115
    }
    }

[2020-10-23 09:18:22.8135][ INFO][P:13308][T: 1] The Mirero.MLS.TestLightHouse4061 service is now running, press Control+C to exit. [2020-10-23 09:18:22.8305][ INFO][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/user/petabridge.cmd petabridge.cmd host bound to [0.0.0.0:9115] [2020-10-23 09:18:22.8305][ INFO][P:13308][T: 22] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] is JOINING itself (with roles [lighthouse]) and forming a new cluster [2020-10-23 09:18:22.8435][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - is the new leader among reachable nodes (more leaders may exist) [2020-10-23 09:18:22.8655][ INFO][P:13308][T: 5] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [Up] [2020-10-23 09:18:30.8114][DEBUG][P:13308][T: 19] remoting (akka://mls-cluster-system) Associated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:3893 [2020-10-23 09:18:30.9214][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Received InitJoin message from [[akka.tcp://mls-cluster-system@127.0.0.1:3893/system/cluster/core/daemon/joinSeedNodeProcess-1#928698614]] to [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:30.9214][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Sending InitJoinNack message from node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [[akka.tcp://mls-cluster-system@127.0.0.1:3893/system/cluster/core/daemon/joinSeedNodeProcess-1#928698614]] [2020-10-23 09:18:31.0354][ INFO][P:13308][T: 19] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] is JOINING, roles [api] [2020-10-23 09:18:31.2664][DEBUG][P:13308][T: 22] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:31.9224][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:3893] to [Up] [2020-10-23 09:18:32.8939][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:32.9409][ INFO][P:13308][T: 9] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - is no longer leader [2020-10-23 09:18:37.7321][DEBUG][P:13308][T: 4] remoting (akka://mls-cluster-system) Associated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:3895 [2020-10-23 09:18:37.8771][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Received InitJoin message from [[akka.tcp://mls-cluster-system@127.0.0.1:3895/system/cluster/core/daemon/joinSeedNodeProcess-1#370319934]] to [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:37.8771][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Sending InitJoinNack message from node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [[akka.tcp://mls-cluster-system@127.0.0.1:3895/system/cluster/core/daemon/joinSeedNodeProcess-1#370319934]] [2020-10-23 09:18:37.9731][ INFO][P:13308][T: 9] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:3895] is JOINING, roles [role1] [2020-10-23 09:18:38.1161][DEBUG][P:13308][T: 21] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:40.7272][DEBUG][P:13308][T: 26] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:42.9415][DEBUG][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:45.4732][DEBUG][P:13308][T: 5] remoting (akka://mls-cluster-system) Associated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:8112 [2020-10-23 09:18:45.6072][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Received InitJoin message from [[akka.tcp://mls-cluster-system@127.0.0.1:8112/system/cluster/core/daemon/joinSeedNodeProcess-1#713857872]] to [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:45.6072][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Sending InitJoinNack message from node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [[akka.tcp://mls-cluster-system@127.0.0.1:8112/system/cluster/core/daemon/joinSeedNodeProcess-1#713857872]] [2020-10-23 09:18:45.6392][DEBUG][P:13308][T: 30] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:45.7042][ INFO][P:13308][T: 24] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:8112] is JOINING, roles [role2] [2020-10-23 09:18:45.8442][DEBUG][P:13308][T: 24] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:18:47.9410][DEBUG][P:13308][T: 21] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:50.7473][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:51.9473][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Associated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:8113 [2020-10-23 09:18:52.0973][ INFO][P:13308][T: 8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Received InitJoin message from [[akka.tcp://mls-cluster-system@127.0.0.1:8113/system/cluster/core/daemon/joinSeedNodeProcess-1#203439572]] to [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:52.0973][ INFO][P:13308][T: 8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Sending InitJoinNack message from node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [[akka.tcp://mls-cluster-system@127.0.0.1:8113/system/cluster/core/daemon/joinSeedNodeProcess-1#203439572]] [2020-10-23 09:18:52.1923][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:8113] is JOINING, roles [role3] [2020-10-23 09:18:52.3333][DEBUG][P:13308][T: 23] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8113, 2078514987)] [2020-10-23 09:18:52.9553][DEBUG][P:13308][T: 22] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:18:55.6648][DEBUG][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:57.7278][DEBUG][P:13308][T: 22] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:57.8228][DEBUG][P:13308][T: 26] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:57.9648][DEBUG][P:13308][T: 21] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8113, 2078514987)] [2020-10-23 09:19:01.8677][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8113, 2078514987)] [2020-10-23 09:19:02.9681][DEBUG][P:13308][T: 30] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:19:05.6717][DEBUG][P:13308][T: 24] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:19:12.9930][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:19:18.0291][DEBUG][P:13308][T: 21] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:19:20.4801][DEBUG][P:13308][T: 26] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:19:22.7684][DEBUG][P:13308][T: 24] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:35:03.0348][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:35:04.8020][DEBUG][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:35:05.6941][DEBUG][P:13308][T: 8] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:35:05.9421][ INFO][P:13308][T: 22] Control+C detected, attempting to stop service. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-service-unbind] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-unbind] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-requests-done] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-stop] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-cluster-shutdown] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-sharding-shutdown-region] with [0] tasks. [2020-10-23 09:35:05.9531][DEBUG][P:13308][T: 4] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-leave] with [1] tasks: [leave] [2020-10-23 09:35:05.9531][ INFO][P:13308][T: 20] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Marked address [akka.tcp://mls-cluster-system@127.0.0.1:4062] as [Leaving] [2020-10-23 09:35:05.9531][DEBUG][P:13308][T: 5] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-exiting] with [1] tasks: [wait-exiting] [2020-10-23 09:35:06.0446][DEBUG][P:13308][T: 5] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:35:08.5166][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3894-1 Association between local [tcp://mls-cluster-system@0.0.0.0:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3894] was disassociated because the ProtocolStateActor failed: Shutdown [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon "Couldn't establish a causal relationship between "remote" gossip and "local" gossip - Remote[Gossip(members = [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3893, Uid=1373298527 status = Exiting, role=[api], upNumber=2), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3895, Uid=2141762004 status = Exiting, role=[role1], upNumber=3), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Up, role=[lighthouse], upNumber=1), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8112, Uid=1552571697 status = Exiting, role=[role2], upNumber=4), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8113, Uid=2078514987 status = Exiting, role=[role3], upNumber=5)], overview = GossipOverview(seen=[], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->7, B0FD6E6E85BC21C543C59A240889BC7C->6, BFDECCB8DF57F49EDBD5FF0F943FF688->1)] - Local[Gossip(members = [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3893, Uid=1373298527 status = Leaving, role=[api], upNumber=2), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3895, Uid=2141762004 status = Leaving, role=[role1], upNumber=3), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Leaving, role=[lighthouse], upNumber=1), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8112, Uid=1552571697 status = Leaving, role=[role2], upNumber=4), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8113, Uid=2078514987 status = Leaving, role=[role3], upNumber=5)], overview = GossipOverview(seen=[UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:4062, 1685981556), UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->8, B0FD6E6E85BC21C543C59A240889BC7C->5, BFDECCB8DF57F49EDBD5FF0F943FF688->1)] - merged them into [Gossip(members = [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3893, Uid=1373298527 status = Exiting, role=[api], upNumber=2), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3895, Uid=2141762004 status = Exiting, role=[role1], upNumber=3), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Leaving, role=[lighthouse], upNumber=1), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8112, Uid=1552571697 status = Exiting, role=[role2], upNumber=4), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8113, Uid=2078514987 status = Exiting, role=[role3], upNumber=5)], overview = GossipOverview(seen=[], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->8, B0FD6E6E85BC21C543C59A240889BC7C->6, BFDECCB8DF57F49EDBD5FF0F943FF688->1)]" [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://mls-cluster-system@127.0.0.1:3893] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. [2020-10-23 09:35:08.5316][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting confirmed [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:35:08.5316][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting confirmed [akka.tcp://mls-cluster-system@127.0.0.1:3895] [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3896-2 Association between local [tcp://mls-cluster-system@0.0.0.0:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3896] was disassociated because the ProtocolStateActor failed: Shutdown [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://mls-cluster-system@127.0.0.1:3895] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [1] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [2] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 5] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3898-3 Association between local [tcp://mls-cluster-system@0.0.0.0:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3898] was disassociated because the ProtocolStateActor failed: Shutdown [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 27] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3901-4 Association between local [tcp://mls-cluster-system@0.0.0.0:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3901] was disassociated because the ProtocolStateActor failed: Shutdown [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 4] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting confirmed [akka.tcp://mls-cluster-system@127.0.0.1:8112] [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 27] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8113, 2078514987)] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting confirmed [akka.tcp://mls-cluster-system@127.0.0.1:8113] [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://mls-cluster-system@127.0.0.1:8112] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://mls-cluster-system@127.0.0.1:8113] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 8] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3893-1/endpointWriter Disassociated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:3893 [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8113-4/endpointWriter Disassociated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:8113 [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8112-3/endpointWriter Disassociated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:8112 [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3895-2/endpointWriter Disassociated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:3895 [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3895-2 Removing receive buffers for [akka.tcp://mls-cluster-system@127.0.0.1:4062]->[akka.tcp://mls-cluster-system@127.0.0.1:3895] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3893-1 Removing receive buffers for [akka.tcp://mls-cluster-system@127.0.0.1:4062]->[akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8112-3 Removing receive buffers for [akka.tcp://mls-cluster-system@127.0.0.1:4062]->[akka.tcp://mls-cluster-system@127.0.0.1:8112] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8113-4 Removing receive buffers for [akka.tcp://mls-cluster-system@127.0.0.1:4062]->[akka.tcp://mls-cluster-system@127.0.0.1:8113] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 9] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [3] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [4] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - is the new leader among reachable nodes (more leaders may exist) [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 20] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-exiting-done] with [1] tasks: [exiting-completed] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [Exiting] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is removing confirmed Exiting node [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is removing confirmed Exiting node [akka.tcp://mls-cluster-system@127.0.0.1:8112] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is removing confirmed Exiting node [akka.tcp://mls-cluster-system@127.0.0.1:8113] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is removing confirmed Exiting node [akka.tcp://mls-cluster-system@127.0.0.1:3895] [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 26] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:4062, 1685981556)] - Gossip exiting members [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Exiting, role=[lighthouse], upNumber=1)] to the two oldest (per role) [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Exiting, role=[lighthouse], upNumber=1)] (singleton optimization). [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting completed. [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Shutting down... [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Successfully shut down [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 30] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-shutdown] with [1] tasks: [wait-shutdown] [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-actor-system-terminate] with [0] tasks. [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [actor-system-terminate] with [1] tasks: [terminate-system] [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 23] ActorSystem(mls-cluster-system) System shutdown initiated [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 24] akka.tcp://mls-cluster-system@127.0.0.1:4062/user/petabridge.cmd Shutting down petabridge.cmd host... [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/IO-TCP/$a Unbinding endpoint 0.0.0.0:9115 [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 20] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/IO-TCP/$a Unbound endpoint 0.0.0.0:9115, stopping listener [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 24] remoting-terminator Shutting down remote daemon. [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 38] remoting-terminator Remote daemon shut down; proceeding with flushing remote transports. [2020-10-23 09:35:08.7356][ INFO][P:13308][T: 5] remoting (akka://mls-cluster-system) Remoting shut down [2020-10-23 09:35:08.7356][ INFO][P:13308][T: 37] remoting-terminator Remoting shut down. [2020-10-23 09:35:08.7356][DEBUG][P:13308][T: 39] EventStream Shutting down: StandardOutLogger started [2020-10-23 09:35:08.7646][ INFO][P:13308][T: 1] The Mirero.MLS.TestLightHouse4061 service has stopped.

#### TestClusterApi.log

[2020/10/23 09:18:30.051][ 1][ INFO][Topshelf.HostFactory] Configuration Result: [Success] Name Mirero.MLS.TestClusterApi [Success] ServiceName Mirero.MLS.TestClusterApi [2020/10/23 09:18:30.106][ 1][ INFO][Topshelf.HostConfigurators.HostConfiguratorImpl] Topshelf v4.2.1.215, .NET Framework v4.0.30319.42000 [2020/10/23 09:18:30.340][ 4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Starting remoting [2020/10/23 09:18:30.518][ 4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Remoting started; listening on addresses : [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020/10/23 09:18:30.518][ 4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Remoting now listens on addresses: [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020/10/23 09:18:30.544][ 5][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Starting up... [2020/10/23 09:18:30.563][ 9][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Started up successfully [2020/10/23 09:18:30.580][ 10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] akka : { version : "0.0.1 Akka" home : loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"] loggers-dispatcher : akka.actor.default-dispatcher logger-startup-timeout : 5s logger-async-start : false loglevel : DEBUG suppress-json-serializer-warning : on stdout-loglevel : DEBUG log-config-on-start : on log-dead-letters : 10 log-dead-letters-during-shutdown : off log-dead-letters-suspend-duration : "5 minutes" extensions : ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"] daemonic : off actor : { provider : cluster guardian-supervisor-strategy : "Mirero.MLS.TestClusterApi.ActorService.UserSupervisorStrategy, Mirero.MLS.TestClusterApi" creation-timeout : 20s reaper-interval : 5 serialize-messages : off serialize-creators : off unstarted-push-timeout : 10s ask-timeout : infinite typed : { timeout : 5 } inbox : { inbox-size : 1000 default-timeout : 5s } router : { type-mapping : { from-code : Akka.Routing.NoRouter round-robin-pool : Akka.Routing.RoundRobinPool round-robin-group : Akka.Routing.RoundRobinGroup random-pool : Akka.Routing.RandomPool random-group : Akka.Routing.RandomGroup smallest-mailbox-pool : Akka.Routing.SmallestMailboxPool broadcast-pool : Akka.Routing.BroadcastPool broadcast-group : Akka.Routing.BroadcastGroup scatter-gather-pool : Akka.Routing.ScatterGatherFirstCompletedPool scatter-gather-group : Akka.Routing.ScatterGatherFirstCompletedGroup consistent-hashing-pool : Akka.Routing.ConsistentHashingPool consistent-hashing-group : Akka.Routing.ConsistentHashingGroup tail-chopping-pool : Akka.Routing.TailChoppingPool tail-chopping-group : Akka.Routing.TailChoppingGroup } } deployment : { default : { dispatcher : mailbox : router : from-code nr-of-instances : 1 within : "5 s" virtual-nodes-factor : 10 routees : { paths : [] } resizer : { enabled : off lower-bound : 1 upper-bound : 10 pressure-threshold : 1 rampup-rate : 0.2 backoff-threshold : 0.3 backoff-rate : 0.1 messages-per-resize : 10 } remote : target : { nodes : [] } metrics-selector : mix cluster : { enabled : off max-nr-of-instances-per-node : 1 max-total-nr-of-instances : 10000 allow-local-routees : on use-role : } } /ApiMasterActor : { } /ApiMasterActor//TestClusterRole1HelloActor : { router : round-robin-pool nr-of-instances : 100 cluster : { enabled : on max-nr-of-instances-per-node : 1 allow-local-routees : off use-role : role1 } } /ApiMasterActor//TestClusterRole2HelloActor : { router : round-robin-pool nr-of-instances : 100 cluster : { enabled : on max-nr-of-instances-per-node : 1 allow-local-routees : off use-role : role2 } } /ApiMasterActor/*/TestClusterRole3HelloActor : { router : round-robin-pool nr-of-instances : 100 cluster : { enabled : on max-nr-of-instances-per-node : 1 allow-local-routees : off use-role : role3 } } } synchronized-dispatcher : { type : SynchronizedDispatcher executor : current-context-executor throughput : 10 } task-dispatcher : { type : TaskDispatcher executor : task-executor throughput : 30 } default-fork-join-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor throughput : 30 dedicated-thread-pool : { thread-count : 3 threadtype : background } } default-dispatcher : { type : Dispatcher executor : default-executor default-executor : { } thread-pool-executor : { } fork-join-executor : { dedicated-thread-pool : { thread-count : 3 threadtype : background } } current-context-executor : { } shutdown-timeout : 1s throughput : 30 throughput-deadline-time : 0ms attempt-teamwork : on mailbox-requirement : } default-mailbox : { mailbox-type : Akka.Dispatch.UnboundedMailbox mailbox-capacity : 1000 mailbox-push-timeout-time : 10s stash-capacity : -1 } mailbox : { requirements : { Akka.Dispatch.IUnboundedMessageQueueSemantics : akka.actor.mailbox.unbounded-queue-based Akka.Dispatch.IBoundedMessageQueueSemantics : akka.actor.mailbox.bounded-queue-based Akka.Dispatch.IDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based Akka.Dispatch.IUnboundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based Akka.Dispatch.IBoundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.bounded-deque-based Akka.Dispatch.IMultipleConsumerSemantics : akka.actor.mailbox.unbounded-queue-based Akka.Event.ILoggerMessageQueueSemantics : akka.actor.mailbox.logger-queue } unbounded-queue-based : { mailbox-type : Akka.Dispatch.UnboundedMailbox } bounded-queue-based : { mailbox-type : Akka.Dispatch.BoundedMailbox } unbounded-deque-based : { mailbox-type : Akka.Dispatch.UnboundedDequeBasedMailbox } bounded-deque-based : { mailbox-type : Akka.Dispatch.BoundedDequeBasedMailbox } logger-queue : { mailbox-type : Akka.Event.LoggerMailboxType } } debug : { receive : off autoreceive : off lifecycle : off fsm : off event-stream : off unhandled : on router-misconfiguration : off } serializers : { json : "Akka.Serialization.NewtonSoftJsonSerializer, Akka" bytes : "Akka.Serialization.ByteArraySerializer, Akka" akka-containers : "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" akka-misc : "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" primitive : "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" proto : "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" daemon-create : "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" akka-system-msg : "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" akka-cluster : "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" akka-cluster-client : "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" akka-pubsub : "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" } serialization-bindings : { System.Byte[] : bytes System.Object : json "Akka.Actor.ActorSelectionMessage, Akka" : akka-containers "Akka.Remote.DaemonMsgCreate, Akka.Remote" : daemon-create "Google.Protobuf.IMessage, Google.Protobuf" : proto "Akka.Actor.Identify, Akka" : akka-misc "Akka.Actor.ActorIdentity, Akka" : akka-misc "Akka.Actor.IActorRef, Akka" : akka-misc "Akka.Actor.PoisonPill, Akka" : akka-misc "Akka.Actor.Kill, Akka" : akka-misc "Akka.Actor.RemoteScope, Akka" : akka-misc "Akka.Routing.FromConfig, Akka" : akka-misc "Akka.Routing.DefaultResizer, Akka" : akka-misc "Akka.Routing.RoundRobinPool, Akka" : akka-misc "Akka.Routing.BroadcastPool, Akka" : akka-misc "Akka.Routing.RandomPool, Akka" : akka-misc "Akka.Routing.ScatterGatherFirstCompletedPool, Akka" : akka-misc "Akka.Routing.TailChoppingPool, Akka" : akka-misc "Akka.Routing.ConsistentHashingPool, Akka" : akka-misc "Akka.Configuration.Config, Akka" : akka-misc "Akka.Remote.RemoteWatcher+Heartbeat, Akka.Remote" : akka-misc "Akka.Remote.RemoteWatcher+HeartbeatRsp, Akka.Remote" : akka-misc "Akka.Remote.Routing.RemoteRouterConfig, Akka.Remote" : akka-misc "Akka.Dispatch.SysMsg.SystemMessage, Akka" : akka-system-msg System.String : primitive System.Int32 : primitive System.Int64 : primitive "Akka.Cluster.IClusterMessage, Akka.Cluster" : akka-cluster "Akka.Cluster.Routing.ClusterRouterPool, Akka.Cluster" : akka-cluster "Akka.Cluster.Tools.Client.IClusterClientMessage, Akka.Cluster.Tools" : akka-cluster-client "Akka.Cluster.Tools.PublishSubscribe.IDistributedPubSubMessage, Akka.Cluster.Tools" : akka-pubsub "Akka.Cluster.Tools.PublishSubscribe.Internal.SendToOneSubscriber, Akka.Cluster.Tools" : akka-pubsub } serialization-identifiers : { "Akka.Serialization.ByteArraySerializer, Akka" : 4 "Akka.Serialization.NewtonSoftJsonSerializer, Akka" : 1 "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" : 2 "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" : 3 "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" : 6 "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" : 16 "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" : 17 "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" : 22 "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" : 5 "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" : 15 "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" : 9 } serialization-settings : { } } scheduler : { tick-duration : 10ms ticks-per-wheel : 512 implementation : Akka.Actor.HashedWheelTimerScheduler shutdown-timeout : 5s } io : { pinned-dispatcher : { type : PinnedDispatcher executor : fork-join-executor } tcp : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } disabled-buffer-pool : { class : "Akka.IO.Buffers.DisabledBufferPool, Akka" buffer-size : 512 } buffer-pool : akka.io.tcp.disabled-buffer-pool max-channels : 256000 selector-association-retries : 10 batch-accept-limit : 10 register-timeout : 5s max-received-message-size : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher file-io-dispatcher : akka.actor.default-dispatcher file-io-transferTo-limit : 524288 finish-connect-retries : 5 windows-connection-abort-workaround-enabled : off outgoing-socket-force-ipv4 : false } udp : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } buffer-pool : akka.io.udp.direct-buffer-pool nr-of-socket-async-event-args : 32 max-channels : 4096 select-timeout : infinite selector-association-retries : 10 receive-throughput : 3 direct-buffer-size : 18432 direct-buffer-pool-limit : 1000 received-message-size-limit : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher } udp-connected : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } buffer-pool : akka.io.udp-connected.direct-buffer-pool nr-of-socket-async-event-args : 32 max-channels : 4096 select-timeout : infinite selector-association-retries : 10 receive-throughput : 3 direct-buffer-size : 18432 direct-buffer-pool-limit : 1000 received-message-size-limit : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher } dns : { dispatcher : akka.actor.default-dispatcher resolver : inet-address inet-address : { provider-object : Akka.IO.InetAddressDnsProvider positive-ttl : 30s negative-ttl : 10s cache-cleanup-interval : 120s use-ipv6 : true } } } coordinated-shutdown : { default-phase-timeout : "5 s" terminate-actor-system : on exit-clr : off run-by-clr-shutdown-hook : on run-by-actor-system-terminate : on phases : { before-service-unbind : { } service-unbind : { depends-on : [before-service-unbind] } service-requests-done : { depends-on : [service-unbind] } service-stop : { depends-on : [service-requests-done] } before-cluster-shutdown : { depends-on : [service-stop] } cluster-sharding-shutdown-region : { timeout : "10 s" depends-on : [before-cluster-shutdown] } cluster-leave : { depends-on : [cluster-sharding-shutdown-region] } cluster-exiting : { timeout : "10 s" depends-on : [cluster-leave] } cluster-exiting-done : { depends-on : [cluster-exiting] } cluster-shutdown : { depends-on : [cluster-exiting-done] } before-actor-system-terminate : { depends-on : [cluster-shutdown] } actor-system-terminate : { timeout : "10 s" depends-on : [before-actor-system-terminate] } } } remote : { startup-timeout : "10 s" shutdown-timeout : "10 s" flush-wait-on-shutdown : "2 s" use-passive-connections : on backoff-interval : "0.05 s" command-ack-timeout : "30 s" handshake-timeout : "15 s" use-dispatcher : akka.remote.default-remote-dispatcher untrusted-mode : off trusted-selection-paths : [] require-cookie : off secure-cookie : log-received-messages : off log-sent-messages : off log-remote-lifecycle-events : on log-frame-size-exceeding : off log-buffer-size-exceeding : 50000 transport-failure-detector : { implementation-class : Akka.Remote.DeadlineFailureDetector,Akka.Remote heartbeat-interval : 5s acceptable-heartbeat-pause : 50s } watch-failure-detector : { implementation-class : Akka.Remote.PhiAccrualFailureDetector,Akka.Remote heartbeat-interval : "1 s" threshold : 10.0 max-sample-size : 200 min-std-deviation : "100 ms" acceptable-heartbeat-pause : "10 s" unreachable-nodes-reaper-interval : 1s expected-response-after : "1 s" } retry-gate-closed-for : "5 s" prune-quarantine-marker-after : "5 d" quarantine-after-silence : "2 d" system-message-buffer-size : 20000 system-message-ack-piggyback-timeout : "0.3 s" resend-interval : "2 s" resend-limit : 200 initial-system-message-delivery-timeout : "3 m" enabled-transports : [akka.remote.dot-netty.tcp] adapters : { gremlin : Akka.Remote.Transport.FailureInjectorProvider,Akka.Remote trttl : Akka.Remote.Transport.ThrottlerProvider,Akka.Remote } helios : { tcp : { transport-class : "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote.Transport.Helios" } } dot-netty : { tcp : { transport-class : Akka.Remote.Transport.DotNetty.TcpTransport,Akka.Remote applied-adapters : [] transport-protocol : tcp byte-order : little-endian port : 0 public-port : 0 hostname : public-hostname : 127.0.0.1 dns-use-ipv6 : false enforce-ip-family : false enable-ssl : false enable-backwards-compatibility : false connection-timeout : "15 s" batching : { enabled : true max-pending-writes : 30 max-pending-bytes : 16k flush-interval : 40ms } use-dispatcher-for-io : write-buffer-high-water-mark : 0b write-buffer-low-water-mark : 0b send-buffer-size : 2097152b receive-buffer-size : 2097152b maximum-frame-size : 1048576b backlog : 4096 tcp-nodelay : on tcp-keepalive : on tcp-reuse-addr : off-for-windows server-socket-worker-pool : { pool-size-min : 2 pool-size-factor : 1.0 pool-size-max : 2 } client-socket-worker-pool : { pool-size-min : 2 pool-size-factor : 1.0 pool-size-max : 2 } } udp : { transport-protocol : udp } ssl : { certificate : { path : password : use-thumprint-over-file : false thumpbrint : store-name : store-location : current-user } suppress-validation : false } } gremlin : { debug : off } default-remote-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor dedicated-thread-pool : { thread-count : 4 } } backoff-remote-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor dedicated-thread-pool : { thread-count : 4 } } } cluster : { seed-nodes : ["akka.tcp://mls-cluster-system@127.0.0.1:4062"] seed-node-timeout : 5s retry-unsuccessful-join-after : 10s shutdown-after-unsuccessful-join-seed-nodes : off auto-down-unreachable-after : 10s down-removal-margin : off downing-provider-class : allow-weakly-up-members : on roles : [api] run-coordinated-shutdown-when-down : on role : { } min-nr-of-members : 1 log-info : on log-info-verbose : off periodic-tasks-initial-delay : 1s gossip-interval : 5s gossip-time-to-live : 30s leader-actions-interval : 1s unreachable-nodes-reaper-interval : 1s publish-stats-interval : off use-dispatcher : gossip-different-view-probability : 0.8 reduce-gossip-different-view-probability : 400 failure-detector : { implementation-class : "Akka.Remote.PhiAccrualFailureDetector, Akka.Remote" heartbeat-interval : 5s threshold : 8.0 max-sample-size : 1000 min-std-deviation : "100 ms" acceptable-heartbeat-pause : 50s monitored-by-nr-of-members : 9 expected-response-after : "1 s" } split-brain-resolver : { active-strategy : off stable-after : 20s static-quorum : { quorum-size : undefined role : } keep-majority : { role : } keep-oldest : { down-if-alone : on role : } keep-refree : { address : down-all-if-less-than-nodes : 1 } } scheduler : { tick-duration : 33ms ticks-per-wheel : 512 } debug : { verbose-heartbeat-logging : off verbose-receive-gossip-logging : off } client : { receptionist : { name : receptionist role : number-of-contacts : 3 response-tunnel-receive-timeout : 30s use-dispatcher : heartbeat-interval : 2s acceptable-heartbeat-pause : 13s failure-detection-interval : 2s } initial-contacts : [] establishing-get-contacts-interval : 3s refresh-contacts-interval : 60s heartbeat-interval : 2s acceptable-heartbeat-pause : 13s buffer-size : 1000 reconnect-timeout : off } pub-sub : { name : distributedPubSubMediator role : routing-logic : random gossip-interval : 1s removed-time-to-live : 120s max-delta-elements : 3000 use-dispatcher : } singleton : { singleton-name : ClusterSingleton role : role1 hand-over-retry-interval : 1s min-number-of-hand-over-retries : 10 } } service : { name : Mirero.MLS.TestClusterApi description : Mirero.MLS.TestClusterApi } system : { name : mls-cluster-system } } petabridge : { cmd : { host : 0.0.0.0 port : 9111 log-palettes-on-startup : on } } mls : { service : { dbConnect : { ip : 192.168.60.193 port : 1521 service : MLSDB user : mlsuser password : mlsuser } } }

[2020/10/23 09:18:30.606][ 1][ INFO][Topshelf.Hosts.ConsoleRunHost] The Mirero.MLS.TestClusterApi service is now running, press Control+C to exit. [2020/10/23 09:18:31.395][ 10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - path:
[2020/10/23 09:18:31.395][ 10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - frequency: 0 [2020/10/23 09:18:31.395][ 10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - active: False [2020/10/23 09:18:31.395][ 10][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] petabridge.cmd host bound to [0.0.0.0:9111] [2020/10/23 09:18:31.403][ 10][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Welcome from [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020/10/23 09:18:32.867][ 27][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton manager started singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton]
[2020/10/23 09:18:32.867][ 27][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [Start -> Oldest] Akka.Cluster.Tools.Singleton.Uninitialized [2020/10/23 09:18:33.626][ 24][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - is the new leader among reachable nodes (more leaders may exist) [2020/10/23 09:18:40.751][ 10][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:3895] to [Up] [2020/10/23 09:18:50.907][ 25][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8112] to [Up] [2020/10/23 09:18:55.983][ 21][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8113] to [WeaklyUp] [2020/10/23 09:18:58.454][ 25][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] Received petabridge.cmd connection from client [127.0.0.1:3908] [2020/10/23 09:19:09.655][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$a/$c[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:19:09.675][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$a/$c[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:19:11.206][ 7][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8113] to [Up] [2020/10/23 09:19:15.035][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$b/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:19:15.194][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$b/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:19:17.039][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:19:17.118][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:19:17.294][ 5][ INFO]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Akka.Actor.EmptyLocalActorRef] Message [ActorIdentity] from akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 to akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a was not delivered. [1] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020/10/23 09:19:18.779][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:19:18.938][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:19:20.511][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$e/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:19:20.638][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$e/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:19:22.575][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$f/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:19:22.735][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$f/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:19:26.900][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$g/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:19:27.028][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$g/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:19:28.735][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$h/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:19:28.894][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$h/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:19:30.301][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$i/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:19:30.365][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$i/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:19:31.844][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$j/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:19:31.924][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$j/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:19:33.475][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$k/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:19:33.635][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$k/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:20:09.618][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$l/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:09.674][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$l/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:20:11.763][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$m/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:11.842][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$m/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:20:13.840][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$n/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:20:13.918][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$n/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:16.127][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$o/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:20:16.205][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$o/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:17.968][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$p/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:20:18.126][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$p/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:19.697][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:20:19.851][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:21.315][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$r/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:20:21.376][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$r/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:20:24.216][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$s/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:20:24.278][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$s/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:20:26.782][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$t/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:20:26.859][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$t/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:28.825][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$u/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:28.979][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$u/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1, Subject=] [2020/10/23 09:20:30.991][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$v/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:20:31.070][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$v/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:20:34.214][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$w/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:20:34.294][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$w/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:22:05.036][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$x/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:22:05.193][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$x/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:22:08.407][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:22:08.486][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:22:11.401][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$z/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:22:11.480][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$z/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:22:14.348][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$A/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:22:14.427][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$A/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:22:17.155][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$B/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:22:17.234][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$B/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:22:32.395][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$C/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:22:32.471][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$C/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:22:34.026][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$D/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:22:34.104][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$D/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:23:53.942][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$E/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:23:54.005][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$E/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:23:56.216][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$F/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:23:58.383][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$G/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:23:58.446][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$G/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:23:59.900][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$H/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:23:59.963][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$H/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:01.583][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$I/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:01.661][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$I/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:03.174][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$J/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:03.251][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$J/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:04.764][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$K/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:04.922][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$K/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:06.583][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$L/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:06.658][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$L/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:08.312][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$M/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:08.390][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$M/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:09.809][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$N/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:09.887][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$N/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:11.612][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$O/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:11.690][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$O/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:13.079][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$P/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:13.157][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$P/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:14.673][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:14.734][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:16.172][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$R/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:16.251][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$R/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:24.520][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$S/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:24.598][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$S/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:25.824][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$T/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:25.902][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$T/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:27.040][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$U/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:27.118][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$U/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:28.425][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$V/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:28.504][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$V/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:29.567][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$W/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:29.627][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$W/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:30.735][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$X/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:30.799][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$X/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:31.905][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:31.968][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:33.195][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Z/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:33.352][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Z/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1, Subject=] [2020/10/23 09:24:43.135][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$0/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:24:43.292][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$0/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:45.645][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$1/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:24:45.723][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$1/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:47.876][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$2/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:24:48.049][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$2/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:25:57.482][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$3/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:25:57.560][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$3/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:25:59.163][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$4/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:25:59.242][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$4/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:00.792][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$5/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:00.872][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$5/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:02.413][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$6/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:02.473][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$6/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:03.995][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$7/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:04.166][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$7/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:26:05.810][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$8/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:26:05.871][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$8/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:13.033][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$9/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:13.096][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$9/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:26:14.829][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$+/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:26:14.889][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$+/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:16.573][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$~/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:16.634][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$~/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:18.166][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$ab/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:18.246][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$ab/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:26:19.947][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$bb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:20.008][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$bb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:22.014][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$cb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:22.092][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$cb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:23.744][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$db/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:23.808][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$db/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]] [2020/10/23 09:26:25.759][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$eb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:25.835][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$eb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:27.673][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$fb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:26:27.750][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$fb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:29.202][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$gb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]] [2020/10/23 09:26:29.357][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$gb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]] [2020/10/23 09:35:05.318][ 22][ INFO][Topshelf.Hosts.ConsoleRunHost] Control+C detected, attempting to stop service. [2020/10/23 09:35:05.324][ 20][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Marked address [akka.tcp://mls-cluster-system@127.0.0.1:3893] as [Leaving] [2020/10/23 09:35:05.324][ 20][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Exited [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020/10/23 09:35:05.324][ 20][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Oldest observed OldestChanged: [akka.tcp://mls-cluster-system@127.0.0.1:3893 -> ] [2020/10/23 09:35:05.324][ 20][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [Oldest -> WasOldest] Akka.Cluster.Tools.Singleton.OldestData [2020/10/23 09:35:06.355][ 5][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton manager stopping singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton] [2020/10/23 09:35:06.355][ 5][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [WasOldest -> Stopping] Akka.Cluster.Tools.Singleton.WasOldestData [2020/10/23 09:35:06.355][ 5][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton] was terminated [2020/10/23 09:35:08.477][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:3893] to [Exiting] [2020/10/23 09:35:08.481][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:3895] to [Exiting] [2020/10/23 09:35:08.481][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8112] to [Exiting] [2020/10/23 09:35:08.481][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8113] to [Exiting] [2020/10/23 09:35:08.520][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Exiting completed. [2020/10/23 09:35:08.540][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Shutting down... [2020/10/23 09:35:08.682][ 1][ INFO][Topshelf.Hosts.ConsoleRunHost] The Mirero.MLS.TestClusterApi service has stopped.

kimbyungeun commented 3 years ago

@IgorFedchenko

hocon

TestClusterApi.hocon

petabridge.cmd 
{
    host = "0.0.0.0"
    port = 9111
    log-palettes-on-startup = on
}

akka 
{
    service.name = "TestClusterApi"
    service.description = "TestClusterApi"
    system.name = "mls-cluster-system"  

    stdout-loglevel = DEBUG
    loglevel = DEBUG
    log-config-on-start = on
    loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]

    actor
    {
        provider = cluster
        guardian-supervisor-strategy = "TestClusterApi.ActorService.UserSupervisorStrategy, TestClusterApi"

        debug
        {
            receive = off         
            autoreceive = off  
            lifecycle = off       
            event-stream = off
            unhandled = on    
        }

        deployment 
        {
            /ApiMasterActor 
            {

            }

            "/ApiMasterActor/*/TestClusterRole1HelloActor"
            {
                router = round-robin-pool
                nr-of-instances = 100

                cluster
                {
                    enabled = on
                    max-nr-of-instances-per-node = 1
                    allow-local-routees = off
                    use-role = role1
                }
            }

            "/ApiMasterActor/*/TestClusterRole2HelloActor"
            {
                router = round-robin-pool
                nr-of-instances = 100

                cluster
                {
                    enabled = on
                    max-nr-of-instances-per-node = 1
                    allow-local-routees = off
                    use-role = role2
                }
            }

            "/ApiMasterActor/*/TestClusterRole3HelloActor"
            {
                router = round-robin-pool
                nr-of-instances = 100

                cluster
                {
                    enabled = on
                    max-nr-of-instances-per-node = 1
                    allow-local-routees = off
                    use-role = role3
                }
            }
        }
    }

    remote
    {
        dot-netty.tcp 
        {
            public-hostname = 127.0.0.1
            port = 0

            send-buffer-size    = 2097152b  # 2MB
            receive-buffer-size = 2097152b  # 2MB
            maximum-frame-size  = 1048576b  # 1MB
        }

        transport-failure-detector
        {
            heartbeat-interval = 5s                
            acceptable-heartbeat-pause = 50s   
        }
    }

    extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]

    cluster 
    {
        seed-nodes = ["akka.tcp://mls-cluster-system@127.0.0.1:4062"]
        roles = [api]

        auto-down-unreachable-after = 10s 

        gossip-interval = 5s 
        gossip-time-to-live = 30s 

        failure-detector 
        {
            heartbeat-interval = 5s
            acceptable-heartbeat-pause = 50s
        }

        singleton
        {
            singleton-name = "ClusterSingleton"
            role = "role1"
            hand-over-retry-interval = 1s
            min-number-of-hand-over-retries = 10
        }

        #singleton-proxy 
        #{
        #   singleton-name = ${akka.cluster.singleton.singleton-name}
        #   role = "role1"
        #   singleton-identification-interval = 1s
        #   buffer-size = 1000
        #}
    }
}

Lighthouse.hocon

lighthouse
{
    actorsystem: "mls-cluster-system" #POPULATE NAME OF YOUR ACTOR SYSTEM HERE
}

petabridge.cmd
{
    host = "0.0.0.0"
    port = 9115
}

akka 
{
    service.name = "TestLightHouse4061"
    service.description = "TestLightHouse.4061"
    system.name = "mls-cluster-testlighthouse-4061"

    stdout-loglevel = DEBUG
    loglevel = DEBUG
    log-config-on-start = on
    loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]

    actor 
    {
        provider = cluster

        debug
        {  
            receive = off      
            autoreceive = off  
            lifecycle = off    
            event-stream = off
            unhandled = on
        }
    }

    remote 
    {
        log-remote-lifecycle-events = DEBUG

        dot-netty.tcp
        {
            public-hostname = 127.0.0.1
            port = 4062
        }

        transport-failure-detector
        {
            heartbeat-interval = 5s                
            acceptable-heartbeat-pause = 50s       
        }
    }

    extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]

    cluster 
    {
        seed-nodes = ["akka.tcp://mls-cluster-system@127.0.0.1:4062"] 

        roles = [lighthouse]
        auto-down-unreachable-after = 10s

        gossip-interval = 5s # how often should the node send out gossip information?
        gossip-time-to-live = 30s # discard incoming gossip messages if not handled within this duration

        client.receptionist
        {
            role = lighthouse

            response-tunnel-receive-timeout = 1d    # 1day
        }

        failure-detector 
        {
            heartbeat-interval = 5s
            acceptable-heartbeat-pause = 50s
        }
    }
}
IgorFedchenko commented 3 years ago

@kimbyungeun Ok, thanks! I was just hoping that you have a .net project/solution source you can share here to make reproduction simpler.

If you do not have any, we will try do make our own, this will just take more time.

kimbyungeun commented 3 years ago

@IgorFedchenko

Today I will separate the source code and upload it. Please wait a little.

Aaronontheweb commented 3 years ago

We appreicate your help @kimbyungeun !

kimbyungeun commented 3 years ago

@IgorFedchenko Upload the sample program. I hope it helps. ClusterSample.zip

Hello. (안녕하세요) @Aaronontheweb

I have 2 questions about the hocon configuration of our sample(ClusterSample.zip).

Q1


Q2 : acceptable-heartbeat-pause = 240s


IgorFedchenko commented 3 years ago

@kimbyungeun Thank you very much! I or another member of the team will check this out

kimbyungeun commented 3 years ago

@IgorFedchenko

I tested it with Akka.NET v1.4.12 version but it is the same.

Akka.Remote v1.4.12 Akka.Cluster v1.4.12 Petabridge.Cmd.Cluster v0.8.0.1 Petabridge.Cmd.Common v0.8.0.1 Petabridge.Cmd.Host v0.8.0.1

PS C:\Users\be.kim> pbm 127.0.0.1:9111
 ____      _        _          _     _
|  _ \ ___| |_ __ _| |__  _ __(_) __| | __ _  ___
| |_) / _ | __/ _` | '_ \| '__| |/ _` |/ _` |/ _ \
|  __|  __| || (_| | |_) | |  | | (_| | (_| |  __/
|_|   \___|\__\__,_|_.__/|_|  |_|\__,_|\__, |\___|
                                       |___/

petabridge.cmd (0.8.0.0)
Copyright 2017 - 2020, Petabridge®.

successfully connected to [::ffff:127.0.0.1]:9111
Commands downloaded from server. type `help` to see what's available
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$a
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$a/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$b
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$b/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$c
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$c/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$d
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$d/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$e
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$e/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$f
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$f/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$g
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$g/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm>
IgorFedchenko commented 3 years ago

@kimbyungeun Thanks, was thinking of checking that!

Aaronontheweb commented 3 years ago

@IgorFedchenko I think the issue here is we print out the results in any order when they arrive, so the output isn't sorted on a per-level basis in the hierarchy. i.e. all children at the same position in the hierarchy might print out differently each time it's run.

Answer would be to sort the actor paths I guess

IgorFedchenko commented 3 years ago

@Aaronontheweb If I understood the issue right, the problem is that child actors sometimes are not displayed at all. From the latest logs above: image

@kimbyungeun said that actors are running all the time, so they should be displayed all the time.

Right?

Aaronontheweb commented 3 years ago

Ah that makes sense @IgorFedchenko - we do time out the search after a while so that may be what is happening here.

IgorFedchenko commented 3 years ago

Hmm... I will check that - thanks for pointing the direction!

IgorFedchenko commented 3 years ago

@Aaronontheweb I was able to reproduce this issue. But I have a question to you.

Long story short: we have Akka.net cluster, 3 nodes (there are more, but this is not important). Cluster singleton named ApiMasterActor is created, and this singleton on start creates 3 child actors. Then first actor on scheduled time is sending message via IActorRef to second actor, second is sending message to third, third just logging a message. And so on each 5 seconds, timer -> 1 -> 2 -> 3.

Sometimes, Petabridge.Cmd does not give any c1 actor:

[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor

but - sometimes, there is some c1 actor popping up, here and there, as a child of this "hello" actors. Sometimes, as a child of one of this actors, sometimes as a child of two, or even all three of them, for example:

[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1

So, the question is: does Akka.NET create some child actor automatically somehow, when actors are sending or maybe receiving messages, or they are children of a singleton, or under some other conditions?.. Because, there is nothing that creates this child c1 actor on the user side. And I failed to find anything like such child actors creation in the akka.net source in reasonable time.

@kimbyungeun So far, it seems to me like Petabridge.cmd is working fine, and it's just some temporary "service" actor created by Akka.NET under the hood, that is displayed in the hierarchy from time to time.

seungyongshim commented 3 years ago

@IgorFedchenko Excuse, me.

image

@kimbyungeun

IgorFedchenko commented 3 years ago

@seungyongshim

You said, akka.tcp://Sys@localhost:90/remote/akka/tcp/localhost:89/$a is temporary "service" actor, isn't it?

No, my question is about c1 actor. There is no doubt actors with names starting with $a , $b are some internal Akka.NET actors, and there is no problem with them in this issue. The problem is, how I see it right now, that sometimes c1 actors are printed out in the hierarchy, and sometimes they are not.

My current theory is that this c1 actors might not be live all the time - they may be activated on demand to send a remote message for example, or due to some other stuff required for Akka.Remote. Living for a short period of time. And that's why they may be included / excluded from the cmd logs. We do not want petabridge.cmd to show actors that are not working at the moment, right?

And my question is that, I could not find piece of code in Akka that is responsible for such remote reference representation of remote actor link. If I would found it, I could see if this is normal that this reference is not displayed. Or, as @Aaronontheweb said, they might not be discovered due to the timeout of some kind.

Actually, let me try to increase command timeout and see if we will get c1 actors of all three nodes displayed stable.

IgorFedchenko commented 3 years ago

All right guys, @Aaronontheweb was absolutely right. Current timeout per layer is 40ms, and when changed to 400ms utility gives all c1 actors on all three nodes, all the time. Not sure why this is happening with this c1 actors only, but anyway, increasing the timeout does the trick.

Now the bad news is that this timeout is hardcoded as a readonly static field of internal class, so there is no way to configure this on your side. I am going to update our tool to support -t parameter, like actor hierarchy -d 10 -t 400 for example, to make this timeout configurable when someone is getting issues like that.

And we will push this update with 0.8.2 version of the Petabridge.Cmd.Host package. Does it sound good @kimbyungeun ?

IgorFedchenko commented 3 years ago

@kimbyungeun We have just released v0.8.2 of Petabridge.Cmd package pack.

Try to update and use actor hierarchy -d 10 -t 400 and let us know if the problem is resolved this way.

We will also update tool documentation soon, but the -t parameter stands for timeout-per-layer and allows to increase default 40ms timeout to 400ms for example.

Aaronontheweb commented 3 years ago

Resolved via Petabridge.Cmd v0.8.2

kimbyungeun commented 3 years ago

@IgorFedchenko @Aaronontheweb

It will be very helpful. Thank you very much.

actor hierarchy -d 10 -t 90 All actors are printed.

PS C:\Users\be.kim> pbm 127.0.0.1:9111
 ____      _        _          _     _
|  _ \ ___| |_ __ _| |__  _ __(_) __| | __ _  ___
| |_) / _ | __/ _` | '_ \| '__| |/ _` |/ _` |/ _ \
|  __|  __| || (_| | |_) | |  | | (_| | (_| |  __/
|_|   \___|\__\__,_|_.__/|_|  |_|\__,_|\__, |\___|
                                       |___/

petabridge.cmd (0.8.2.0)
Copyright 2017 - 2020, Petabridge®.

successfully connected to [::ffff:127.0.0.1]:9111
Commands downloaded from server. type `help` to see what's available
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$a
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$a/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$b
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$b/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$c
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$c/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm>

@seungyongshim

Aaronontheweb commented 3 years ago

Glad to hear it - please let us know if there's anything else we can do to help