Closed kimbyungeun closed 3 years ago
The reasons why the results are in different orders each time is because the actor hierarchy scan is done recursively in parallel down in the actor hierarchy using wildcard ActorSelections and the completion order of those parallel tasks is arbitrary. We're able to guarantee that a parent doesn't complete its scan until its children have completed theirs and so on, all the way until we reach the specified level of depth. But we don't guarantee the completion order of siblings - i.e. the order in which two children of the
/user
actor complete their scans.
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c3
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c2
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3064
/user/petabridge.cmd/127.0.0.1%3A3064/actor
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler/$h
/user/petabridge.cmd/127.0.0.1%3A3064/actor/handler/$h/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9115] pbm> cluster show
akka.tcp://mls-cluster-system@127.0.0.1:3507 | [api] | up |
akka.tcp://mls-cluster-system@127.0.0.1:3511 | [role1] | up |
akka.tcp://mls-cluster-system@127.0.0.1:4062 | [lighthouse] | up |
akka.tcp://mls-cluster-system@127.0.0.1:8112 | [role2] | up |
akka.tcp://mls-cluster-system@127.0.0.1:8113 | [role3] | up |
Count: 5 nodes
[127.0.0.1:9115] pbm>
____ _ _ _ _
| _ \ ___| |_ __ _| |__ _ __(_) __| | __ _ ___
| |_) / _ | __/ _` | '_ \| '__| |/ _` |/ _` |/ _ \
| __| __| || (_| | |_) | | | | (_| | (_| | __/
|_| \___|\__\__,_|_.__/|_| |_|\__,_|\__, |\___|
|___/
petabridge.cmd (0.8.0.0) Copyright 2017 - 2020, Petabridge®.
successfully connected to [::ffff:127.0.0.1]:9111
Commands downloaded from server. type help
to see what's available
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3943
/user/petabridge.cmd/127.0.0.1%3A3943/actor
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3943
/user/petabridge.cmd/127.0.0.1%3A3943/actor
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$b/$b
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3943
/user/petabridge.cmd/127.0.0.1%3A3943/actor
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$c/$b
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/ClusterSingleton
/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor
/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A3943
/user/petabridge.cmd/127.0.0.1%3A3943/actor
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d
/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$d/$b
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm>
* The following `Received unexpected ActorIdentity` warning log is displayed in the [api] role.
[2020/10/22 19:50:48.525][ 27][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] Received petabridge.cmd connection from client [127.0.0.1:3943]
[2020/10/22 19:50:48.525][ 27][DEBUG]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943[Petabridge.Cmd.Host.AkkaCmdController] Sending palettes to client...
[2020/10/22 19:50:57.714][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3925/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#565633793]]
[2020/10/22 19:50:57.776][ 32][DEBUG]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3943/actor/handler/$a/$b[Akka.Actor.LocalActorRef] Unhandled message from akka.tcp://mls-cluster-system@127.0.0.1:3927/remote/akka.tcp/mls-cluster-system@127.0.0.1:3925/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 :
### [api] role hocon
petabridge.cmd { host = "0.0.0.0" port = 9111 log-palettes-on-startup = on }
akka { service.name = "TestClusterApi" service.description = "TestClusterApi" system.name = "mls-cluster-system"
stdout-loglevel = DEBUG
loglevel = DEBUG
log-config-on-start = on
loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
actor
{
provider = cluster
guardian-supervisor-strategy = "TestClusterApi.ActorService.UserSupervisorStrategy, TestClusterApi"
debug
{
receive = off
autoreceive = off
lifecycle = off
event-stream = off
unhandled = on
}
deployment
{
/ApiMasterActor
{
}
"/ApiMasterActor/*/TestClusterRole1HelloActor"
{
router = round-robin-pool
nr-of-instances = 100
cluster
{
enabled = on
max-nr-of-instances-per-node = 1
allow-local-routees = off
use-role = role1
}
}
"/ApiMasterActor/*/TestClusterRole2HelloActor"
{
router = round-robin-pool
nr-of-instances = 100
cluster
{
enabled = on
max-nr-of-instances-per-node = 1
allow-local-routees = off
use-role = role2
}
}
"/ApiMasterActor/*/TestClusterRole3HelloActor"
{
router = round-robin-pool
nr-of-instances = 100
cluster
{
enabled = on
max-nr-of-instances-per-node = 1
allow-local-routees = off
use-role = role3
}
}
}
}
remote
{
dot-netty.tcp
{
public-hostname = 127.0.0.1
port = 0
send-buffer-size = 2097152b # 2MB
receive-buffer-size = 2097152b # 2MB
maximum-frame-size = 1048576b # 1MB
}
transport-failure-detector
{
heartbeat-interval = 5s
acceptable-heartbeat-pause = 50s
}
}
extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]
cluster
{
seed-nodes = ["akka.tcp://mls-cluster-system@127.0.0.1:4062"]
roles = [api]
auto-down-unreachable-after = 10s
gossip-interval = 5s
gossip-time-to-live = 30s
failure-detector
{
heartbeat-interval = 5s
acceptable-heartbeat-pause = 50s
}
singleton
{
singleton-name = "ClusterSingleton"
role = "role1"
hand-over-retry-interval = 1s
min-number-of-hand-over-retries = 10
}
}
}
we'll take a look at this - wonder if we made any changes to ActorSelection
in the meantime.
@kimbyungeun I am going to look into this. From the logs in your last message, it seems like under Akka 1.4+ not all actors are displayed, right? But all c1/c2/c3 child actors under TestClusterRole1HelloActor
are running all the time, right (I mean, it is not that they are shutting down and starting from time to time)?
Do you have a sample project you used to generate this logs? If you do, could you share in as a zip file or Github repo to help me with reproduction? That would be extremely useful.
@IgorFedchenko
I am going to look into this. From the logs in your last message, it seems like under Akka 1.4+ not all actors are displayed, right?
- YES
Q2
But all c1/c2/c3 child actors under TestClusterRole1HelloActor are running all the time, right (I mean, it is not that they are shutting down and starting from time to time)?
- YES
Lighthouse.log
[2020-10-23 09:18:22.1925][ INFO][P:13308][T: 1] Configuration Result: [Success] Name Mirero.MLS.TestLightHouse4061 [Success] Description Mirero.MLS.TestLightHouse.4061 [Success] ServiceName Mirero.MLS.TestLightHouse4061 [2020-10-23 09:18:22.2125][ INFO][P:13308][T: 1] Topshelf v4.2.1.215, .NET Framework v4.0.30319.42000 [2020-10-23 09:18:22.2355][DEBUG][P:13308][T: 1] Running as a console application, creating the console host. [2020-10-23 09:18:22.2355][DEBUG][P:13308][T: 1] Starting up as a console application [2020-10-23 09:18:22.5005][DEBUG][P:13308][T: 5] EventStream(mls-cluster-system) Logger log1-NLogLogger [NLogLogger] started [2020-10-23 09:18:22.5005][DEBUG][P:13308][T: 5] EventStream(mls-cluster-system) StandardOutLogger being removed [2020-10-23 09:18:22.5005][DEBUG][P:13308][T: 5] EventStream(mls-cluster-system) Default Loggers started [2020-10-23 09:18:22.5135][ INFO][P:13308][T: 8] remoting (akka://mls-cluster-system) Starting remoting [2020-10-23 09:18:22.5285][DEBUG][P:13308][T: 8] remoting (akka://mls-cluster-system) Starting prune timer for endpoint manager... [2020-10-23 09:18:22.7185][ INFO][P:13308][T: 5] remoting (akka://mls-cluster-system) Remoting started; listening on addresses : [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:22.7185][ INFO][P:13308][T: 5] remoting (akka://mls-cluster-system) Remoting now listens on addresses: [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:22.7515][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Starting up... [2020-10-23 09:18:22.7755][ INFO][P:13308][T: 6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Started up successfully [2020-10-23 09:18:22.8015][ INFO][P:13308][T: 19] ActorSystem(mls-cluster-system) akka : { version : "0.0.1 Akka" home : loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"] loggers-dispatcher : akka.actor.default-dispatcher logger-startup-timeout : 5s logger-async-start : false loglevel : DEBUG suppress-json-serializer-warning : on stdout-loglevel : DEBUG log-config-on-start : on log-dead-letters : 10 log-dead-letters-during-shutdown : off log-dead-letters-suspend-duration : "5 minutes" extensions : ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"] daemonic : off actor : { provider : cluster guardian-supervisor-strategy : Akka.Actor.DefaultSupervisorStrategy creation-timeout : 20s reaper-interval : 5 serialize-messages : off serialize-creators : off unstarted-push-timeout : 10s ask-timeout : infinite typed : { timeout : 5 } inbox : { inbox-size : 1000 default-timeout : 5s } router : { type-mapping : { from-code : Akka.Routing.NoRouter round-robin-pool : Akka.Routing.RoundRobinPool round-robin-group : Akka.Routing.RoundRobinGroup random-pool : Akka.Routing.RandomPool random-group : Akka.Routing.RandomGroup smallest-mailbox-pool : Akka.Routing.SmallestMailboxPool broadcast-pool : Akka.Routing.BroadcastPool broadcast-group : Akka.Routing.BroadcastGroup scatter-gather-pool : Akka.Routing.ScatterGatherFirstCompletedPool scatter-gather-group : Akka.Routing.ScatterGatherFirstCompletedGroup consistent-hashing-pool : Akka.Routing.ConsistentHashingPool consistent-hashing-group : Akka.Routing.ConsistentHashingGroup tail-chopping-pool : Akka.Routing.TailChoppingPool tail-chopping-group : Akka.Routing.TailChoppingGroup } } deployment : { default : { dispatcher : mailbox : router : from-code nr-of-instances : 1 within : "5 s" virtual-nodes-factor : 10 routees : { paths : [] } resizer : { enabled : off lower-bound : 1 upper-bound : 10 pressure-threshold : 1 rampup-rate : 0.2 backoff-threshold : 0.3 backoff-rate : 0.1 messages-per-resize : 10 } remote : target : { nodes : [] } metrics-selector : mix cluster : { enabled : off max-nr-of-instances-per-node : 1 max-total-nr-of-instances : 10000 allow-local-routees : on use-role : } } } synchronized-dispatcher : { type : SynchronizedDispatcher executor : current-context-executor throughput : 10 } task-dispatcher : { type : TaskDispatcher executor : task-executor throughput : 30 } default-fork-join-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor throughput : 30 dedicated-thread-pool : { thread-count : 3 threadtype : background } } default-dispatcher : { type : Dispatcher executor : default-executor default-executor : { } thread-pool-executor : { } fork-join-executor : { dedicated-thread-pool : { thread-count : 3 threadtype : background } } current-context-executor : { } shutdown-timeout : 1s throughput : 30 throughput-deadline-time : 0ms attempt-teamwork : on mailbox-requirement : } default-mailbox : { mailbox-type : Akka.Dispatch.UnboundedMailbox mailbox-capacity : 1000 mailbox-push-timeout-time : 10s stash-capacity : -1 } mailbox : { requirements : { Akka.Dispatch.IUnboundedMessageQueueSemantics : akka.actor.mailbox.unbounded-queue-based Akka.Dispatch.IBoundedMessageQueueSemantics : akka.actor.mailbox.bounded-queue-based Akka.Dispatch.IDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based Akka.Dispatch.IUnboundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based Akka.Dispatch.IBoundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.bounded-deque-based Akka.Dispatch.IMultipleConsumerSemantics : akka.actor.mailbox.unbounded-queue-based Akka.Event.ILoggerMessageQueueSemantics : akka.actor.mailbox.logger-queue } unbounded-queue-based : { mailbox-type : Akka.Dispatch.UnboundedMailbox } bounded-queue-based : { mailbox-type : Akka.Dispatch.BoundedMailbox } unbounded-deque-based : { mailbox-type : Akka.Dispatch.UnboundedDequeBasedMailbox } bounded-deque-based : { mailbox-type : Akka.Dispatch.BoundedDequeBasedMailbox } logger-queue : { mailbox-type : Akka.Event.LoggerMailboxType } } debug : { receive : off autoreceive : off lifecycle : off fsm : off event-stream : off unhandled : on router-misconfiguration : off } serializers : { json : "Akka.Serialization.NewtonSoftJsonSerializer, Akka" bytes : "Akka.Serialization.ByteArraySerializer, Akka" akka-containers : "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" akka-misc : "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" primitive : "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" proto : "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" daemon-create : "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" akka-system-msg : "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" akka-cluster : "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" akka-cluster-client : "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" akka-pubsub : "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" } serialization-bindings : { System.Byte[] : bytes System.Object : json "Akka.Actor.ActorSelectionMessage, Akka" : akka-containers "Akka.Remote.DaemonMsgCreate, Akka.Remote" : daemon-create "Google.Protobuf.IMessage, Google.Protobuf" : proto "Akka.Actor.Identify, Akka" : akka-misc "Akka.Actor.ActorIdentity, Akka" : akka-misc "Akka.Actor.IActorRef, Akka" : akka-misc "Akka.Actor.PoisonPill, Akka" : akka-misc "Akka.Actor.Kill, Akka" : akka-misc "Akka.Actor.RemoteScope, Akka" : akka-misc "Akka.Routing.FromConfig, Akka" : akka-misc "Akka.Routing.DefaultResizer, Akka" : akka-misc "Akka.Routing.RoundRobinPool, Akka" : akka-misc "Akka.Routing.BroadcastPool, Akka" : akka-misc "Akka.Routing.RandomPool, Akka" : akka-misc "Akka.Routing.ScatterGatherFirstCompletedPool, Akka" : akka-misc "Akka.Routing.TailChoppingPool, Akka" : akka-misc "Akka.Routing.ConsistentHashingPool, Akka" : akka-misc "Akka.Configuration.Config, Akka" : akka-misc "Akka.Remote.RemoteWatcher+Heartbeat, Akka.Remote" : akka-misc "Akka.Remote.RemoteWatcher+HeartbeatRsp, Akka.Remote" : akka-misc "Akka.Remote.Routing.RemoteRouterConfig, Akka.Remote" : akka-misc "Akka.Dispatch.SysMsg.SystemMessage, Akka" : akka-system-msg System.String : primitive System.Int32 : primitive System.Int64 : primitive "Akka.Cluster.IClusterMessage, Akka.Cluster" : akka-cluster "Akka.Cluster.Routing.ClusterRouterPool, Akka.Cluster" : akka-cluster "Akka.Cluster.Tools.Client.IClusterClientMessage, Akka.Cluster.Tools" : akka-cluster-client "Akka.Cluster.Tools.PublishSubscribe.IDistributedPubSubMessage, Akka.Cluster.Tools" : akka-pubsub "Akka.Cluster.Tools.PublishSubscribe.Internal.SendToOneSubscriber, Akka.Cluster.Tools" : akka-pubsub } serialization-identifiers : { "Akka.Serialization.ByteArraySerializer, Akka" : 4 "Akka.Serialization.NewtonSoftJsonSerializer, Akka" : 1 "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" : 2 "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" : 3 "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" : 6 "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" : 16 "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" : 17 "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" : 22 "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" : 5 "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" : 15 "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" : 9 } serialization-settings : { } } scheduler : { tick-duration : 10ms ticks-per-wheel : 512 implementation : Akka.Actor.HashedWheelTimerScheduler shutdown-timeout : 5s } io : { pinned-dispatcher : { type : PinnedDispatcher executor : fork-join-executor } tcp : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } disabled-buffer-pool : { class : "Akka.IO.Buffers.DisabledBufferPool, Akka" buffer-size : 512 } buffer-pool : akka.io.tcp.disabled-buffer-pool max-channels : 256000 selector-association-retries : 10 batch-accept-limit : 10 register-timeout : 5s max-received-message-size : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher file-io-dispatcher : akka.actor.default-dispatcher file-io-transferTo-limit : 524288 finish-connect-retries : 5 windows-connection-abort-workaround-enabled : off outgoing-socket-force-ipv4 : false } udp : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } buffer-pool : akka.io.udp.direct-buffer-pool nr-of-socket-async-event-args : 32 max-channels : 4096 select-timeout : infinite selector-association-retries : 10 receive-throughput : 3 direct-buffer-size : 18432 direct-buffer-pool-limit : 1000 received-message-size-limit : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher } udp-connected : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } buffer-pool : akka.io.udp-connected.direct-buffer-pool nr-of-socket-async-event-args : 32 max-channels : 4096 select-timeout : infinite selector-association-retries : 10 receive-throughput : 3 direct-buffer-size : 18432 direct-buffer-pool-limit : 1000 received-message-size-limit : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher } dns : { dispatcher : akka.actor.default-dispatcher resolver : inet-address inet-address : { provider-object : Akka.IO.InetAddressDnsProvider positive-ttl : 30s negative-ttl : 10s cache-cleanup-interval : 120s use-ipv6 : true } } } coordinated-shutdown : { default-phase-timeout : "5 s" terminate-actor-system : on exit-clr : off run-by-clr-shutdown-hook : on run-by-actor-system-terminate : on phases : { before-service-unbind : { } service-unbind : { depends-on : [before-service-unbind] } service-requests-done : { depends-on : [service-unbind] } service-stop : { depends-on : [service-requests-done] } before-cluster-shutdown : { depends-on : [service-stop] } cluster-sharding-shutdown-region : { timeout : "10 s" depends-on : [before-cluster-shutdown] } cluster-leave : { depends-on : [cluster-sharding-shutdown-region] } cluster-exiting : { timeout : "10 s" depends-on : [cluster-leave] } cluster-exiting-done : { depends-on : [cluster-exiting] } cluster-shutdown : { depends-on : [cluster-exiting-done] } before-actor-system-terminate : { depends-on : [cluster-shutdown] } actor-system-terminate : { timeout : "10 s" depends-on : [before-actor-system-terminate] } } } remote : { startup-timeout : "10 s" shutdown-timeout : "10 s" flush-wait-on-shutdown : "2 s" use-passive-connections : on backoff-interval : "0.05 s" command-ack-timeout : "30 s" handshake-timeout : "15 s" use-dispatcher : akka.remote.default-remote-dispatcher untrusted-mode : off trusted-selection-paths : [] require-cookie : off secure-cookie : log-received-messages : off log-sent-messages : off log-remote-lifecycle-events : DEBUG log-frame-size-exceeding : off log-buffer-size-exceeding : 50000 transport-failure-detector : { implementation-class : Akka.Remote.DeadlineFailureDetector,Akka.Remote heartbeat-interval : 5s acceptable-heartbeat-pause : 50s } watch-failure-detector : { implementation-class : Akka.Remote.PhiAccrualFailureDetector,Akka.Remote heartbeat-interval : "1 s" threshold : 10.0 max-sample-size : 200 min-std-deviation : "100 ms" acceptable-heartbeat-pause : "10 s" unreachable-nodes-reaper-interval : 1s expected-response-after : "1 s" } retry-gate-closed-for : "5 s" prune-quarantine-marker-after : "5 d" quarantine-after-silence : "2 d" system-message-buffer-size : 20000 system-message-ack-piggyback-timeout : "0.3 s" resend-interval : "2 s" resend-limit : 200 initial-system-message-delivery-timeout : "3 m" enabled-transports : [akka.remote.dot-netty.tcp] adapters : { gremlin : Akka.Remote.Transport.FailureInjectorProvider,Akka.Remote trttl : Akka.Remote.Transport.ThrottlerProvider,Akka.Remote } helios : { tcp : { transport-class : "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote.Transport.Helios" } } dot-netty : { tcp : { transport-class : Akka.Remote.Transport.DotNetty.TcpTransport,Akka.Remote applied-adapters : [] transport-protocol : tcp byte-order : little-endian port : 4062 public-port : 0 hostname : public-hostname : 127.0.0.1 dns-use-ipv6 : false enforce-ip-family : false enable-ssl : false enable-backwards-compatibility : false connection-timeout : "15 s" batching : { enabled : true max-pending-writes : 30 max-pending-bytes : 16k flush-interval : 40ms } use-dispatcher-for-io : write-buffer-high-water-mark : 0b write-buffer-low-water-mark : 0b send-buffer-size : 256000b receive-buffer-size : 256000b maximum-frame-size : 128000b backlog : 4096 tcp-nodelay : on tcp-keepalive : on tcp-reuse-addr : off-for-windows server-socket-worker-pool : { pool-size-min : 2 pool-size-factor : 1.0 pool-size-max : 2 } client-socket-worker-pool : { pool-size-min : 2 pool-size-factor : 1.0 pool-size-max : 2 } } udp : { transport-protocol : udp } ssl : { certificate : { path : password : use-thumprint-over-file : false thumpbrint : store-name : store-location : current-user } suppress-validation : false } } gremlin : { debug : off } default-remote-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor dedicated-thread-pool : { thread-count : 4 } } backoff-remote-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor dedicated-thread-pool : { thread-count : 4 } } } cluster : { seed-nodes : ["akka.tcp://mls-cluster-system@127.0.0.1:4062"] seed-node-timeout : 5s retry-unsuccessful-join-after : 10s shutdown-after-unsuccessful-join-seed-nodes : off auto-down-unreachable-after : 10s down-removal-margin : off downing-provider-class : allow-weakly-up-members : on roles : [lighthouse] run-coordinated-shutdown-when-down : on role : { } min-nr-of-members : 1 log-info : on log-info-verbose : off periodic-tasks-initial-delay : 1s gossip-interval : 5s gossip-time-to-live : 30s leader-actions-interval : 1s unreachable-nodes-reaper-interval : 1s publish-stats-interval : off use-dispatcher : gossip-different-view-probability : 0.8 reduce-gossip-different-view-probability : 400 failure-detector : { implementation-class : "Akka.Remote.PhiAccrualFailureDetector, Akka.Remote" heartbeat-interval : 5s threshold : 8.0 max-sample-size : 1000 min-std-deviation : "100 ms" acceptable-heartbeat-pause : 50s monitored-by-nr-of-members : 9 expected-response-after : "1 s" } split-brain-resolver : { active-strategy : off stable-after : 20s static-quorum : { quorum-size : undefined role : } keep-majority : { role : } keep-oldest : { down-if-alone : on role : } keep-refree : { address : down-all-if-less-than-nodes : 1 } } scheduler : { tick-duration : 33ms ticks-per-wheel : 512 } debug : { verbose-heartbeat-logging : off verbose-receive-gossip-logging : off } client : { receptionist : { name : receptionist role : lighthouse number-of-contacts : 3 response-tunnel-receive-timeout : 1d use-dispatcher : heartbeat-interval : 2s acceptable-heartbeat-pause : 13s failure-detection-interval : 2s } initial-contacts : [] establishing-get-contacts-interval : 3s refresh-contacts-interval : 60s heartbeat-interval : 2s acceptable-heartbeat-pause : 13s buffer-size : 1000 reconnect-timeout : off } pub-sub : { name : distributedPubSubMediator role : routing-logic : random gossip-interval : 1s removed-time-to-live : 120s max-delta-elements : 3000 use-dispatcher : } } service : { name : Mirero.MLS.TestLightHouse4061 description : Mirero.MLS.TestLightHouse.4061 } system : { name : mls-cluster-testlighthouse-4061 } } lighthouse : { actorsystem : mls-cluster-system } petabridge : { cmd : { host : 0.0.0.0 port : 9115 } }
[2020-10-23 09:18:22.8135][ INFO][P:13308][T: 1] The Mirero.MLS.TestLightHouse4061 service is now running, press Control+C to exit. [2020-10-23 09:18:22.8305][ INFO][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/user/petabridge.cmd petabridge.cmd host bound to [0.0.0.0:9115] [2020-10-23 09:18:22.8305][ INFO][P:13308][T: 22] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] is JOINING itself (with roles [lighthouse]) and forming a new cluster [2020-10-23 09:18:22.8435][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - is the new leader among reachable nodes (more leaders may exist) [2020-10-23 09:18:22.8655][ INFO][P:13308][T: 5] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [Up] [2020-10-23 09:18:30.8114][DEBUG][P:13308][T: 19] remoting (akka://mls-cluster-system) Associated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:3893 [2020-10-23 09:18:30.9214][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Received InitJoin message from [[akka.tcp://mls-cluster-system@127.0.0.1:3893/system/cluster/core/daemon/joinSeedNodeProcess-1#928698614]] to [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:30.9214][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Sending InitJoinNack message from node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [[akka.tcp://mls-cluster-system@127.0.0.1:3893/system/cluster/core/daemon/joinSeedNodeProcess-1#928698614]] [2020-10-23 09:18:31.0354][ INFO][P:13308][T: 19] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] is JOINING, roles [api] [2020-10-23 09:18:31.2664][DEBUG][P:13308][T: 22] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:31.9224][ INFO][P:13308][T: 21] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:3893] to [Up] [2020-10-23 09:18:32.8939][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:32.9409][ INFO][P:13308][T: 9] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - is no longer leader [2020-10-23 09:18:37.7321][DEBUG][P:13308][T: 4] remoting (akka://mls-cluster-system) Associated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:3895 [2020-10-23 09:18:37.8771][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Received InitJoin message from [[akka.tcp://mls-cluster-system@127.0.0.1:3895/system/cluster/core/daemon/joinSeedNodeProcess-1#370319934]] to [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:37.8771][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Sending InitJoinNack message from node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [[akka.tcp://mls-cluster-system@127.0.0.1:3895/system/cluster/core/daemon/joinSeedNodeProcess-1#370319934]] [2020-10-23 09:18:37.9731][ INFO][P:13308][T: 9] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:3895] is JOINING, roles [role1] [2020-10-23 09:18:38.1161][DEBUG][P:13308][T: 21] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:40.7272][DEBUG][P:13308][T: 26] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:42.9415][DEBUG][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:45.4732][DEBUG][P:13308][T: 5] remoting (akka://mls-cluster-system) Associated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:8112 [2020-10-23 09:18:45.6072][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Received InitJoin message from [[akka.tcp://mls-cluster-system@127.0.0.1:8112/system/cluster/core/daemon/joinSeedNodeProcess-1#713857872]] to [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:45.6072][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Sending InitJoinNack message from node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [[akka.tcp://mls-cluster-system@127.0.0.1:8112/system/cluster/core/daemon/joinSeedNodeProcess-1#713857872]] [2020-10-23 09:18:45.6392][DEBUG][P:13308][T: 30] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:45.7042][ INFO][P:13308][T: 24] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:8112] is JOINING, roles [role2] [2020-10-23 09:18:45.8442][DEBUG][P:13308][T: 24] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:18:47.9410][DEBUG][P:13308][T: 21] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:50.7473][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:51.9473][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Associated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:8113 [2020-10-23 09:18:52.0973][ INFO][P:13308][T: 8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Received InitJoin message from [[akka.tcp://mls-cluster-system@127.0.0.1:8113/system/cluster/core/daemon/joinSeedNodeProcess-1#203439572]] to [akka.tcp://mls-cluster-system@127.0.0.1:4062] [2020-10-23 09:18:52.0973][ INFO][P:13308][T: 8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Sending InitJoinNack message from node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [[akka.tcp://mls-cluster-system@127.0.0.1:8113/system/cluster/core/daemon/joinSeedNodeProcess-1#203439572]] [2020-10-23 09:18:52.1923][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Node [akka.tcp://mls-cluster-system@127.0.0.1:8113] is JOINING, roles [role3] [2020-10-23 09:18:52.3333][DEBUG][P:13308][T: 23] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8113, 2078514987)] [2020-10-23 09:18:52.9553][DEBUG][P:13308][T: 22] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:18:55.6648][DEBUG][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:18:57.7278][DEBUG][P:13308][T: 22] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:57.8228][DEBUG][P:13308][T: 26] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:18:57.9648][DEBUG][P:13308][T: 21] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8113, 2078514987)] [2020-10-23 09:19:01.8677][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8113, 2078514987)] [2020-10-23 09:19:02.9681][DEBUG][P:13308][T: 30] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:19:05.6717][DEBUG][P:13308][T: 24] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:19:12.9930][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:19:18.0291][DEBUG][P:13308][T: 21] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:19:20.4801][DEBUG][P:13308][T: 26] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:19:22.7684][DEBUG][P:13308][T: 24] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:35:03.0348][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:35:04.8020][DEBUG][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:35:05.6941][DEBUG][P:13308][T: 8] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:35:05.9421][ INFO][P:13308][T: 22] Control+C detected, attempting to stop service. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-service-unbind] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-unbind] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-requests-done] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [service-stop] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-cluster-shutdown] with [0] tasks. [2020-10-23 09:35:05.9421][DEBUG][P:13308][T: 23] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-sharding-shutdown-region] with [0] tasks. [2020-10-23 09:35:05.9531][DEBUG][P:13308][T: 4] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-leave] with [1] tasks: [leave] [2020-10-23 09:35:05.9531][ INFO][P:13308][T: 20] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Marked address [akka.tcp://mls-cluster-system@127.0.0.1:4062] as [Leaving] [2020-10-23 09:35:05.9531][DEBUG][P:13308][T: 5] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-exiting] with [1] tasks: [wait-exiting] [2020-10-23 09:35:06.0446][DEBUG][P:13308][T: 5] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:35:08.5166][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3894-1 Association between local [tcp://mls-cluster-system@0.0.0.0:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3894] was disassociated because the ProtocolStateActor failed: Shutdown [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon "Couldn't establish a causal relationship between "remote" gossip and "local" gossip - Remote[Gossip(members = [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3893, Uid=1373298527 status = Exiting, role=[api], upNumber=2), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3895, Uid=2141762004 status = Exiting, role=[role1], upNumber=3), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Up, role=[lighthouse], upNumber=1), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8112, Uid=1552571697 status = Exiting, role=[role2], upNumber=4), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8113, Uid=2078514987 status = Exiting, role=[role3], upNumber=5)], overview = GossipOverview(seen=[], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->7, B0FD6E6E85BC21C543C59A240889BC7C->6, BFDECCB8DF57F49EDBD5FF0F943FF688->1)] - Local[Gossip(members = [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3893, Uid=1373298527 status = Leaving, role=[api], upNumber=2), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3895, Uid=2141762004 status = Leaving, role=[role1], upNumber=3), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Leaving, role=[lighthouse], upNumber=1), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8112, Uid=1552571697 status = Leaving, role=[role2], upNumber=4), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8113, Uid=2078514987 status = Leaving, role=[role3], upNumber=5)], overview = GossipOverview(seen=[UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:4062, 1685981556), UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->8, B0FD6E6E85BC21C543C59A240889BC7C->5, BFDECCB8DF57F49EDBD5FF0F943FF688->1)] - merged them into [Gossip(members = [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3893, Uid=1373298527 status = Exiting, role=[api], upNumber=2), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:3895, Uid=2141762004 status = Exiting, role=[role1], upNumber=3), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Leaving, role=[lighthouse], upNumber=1), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8112, Uid=1552571697 status = Exiting, role=[role2], upNumber=4), Member(address = akka.tcp://mls-cluster-system@127.0.0.1:8113, Uid=2078514987 status = Exiting, role=[role3], upNumber=5)], overview = GossipOverview(seen=[], reachability=Reachability()), version = VectorClock(338FE94A81D6B5502DD6250BF7248245->1, 3BF7D15B5FC501EA124E1F827EAFF1A0->1, 5289C652F346DAD0EB879B042CD0AB9F->8, B0FD6E6E85BC21C543C59A240889BC7C->6, BFDECCB8DF57F49EDBD5FF0F943FF688->1)]" [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3893, 1373298527)] [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://mls-cluster-system@127.0.0.1:3893] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. [2020-10-23 09:35:08.5316][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting confirmed [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:3895, 2141762004)] [2020-10-23 09:35:08.5316][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting confirmed [akka.tcp://mls-cluster-system@127.0.0.1:3895] [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3896-2 Association between local [tcp://mls-cluster-system@0.0.0.0:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3896] was disassociated because the ProtocolStateActor failed: Shutdown [2020-10-23 09:35:08.5316][DEBUG][P:13308][T: 25] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://mls-cluster-system@127.0.0.1:3895] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [1] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [2] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 5] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3898-3 Association between local [tcp://mls-cluster-system@0.0.0.0:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3898] was disassociated because the ProtocolStateActor failed: Shutdown [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 27] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fmls-cluster-system%40%5B%3A%3Affff%3A127.0.0.1%5D%3A3901-4 Association between local [tcp://mls-cluster-system@0.0.0.0:4062] and remote [tcp://mls-cluster-system@[::ffff:127.0.0.1]:3901] was disassociated because the ProtocolStateActor failed: Shutdown [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 4] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8112, 1552571697)] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 8] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting confirmed [akka.tcp://mls-cluster-system@127.0.0.1:8112] [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 27] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Receiving gossip from [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:8113, 2078514987)] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting confirmed [akka.tcp://mls-cluster-system@127.0.0.1:8113] [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://mls-cluster-system@127.0.0.1:8112] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] remoting (akka://mls-cluster-system) Remote system with address [akka.tcp://mls-cluster-system@127.0.0.1:8113] has shut down. Address is now gated for 5000ms, all messages to this address will be delivered to dead letters. [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 8] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3893-1/endpointWriter Disassociated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:3893 [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8113-4/endpointWriter Disassociated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:8113 [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8112-3/endpointWriter Disassociated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:8112 [2020-10-23 09:35:08.5446][DEBUG][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3895-2/endpointWriter Disassociated [akka.tcp://mls-cluster-system@127.0.0.1:4062] <- akka.tcp://mls-cluster-system@127.0.0.1:3895 [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3895-2 Removing receive buffers for [akka.tcp://mls-cluster-system@127.0.0.1:4062]->[akka.tcp://mls-cluster-system@127.0.0.1:3895] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 27] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A3893-1 Removing receive buffers for [akka.tcp://mls-cluster-system@127.0.0.1:4062]->[akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8112-3 Removing receive buffers for [akka.tcp://mls-cluster-system@127.0.0.1:4062]->[akka.tcp://mls-cluster-system@127.0.0.1:8112] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 9] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fmls-cluster-system%40127.0.0.1%3A8113-4 Removing receive buffers for [akka.tcp://mls-cluster-system@127.0.0.1:4062]->[akka.tcp://mls-cluster-system@127.0.0.1:8113] [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 9] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [3] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020-10-23 09:35:08.5446][ INFO][P:13308][T: 8] akka://mls-cluster-system/deadLetters Message [GossipEnvelope] from akka://mls-cluster-system/system/cluster/core/daemon to akka://mls-cluster-system/deadLetters was not delivered. [4] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/deadLetters may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - is the new leader among reachable nodes (more leaders may exist) [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 20] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-exiting-done] with [1] tasks: [exiting-completed] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:4062] to [Exiting] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is removing confirmed Exiting node [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is removing confirmed Exiting node [akka.tcp://mls-cluster-system@127.0.0.1:8112] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is removing confirmed Exiting node [akka.tcp://mls-cluster-system@127.0.0.1:8113] [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 4] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Leader is removing confirmed Exiting node [akka.tcp://mls-cluster-system@127.0.0.1:3895] [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 26] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/cluster/core/daemon Cluster Node [UniqueAddress: (akka.tcp://mls-cluster-system@127.0.0.1:4062, 1685981556)] - Gossip exiting members [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Exiting, role=[lighthouse], upNumber=1)] to the two oldest (per role) [Member(address = akka.tcp://mls-cluster-system@127.0.0.1:4062, Uid=1685981556 status = Exiting, role=[lighthouse], upNumber=1)] (singleton optimization). [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 29] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Exiting completed. [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 25] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Shutting down... [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 6] Cluster (akka://mls-cluster-system) Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:4062] - Successfully shut down [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 30] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [cluster-shutdown] with [1] tasks: [wait-shutdown] [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [before-actor-system-terminate] with [0] tasks. [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 27] CoordinatedShutdown (akka://mls-cluster-system) Performing phase [actor-system-terminate] with [1] tasks: [terminate-system] [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 23] ActorSystem(mls-cluster-system) System shutdown initiated [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 24] akka.tcp://mls-cluster-system@127.0.0.1:4062/user/petabridge.cmd Shutting down petabridge.cmd host... [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 6] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/IO-TCP/$a Unbinding endpoint 0.0.0.0:9115 [2020-10-23 09:35:08.6996][DEBUG][P:13308][T: 20] akka.tcp://mls-cluster-system@127.0.0.1:4062/system/IO-TCP/$a Unbound endpoint 0.0.0.0:9115, stopping listener [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 24] remoting-terminator Shutting down remote daemon. [2020-10-23 09:35:08.6996][ INFO][P:13308][T: 38] remoting-terminator Remote daemon shut down; proceeding with flushing remote transports. [2020-10-23 09:35:08.7356][ INFO][P:13308][T: 5] remoting (akka://mls-cluster-system) Remoting shut down [2020-10-23 09:35:08.7356][ INFO][P:13308][T: 37] remoting-terminator Remoting shut down. [2020-10-23 09:35:08.7356][DEBUG][P:13308][T: 39] EventStream Shutting down: StandardOutLogger started [2020-10-23 09:35:08.7646][ INFO][P:13308][T: 1] The Mirero.MLS.TestLightHouse4061 service has stopped.
#### TestClusterApi.log
[2020/10/23 09:18:30.051][ 1][ INFO][Topshelf.HostFactory] Configuration Result: [Success] Name Mirero.MLS.TestClusterApi [Success] ServiceName Mirero.MLS.TestClusterApi [2020/10/23 09:18:30.106][ 1][ INFO][Topshelf.HostConfigurators.HostConfiguratorImpl] Topshelf v4.2.1.215, .NET Framework v4.0.30319.42000 [2020/10/23 09:18:30.340][ 4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Starting remoting [2020/10/23 09:18:30.518][ 4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Remoting started; listening on addresses : [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020/10/23 09:18:30.518][ 4][ INFO]remoting (akka://mls-cluster-system)[remoting (akka://mls-cluster-system)] Remoting now listens on addresses: [akka.tcp://mls-cluster-system@127.0.0.1:3893] [2020/10/23 09:18:30.544][ 5][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Starting up... [2020/10/23 09:18:30.563][ 9][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Started up successfully [2020/10/23 09:18:30.580][ 10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] akka : { version : "0.0.1 Akka" home : loggers : ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"] loggers-dispatcher : akka.actor.default-dispatcher logger-startup-timeout : 5s logger-async-start : false loglevel : DEBUG suppress-json-serializer-warning : on stdout-loglevel : DEBUG log-config-on-start : on log-dead-letters : 10 log-dead-letters-during-shutdown : off log-dead-letters-suspend-duration : "5 minutes" extensions : ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"] daemonic : off actor : { provider : cluster guardian-supervisor-strategy : "Mirero.MLS.TestClusterApi.ActorService.UserSupervisorStrategy, Mirero.MLS.TestClusterApi" creation-timeout : 20s reaper-interval : 5 serialize-messages : off serialize-creators : off unstarted-push-timeout : 10s ask-timeout : infinite typed : { timeout : 5 } inbox : { inbox-size : 1000 default-timeout : 5s } router : { type-mapping : { from-code : Akka.Routing.NoRouter round-robin-pool : Akka.Routing.RoundRobinPool round-robin-group : Akka.Routing.RoundRobinGroup random-pool : Akka.Routing.RandomPool random-group : Akka.Routing.RandomGroup smallest-mailbox-pool : Akka.Routing.SmallestMailboxPool broadcast-pool : Akka.Routing.BroadcastPool broadcast-group : Akka.Routing.BroadcastGroup scatter-gather-pool : Akka.Routing.ScatterGatherFirstCompletedPool scatter-gather-group : Akka.Routing.ScatterGatherFirstCompletedGroup consistent-hashing-pool : Akka.Routing.ConsistentHashingPool consistent-hashing-group : Akka.Routing.ConsistentHashingGroup tail-chopping-pool : Akka.Routing.TailChoppingPool tail-chopping-group : Akka.Routing.TailChoppingGroup } } deployment : { default : { dispatcher : mailbox : router : from-code nr-of-instances : 1 within : "5 s" virtual-nodes-factor : 10 routees : { paths : [] } resizer : { enabled : off lower-bound : 1 upper-bound : 10 pressure-threshold : 1 rampup-rate : 0.2 backoff-threshold : 0.3 backoff-rate : 0.1 messages-per-resize : 10 } remote : target : { nodes : [] } metrics-selector : mix cluster : { enabled : off max-nr-of-instances-per-node : 1 max-total-nr-of-instances : 10000 allow-local-routees : on use-role : } } /ApiMasterActor : { } /ApiMasterActor//TestClusterRole1HelloActor : { router : round-robin-pool nr-of-instances : 100 cluster : { enabled : on max-nr-of-instances-per-node : 1 allow-local-routees : off use-role : role1 } } /ApiMasterActor//TestClusterRole2HelloActor : { router : round-robin-pool nr-of-instances : 100 cluster : { enabled : on max-nr-of-instances-per-node : 1 allow-local-routees : off use-role : role2 } } /ApiMasterActor/*/TestClusterRole3HelloActor : { router : round-robin-pool nr-of-instances : 100 cluster : { enabled : on max-nr-of-instances-per-node : 1 allow-local-routees : off use-role : role3 } } } synchronized-dispatcher : { type : SynchronizedDispatcher executor : current-context-executor throughput : 10 } task-dispatcher : { type : TaskDispatcher executor : task-executor throughput : 30 } default-fork-join-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor throughput : 30 dedicated-thread-pool : { thread-count : 3 threadtype : background } } default-dispatcher : { type : Dispatcher executor : default-executor default-executor : { } thread-pool-executor : { } fork-join-executor : { dedicated-thread-pool : { thread-count : 3 threadtype : background } } current-context-executor : { } shutdown-timeout : 1s throughput : 30 throughput-deadline-time : 0ms attempt-teamwork : on mailbox-requirement : } default-mailbox : { mailbox-type : Akka.Dispatch.UnboundedMailbox mailbox-capacity : 1000 mailbox-push-timeout-time : 10s stash-capacity : -1 } mailbox : { requirements : { Akka.Dispatch.IUnboundedMessageQueueSemantics : akka.actor.mailbox.unbounded-queue-based Akka.Dispatch.IBoundedMessageQueueSemantics : akka.actor.mailbox.bounded-queue-based Akka.Dispatch.IDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based Akka.Dispatch.IUnboundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.unbounded-deque-based Akka.Dispatch.IBoundedDequeBasedMessageQueueSemantics : akka.actor.mailbox.bounded-deque-based Akka.Dispatch.IMultipleConsumerSemantics : akka.actor.mailbox.unbounded-queue-based Akka.Event.ILoggerMessageQueueSemantics : akka.actor.mailbox.logger-queue } unbounded-queue-based : { mailbox-type : Akka.Dispatch.UnboundedMailbox } bounded-queue-based : { mailbox-type : Akka.Dispatch.BoundedMailbox } unbounded-deque-based : { mailbox-type : Akka.Dispatch.UnboundedDequeBasedMailbox } bounded-deque-based : { mailbox-type : Akka.Dispatch.BoundedDequeBasedMailbox } logger-queue : { mailbox-type : Akka.Event.LoggerMailboxType } } debug : { receive : off autoreceive : off lifecycle : off fsm : off event-stream : off unhandled : on router-misconfiguration : off } serializers : { json : "Akka.Serialization.NewtonSoftJsonSerializer, Akka" bytes : "Akka.Serialization.ByteArraySerializer, Akka" akka-containers : "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" akka-misc : "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" primitive : "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" proto : "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" daemon-create : "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" akka-system-msg : "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" akka-cluster : "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" akka-cluster-client : "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" akka-pubsub : "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" } serialization-bindings : { System.Byte[] : bytes System.Object : json "Akka.Actor.ActorSelectionMessage, Akka" : akka-containers "Akka.Remote.DaemonMsgCreate, Akka.Remote" : daemon-create "Google.Protobuf.IMessage, Google.Protobuf" : proto "Akka.Actor.Identify, Akka" : akka-misc "Akka.Actor.ActorIdentity, Akka" : akka-misc "Akka.Actor.IActorRef, Akka" : akka-misc "Akka.Actor.PoisonPill, Akka" : akka-misc "Akka.Actor.Kill, Akka" : akka-misc "Akka.Actor.RemoteScope, Akka" : akka-misc "Akka.Routing.FromConfig, Akka" : akka-misc "Akka.Routing.DefaultResizer, Akka" : akka-misc "Akka.Routing.RoundRobinPool, Akka" : akka-misc "Akka.Routing.BroadcastPool, Akka" : akka-misc "Akka.Routing.RandomPool, Akka" : akka-misc "Akka.Routing.ScatterGatherFirstCompletedPool, Akka" : akka-misc "Akka.Routing.TailChoppingPool, Akka" : akka-misc "Akka.Routing.ConsistentHashingPool, Akka" : akka-misc "Akka.Configuration.Config, Akka" : akka-misc "Akka.Remote.RemoteWatcher+Heartbeat, Akka.Remote" : akka-misc "Akka.Remote.RemoteWatcher+HeartbeatRsp, Akka.Remote" : akka-misc "Akka.Remote.Routing.RemoteRouterConfig, Akka.Remote" : akka-misc "Akka.Dispatch.SysMsg.SystemMessage, Akka" : akka-system-msg System.String : primitive System.Int32 : primitive System.Int64 : primitive "Akka.Cluster.IClusterMessage, Akka.Cluster" : akka-cluster "Akka.Cluster.Routing.ClusterRouterPool, Akka.Cluster" : akka-cluster "Akka.Cluster.Tools.Client.IClusterClientMessage, Akka.Cluster.Tools" : akka-cluster-client "Akka.Cluster.Tools.PublishSubscribe.IDistributedPubSubMessage, Akka.Cluster.Tools" : akka-pubsub "Akka.Cluster.Tools.PublishSubscribe.Internal.SendToOneSubscriber, Akka.Cluster.Tools" : akka-pubsub } serialization-identifiers : { "Akka.Serialization.ByteArraySerializer, Akka" : 4 "Akka.Serialization.NewtonSoftJsonSerializer, Akka" : 1 "Akka.Remote.Serialization.ProtobufSerializer, Akka.Remote" : 2 "Akka.Remote.Serialization.DaemonMsgCreateSerializer, Akka.Remote" : 3 "Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote" : 6 "Akka.Remote.Serialization.MiscMessageSerializer, Akka.Remote" : 16 "Akka.Remote.Serialization.PrimitiveSerializers, Akka.Remote" : 17 "Akka.Remote.Serialization.SystemMessageSerializer, Akka.Remote" : 22 "Akka.Cluster.Serialization.ClusterMessageSerializer, Akka.Cluster" : 5 "Akka.Cluster.Tools.Client.Serialization.ClusterClientMessageSerializer, Akka.Cluster.Tools" : 15 "Akka.Cluster.Tools.PublishSubscribe.Serialization.DistributedPubSubMessageSerializer, Akka.Cluster.Tools" : 9 } serialization-settings : { } } scheduler : { tick-duration : 10ms ticks-per-wheel : 512 implementation : Akka.Actor.HashedWheelTimerScheduler shutdown-timeout : 5s } io : { pinned-dispatcher : { type : PinnedDispatcher executor : fork-join-executor } tcp : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } disabled-buffer-pool : { class : "Akka.IO.Buffers.DisabledBufferPool, Akka" buffer-size : 512 } buffer-pool : akka.io.tcp.disabled-buffer-pool max-channels : 256000 selector-association-retries : 10 batch-accept-limit : 10 register-timeout : 5s max-received-message-size : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher file-io-dispatcher : akka.actor.default-dispatcher file-io-transferTo-limit : 524288 finish-connect-retries : 5 windows-connection-abort-workaround-enabled : off outgoing-socket-force-ipv4 : false } udp : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } buffer-pool : akka.io.udp.direct-buffer-pool nr-of-socket-async-event-args : 32 max-channels : 4096 select-timeout : infinite selector-association-retries : 10 receive-throughput : 3 direct-buffer-size : 18432 direct-buffer-pool-limit : 1000 received-message-size-limit : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher } udp-connected : { direct-buffer-pool : { class : "Akka.IO.Buffers.DirectBufferPool, Akka" buffer-size : 512 buffers-per-segment : 500 initial-segments : 1 buffer-pool-limit : 1024 } buffer-pool : akka.io.udp-connected.direct-buffer-pool nr-of-socket-async-event-args : 32 max-channels : 4096 select-timeout : infinite selector-association-retries : 10 receive-throughput : 3 direct-buffer-size : 18432 direct-buffer-pool-limit : 1000 received-message-size-limit : unlimited trace-logging : off selector-dispatcher : akka.io.pinned-dispatcher worker-dispatcher : akka.actor.default-dispatcher management-dispatcher : akka.actor.default-dispatcher } dns : { dispatcher : akka.actor.default-dispatcher resolver : inet-address inet-address : { provider-object : Akka.IO.InetAddressDnsProvider positive-ttl : 30s negative-ttl : 10s cache-cleanup-interval : 120s use-ipv6 : true } } } coordinated-shutdown : { default-phase-timeout : "5 s" terminate-actor-system : on exit-clr : off run-by-clr-shutdown-hook : on run-by-actor-system-terminate : on phases : { before-service-unbind : { } service-unbind : { depends-on : [before-service-unbind] } service-requests-done : { depends-on : [service-unbind] } service-stop : { depends-on : [service-requests-done] } before-cluster-shutdown : { depends-on : [service-stop] } cluster-sharding-shutdown-region : { timeout : "10 s" depends-on : [before-cluster-shutdown] } cluster-leave : { depends-on : [cluster-sharding-shutdown-region] } cluster-exiting : { timeout : "10 s" depends-on : [cluster-leave] } cluster-exiting-done : { depends-on : [cluster-exiting] } cluster-shutdown : { depends-on : [cluster-exiting-done] } before-actor-system-terminate : { depends-on : [cluster-shutdown] } actor-system-terminate : { timeout : "10 s" depends-on : [before-actor-system-terminate] } } } remote : { startup-timeout : "10 s" shutdown-timeout : "10 s" flush-wait-on-shutdown : "2 s" use-passive-connections : on backoff-interval : "0.05 s" command-ack-timeout : "30 s" handshake-timeout : "15 s" use-dispatcher : akka.remote.default-remote-dispatcher untrusted-mode : off trusted-selection-paths : [] require-cookie : off secure-cookie : log-received-messages : off log-sent-messages : off log-remote-lifecycle-events : on log-frame-size-exceeding : off log-buffer-size-exceeding : 50000 transport-failure-detector : { implementation-class : Akka.Remote.DeadlineFailureDetector,Akka.Remote heartbeat-interval : 5s acceptable-heartbeat-pause : 50s } watch-failure-detector : { implementation-class : Akka.Remote.PhiAccrualFailureDetector,Akka.Remote heartbeat-interval : "1 s" threshold : 10.0 max-sample-size : 200 min-std-deviation : "100 ms" acceptable-heartbeat-pause : "10 s" unreachable-nodes-reaper-interval : 1s expected-response-after : "1 s" } retry-gate-closed-for : "5 s" prune-quarantine-marker-after : "5 d" quarantine-after-silence : "2 d" system-message-buffer-size : 20000 system-message-ack-piggyback-timeout : "0.3 s" resend-interval : "2 s" resend-limit : 200 initial-system-message-delivery-timeout : "3 m" enabled-transports : [akka.remote.dot-netty.tcp] adapters : { gremlin : Akka.Remote.Transport.FailureInjectorProvider,Akka.Remote trttl : Akka.Remote.Transport.ThrottlerProvider,Akka.Remote } helios : { tcp : { transport-class : "Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote.Transport.Helios" } } dot-netty : { tcp : { transport-class : Akka.Remote.Transport.DotNetty.TcpTransport,Akka.Remote applied-adapters : [] transport-protocol : tcp byte-order : little-endian port : 0 public-port : 0 hostname : public-hostname : 127.0.0.1 dns-use-ipv6 : false enforce-ip-family : false enable-ssl : false enable-backwards-compatibility : false connection-timeout : "15 s" batching : { enabled : true max-pending-writes : 30 max-pending-bytes : 16k flush-interval : 40ms } use-dispatcher-for-io : write-buffer-high-water-mark : 0b write-buffer-low-water-mark : 0b send-buffer-size : 2097152b receive-buffer-size : 2097152b maximum-frame-size : 1048576b backlog : 4096 tcp-nodelay : on tcp-keepalive : on tcp-reuse-addr : off-for-windows server-socket-worker-pool : { pool-size-min : 2 pool-size-factor : 1.0 pool-size-max : 2 } client-socket-worker-pool : { pool-size-min : 2 pool-size-factor : 1.0 pool-size-max : 2 } } udp : { transport-protocol : udp } ssl : { certificate : { path : password : use-thumprint-over-file : false thumpbrint : store-name : store-location : current-user } suppress-validation : false } } gremlin : { debug : off } default-remote-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor dedicated-thread-pool : { thread-count : 4 } } backoff-remote-dispatcher : { type : ForkJoinDispatcher executor : fork-join-executor dedicated-thread-pool : { thread-count : 4 } } } cluster : { seed-nodes : ["akka.tcp://mls-cluster-system@127.0.0.1:4062"] seed-node-timeout : 5s retry-unsuccessful-join-after : 10s shutdown-after-unsuccessful-join-seed-nodes : off auto-down-unreachable-after : 10s down-removal-margin : off downing-provider-class : allow-weakly-up-members : on roles : [api] run-coordinated-shutdown-when-down : on role : { } min-nr-of-members : 1 log-info : on log-info-verbose : off periodic-tasks-initial-delay : 1s gossip-interval : 5s gossip-time-to-live : 30s leader-actions-interval : 1s unreachable-nodes-reaper-interval : 1s publish-stats-interval : off use-dispatcher : gossip-different-view-probability : 0.8 reduce-gossip-different-view-probability : 400 failure-detector : { implementation-class : "Akka.Remote.PhiAccrualFailureDetector, Akka.Remote" heartbeat-interval : 5s threshold : 8.0 max-sample-size : 1000 min-std-deviation : "100 ms" acceptable-heartbeat-pause : 50s monitored-by-nr-of-members : 9 expected-response-after : "1 s" } split-brain-resolver : { active-strategy : off stable-after : 20s static-quorum : { quorum-size : undefined role : } keep-majority : { role : } keep-oldest : { down-if-alone : on role : } keep-refree : { address : down-all-if-less-than-nodes : 1 } } scheduler : { tick-duration : 33ms ticks-per-wheel : 512 } debug : { verbose-heartbeat-logging : off verbose-receive-gossip-logging : off } client : { receptionist : { name : receptionist role : number-of-contacts : 3 response-tunnel-receive-timeout : 30s use-dispatcher : heartbeat-interval : 2s acceptable-heartbeat-pause : 13s failure-detection-interval : 2s } initial-contacts : [] establishing-get-contacts-interval : 3s refresh-contacts-interval : 60s heartbeat-interval : 2s acceptable-heartbeat-pause : 13s buffer-size : 1000 reconnect-timeout : off } pub-sub : { name : distributedPubSubMediator role : routing-logic : random gossip-interval : 1s removed-time-to-live : 120s max-delta-elements : 3000 use-dispatcher : } singleton : { singleton-name : ClusterSingleton role : role1 hand-over-retry-interval : 1s min-number-of-hand-over-retries : 10 } } service : { name : Mirero.MLS.TestClusterApi description : Mirero.MLS.TestClusterApi } system : { name : mls-cluster-system } } petabridge : { cmd : { host : 0.0.0.0 port : 9111 log-palettes-on-startup : on } } mls : { service : { dbConnect : { ip : 192.168.60.193 port : 1521 service : MLSDB user : mlsuser password : mlsuser } } }
[2020/10/23 09:18:30.606][ 1][ INFO][Topshelf.Hosts.ConsoleRunHost] The Mirero.MLS.TestClusterApi service is now running, press Control+C to exit.
[2020/10/23 09:18:31.395][ 10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - path:
[2020/10/23 09:18:31.395][ 10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - frequency: 0
[2020/10/23 09:18:31.395][ 10][ INFO]ActorSystem(mls-cluster-system)[Akka.Actor.Internal.ActorSystemImpl] Service Command Line - active: False
[2020/10/23 09:18:31.395][ 10][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] petabridge.cmd host bound to [0.0.0.0:9111]
[2020/10/23 09:18:31.403][ 10][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Welcome from [akka.tcp://mls-cluster-system@127.0.0.1:4062]
[2020/10/23 09:18:32.867][ 27][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton manager started singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton]
[2020/10/23 09:18:32.867][ 27][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [Start -> Oldest] Akka.Cluster.Tools.Singleton.Uninitialized
[2020/10/23 09:18:33.626][ 24][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - is the new leader among reachable nodes (more leaders may exist)
[2020/10/23 09:18:40.751][ 10][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:3895] to [Up]
[2020/10/23 09:18:50.907][ 25][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8112] to [Up]
[2020/10/23 09:18:55.983][ 21][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8113] to [WeaklyUp]
[2020/10/23 09:18:58.454][ 25][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd[Petabridge.Cmd.Host.AkkaCmdListener] Received petabridge.cmd connection from client [127.0.0.1:3908]
[2020/10/23 09:19:09.655][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$a/$c[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:19:09.675][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$a/$c[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:19:11.206][ 7][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8113] to [Up]
[2020/10/23 09:19:15.035][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$b/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:19:15.194][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$b/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:19:17.039][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:19:17.118][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:19:17.294][ 5][ INFO]akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a[Akka.Actor.EmptyLocalActorRef] Message [ActorIdentity] from akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1 to akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a was not delivered. [1] dead letters encountered. If this is not an expected behavior then akka://mls-cluster-system/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$c/$a may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2020/10/23 09:19:18.779][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:19:18.938][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$d/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:19:20.511][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$e/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:19:20.638][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$e/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:19:22.575][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$f/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:19:22.735][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$f/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:19:26.900][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$g/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:19:27.028][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$g/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:19:28.735][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$h/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:19:28.894][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$h/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:19:30.301][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$i/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:19:30.365][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$i/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:19:31.844][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$j/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:19:31.924][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$j/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:19:33.475][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$k/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:19:33.635][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$k/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:20:09.618][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$l/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:09.674][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$l/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:20:11.763][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$m/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:11.842][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$m/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:20:13.840][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$n/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:20:13.918][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$n/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:16.127][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$o/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:20:16.205][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$o/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:17.968][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$p/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:20:18.126][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$p/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:19.697][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:20:19.851][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:21.315][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$r/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:20:21.376][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$r/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:20:24.216][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$s/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:20:24.278][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$s/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:20:26.782][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$t/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:20:26.859][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$t/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:28.825][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$u/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:28.979][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$u/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1, Subject=]
[2020/10/23 09:20:30.991][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$v/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:20:31.070][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$v/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:20:34.214][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$w/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:20:34.294][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$w/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:22:05.036][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$x/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:22:05.193][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$x/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:22:08.407][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:22:08.486][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:22:11.401][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$z/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:22:11.480][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$z/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:22:14.348][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$A/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:22:14.427][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$A/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:22:17.155][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$B/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:22:17.234][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$B/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:22:32.395][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$C/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:22:32.471][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$C/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:22:34.026][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$D/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:22:34.104][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$D/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:23:53.942][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$E/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:23:54.005][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$E/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:23:56.216][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$F/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:23:58.383][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$G/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:23:58.446][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$G/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:23:59.900][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$H/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:23:59.963][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$H/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:01.583][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$I/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:01.661][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$I/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:03.174][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$J/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:03.251][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$J/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:04.764][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$K/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:04.922][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$K/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:06.583][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$L/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:06.658][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$L/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:08.312][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$M/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:08.390][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$M/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:09.809][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$N/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:09.887][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$N/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:11.612][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$O/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:11.690][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$O/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:13.079][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$P/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:13.157][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$P/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:14.673][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:14.734][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Q/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:16.172][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$R/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:16.251][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$R/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:24.520][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$S/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:24.598][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$S/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:25.824][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$T/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:25.902][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$T/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:27.040][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$U/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:27.118][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$U/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:28.425][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$V/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:28.504][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$V/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:29.567][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$W/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:29.627][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$W/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:30.735][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$X/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:30.799][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$X/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:31.905][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:31.968][ 29][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Y/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:33.195][ 6][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Z/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:33.352][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$Z/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1, Subject=]
[2020/10/23 09:24:43.135][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$0/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:24:43.292][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$0/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:45.645][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$1/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:24:45.723][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$1/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:47.876][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$2/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:24:48.049][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$2/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:25:57.482][ 32][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$3/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:25:57.560][ 23][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$3/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:25:59.163][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$4/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:25:59.242][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$4/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:00.792][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$5/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:00.872][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$5/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:02.413][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$6/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:02.473][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$6/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:03.995][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$7/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:04.166][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$7/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:26:05.810][ 5][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$8/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:26:05.871][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$8/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:13.033][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$9/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:13.096][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$9/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:26:14.829][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$+/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:26:14.889][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$+/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:16.573][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$~/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:16.634][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$~/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:18.166][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$ab/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:18.246][ 4][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$ab/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:26:19.947][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$bb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:20.008][ 31][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$bb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:22.014][ 26][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$cb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:22.092][ 24][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$cb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:23.744][ 21][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$db/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:23.808][ 22][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$db/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:3895/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole1HelloActor/c1#776426495]]
[2020/10/23 09:26:25.759][ 20][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$eb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:25.835][ 25][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$eb/$b[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:27.673][ 7][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$fb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:26:27.750][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$fb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:29.202][ 10][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$gb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8113/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole3HelloActor/c1#616049922]]
[2020/10/23 09:26:29.357][ 9][ WARN]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/petabridge.cmd/127.0.0.1%3A3908/actor/handler/$gb/$a[Petabridge.Cmd.Host.Default.Actor.ActorTracer] Received unexpected ActorIdentity: [Id=akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor, Subject=[akka.tcp://mls-cluster-system@127.0.0.1:8112/remote/akka.tcp/mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor/ClusterSingleton/TestClusterRole2HelloActor/c1#19286537]]
[2020/10/23 09:35:05.318][ 22][ INFO][Topshelf.Hosts.ConsoleRunHost] Control+C detected, attempting to stop service.
[2020/10/23 09:35:05.324][ 20][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Marked address [akka.tcp://mls-cluster-system@127.0.0.1:3893] as [Leaving]
[2020/10/23 09:35:05.324][ 20][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Exited [akka.tcp://mls-cluster-system@127.0.0.1:3893]
[2020/10/23 09:35:05.324][ 20][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Oldest observed OldestChanged: [akka.tcp://mls-cluster-system@127.0.0.1:3893 -> ]
[2020/10/23 09:35:05.324][ 20][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [Oldest -> WasOldest] Akka.Cluster.Tools.Singleton.OldestData
[2020/10/23 09:35:06.355][ 5][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton manager stopping singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton]
[2020/10/23 09:35:06.355][ 5][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] ClusterSingletonManager state change [WasOldest -> Stopping] Akka.Cluster.Tools.Singleton.WasOldestData
[2020/10/23 09:35:06.355][ 5][ INFO]akka.tcp://mls-cluster-system@127.0.0.1:3893/user/ApiMasterActor[Akka.Cluster.Tools.Singleton.ClusterSingletonManager] Singleton actor [akka://mls-cluster-system/user/ApiMasterActor/ClusterSingleton] was terminated
[2020/10/23 09:35:08.477][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:3893] to [Exiting]
[2020/10/23 09:35:08.481][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:3895] to [Exiting]
[2020/10/23 09:35:08.481][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8112] to [Exiting]
[2020/10/23 09:35:08.481][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Leader is moving node [akka.tcp://mls-cluster-system@127.0.0.1:8113] to [Exiting]
[2020/10/23 09:35:08.520][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Exiting completed.
[2020/10/23 09:35:08.540][ 31][ INFO]Cluster (akka://mls-cluster-system)[Cluster (akka://mls-cluster-system)] Cluster Node [akka.tcp://mls-cluster-system@127.0.0.1:3893] - Shutting down...
[2020/10/23 09:35:08.682][ 1][ INFO][Topshelf.Hosts.ConsoleRunHost] The Mirero.MLS.TestClusterApi service has stopped.
@IgorFedchenko
petabridge.cmd
{
host = "0.0.0.0"
port = 9111
log-palettes-on-startup = on
}
akka
{
service.name = "TestClusterApi"
service.description = "TestClusterApi"
system.name = "mls-cluster-system"
stdout-loglevel = DEBUG
loglevel = DEBUG
log-config-on-start = on
loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
actor
{
provider = cluster
guardian-supervisor-strategy = "TestClusterApi.ActorService.UserSupervisorStrategy, TestClusterApi"
debug
{
receive = off
autoreceive = off
lifecycle = off
event-stream = off
unhandled = on
}
deployment
{
/ApiMasterActor
{
}
"/ApiMasterActor/*/TestClusterRole1HelloActor"
{
router = round-robin-pool
nr-of-instances = 100
cluster
{
enabled = on
max-nr-of-instances-per-node = 1
allow-local-routees = off
use-role = role1
}
}
"/ApiMasterActor/*/TestClusterRole2HelloActor"
{
router = round-robin-pool
nr-of-instances = 100
cluster
{
enabled = on
max-nr-of-instances-per-node = 1
allow-local-routees = off
use-role = role2
}
}
"/ApiMasterActor/*/TestClusterRole3HelloActor"
{
router = round-robin-pool
nr-of-instances = 100
cluster
{
enabled = on
max-nr-of-instances-per-node = 1
allow-local-routees = off
use-role = role3
}
}
}
}
remote
{
dot-netty.tcp
{
public-hostname = 127.0.0.1
port = 0
send-buffer-size = 2097152b # 2MB
receive-buffer-size = 2097152b # 2MB
maximum-frame-size = 1048576b # 1MB
}
transport-failure-detector
{
heartbeat-interval = 5s
acceptable-heartbeat-pause = 50s
}
}
extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]
cluster
{
seed-nodes = ["akka.tcp://mls-cluster-system@127.0.0.1:4062"]
roles = [api]
auto-down-unreachable-after = 10s
gossip-interval = 5s
gossip-time-to-live = 30s
failure-detector
{
heartbeat-interval = 5s
acceptable-heartbeat-pause = 50s
}
singleton
{
singleton-name = "ClusterSingleton"
role = "role1"
hand-over-retry-interval = 1s
min-number-of-hand-over-retries = 10
}
#singleton-proxy
#{
# singleton-name = ${akka.cluster.singleton.singleton-name}
# role = "role1"
# singleton-identification-interval = 1s
# buffer-size = 1000
#}
}
}
lighthouse
{
actorsystem: "mls-cluster-system" #POPULATE NAME OF YOUR ACTOR SYSTEM HERE
}
petabridge.cmd
{
host = "0.0.0.0"
port = 9115
}
akka
{
service.name = "TestLightHouse4061"
service.description = "TestLightHouse.4061"
system.name = "mls-cluster-testlighthouse-4061"
stdout-loglevel = DEBUG
loglevel = DEBUG
log-config-on-start = on
loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]
actor
{
provider = cluster
debug
{
receive = off
autoreceive = off
lifecycle = off
event-stream = off
unhandled = on
}
}
remote
{
log-remote-lifecycle-events = DEBUG
dot-netty.tcp
{
public-hostname = 127.0.0.1
port = 4062
}
transport-failure-detector
{
heartbeat-interval = 5s
acceptable-heartbeat-pause = 50s
}
}
extensions = ["Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"]
cluster
{
seed-nodes = ["akka.tcp://mls-cluster-system@127.0.0.1:4062"]
roles = [lighthouse]
auto-down-unreachable-after = 10s
gossip-interval = 5s # how often should the node send out gossip information?
gossip-time-to-live = 30s # discard incoming gossip messages if not handled within this duration
client.receptionist
{
role = lighthouse
response-tunnel-receive-timeout = 1d # 1day
}
failure-detector
{
heartbeat-interval = 5s
acceptable-heartbeat-pause = 50s
}
}
}
@kimbyungeun Ok, thanks! I was just hoping that you have a .net project/solution source you can share here to make reproduction simpler.
If you do not have any, we will try do make our own, this will just take more time.
@IgorFedchenko
Today I will separate the source code and upload it. Please wait a little.
We appreicate your help @kimbyungeun !
@IgorFedchenko Upload the sample program. I hope it helps. ClusterSample.zip
Hello. (안녕하세요) @Aaronontheweb
I have 2 questions about the hocon configuration of our sample(ClusterSample.zip).
acceptable-heartbeat-pause = 240s
240s
setting is correct.15s
is set in the customer environment, it is disassociated
from an unspecified service and Marking node(s) as UNREACHABLE
occurs.disassociated
does not occur when setting 15s
.@kimbyungeun Thank you very much! I or another member of the team will check this out
@IgorFedchenko
I tested it with Akka.NET v1.4.12 version but it is the same.
Akka.Remote v1.4.12 Akka.Cluster v1.4.12 Petabridge.Cmd.Cluster v0.8.0.1 Petabridge.Cmd.Common v0.8.0.1 Petabridge.Cmd.Host v0.8.0.1
PS C:\Users\be.kim> pbm 127.0.0.1:9111
____ _ _ _ _
| _ \ ___| |_ __ _| |__ _ __(_) __| | __ _ ___
| |_) / _ | __/ _` | '_ \| '__| |/ _` |/ _` |/ _ \
| __| __| || (_| | |_) | | | | (_| | (_| | __/
|_| \___|\__\__,_|_.__/|_| |_|\__,_|\__, |\___|
|___/
petabridge.cmd (0.8.0.0)
Copyright 2017 - 2020, Petabridge®.
successfully connected to [::ffff:127.0.0.1]:9111
Commands downloaded from server. type `help` to see what's available
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$a
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$a/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$b
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$b/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$c
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$c/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$d
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$d/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$e
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$e/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$f
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$f/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A10795
/user/petabridge.cmd/127.0.0.1%3A10795/actor
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$g
/user/petabridge.cmd/127.0.0.1%3A10795/actor/handler/$g/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm>
@kimbyungeun Thanks, was thinking of checking that!
@IgorFedchenko I think the issue here is we print out the results in any order when they arrive, so the output isn't sorted on a per-level basis in the hierarchy. i.e. all children at the same position in the hierarchy might print out differently each time it's run.
Answer would be to sort the actor paths I guess
@Aaronontheweb If I understood the issue right, the problem is that child actors sometimes are not displayed at all. From the latest logs above:
@kimbyungeun said that actors are running all the time, so they should be displayed all the time.
Right?
Ah that makes sense @IgorFedchenko - we do time out the search after a while so that may be what is happening here.
Hmm... I will check that - thanks for pointing the direction!
@Aaronontheweb I was able to reproduce this issue. But I have a question to you.
Long story short: we have Akka.net cluster, 3 nodes (there are more, but this is not important). Cluster singleton named ApiMasterActor
is created, and this singleton on start creates 3 child actors. Then first actor on scheduled time is sending message via IActorRef
to second actor, second is sending message to third, third just logging a message. And so on each 5 seconds, timer -> 1 -> 2 -> 3.
Sometimes, Petabridge.Cmd does not give any c1
actor:
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
but - sometimes, there is some c1 actor popping up, here and there, as a child of this "hello" actors. Sometimes, as a child of one of this actors, sometimes as a child of two, or even all three of them, for example:
[127.0.0.1:9111] pbm> actor hierarchy -d 10
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
So, the question is: does Akka.NET create some child actor automatically somehow, when actors are sending or maybe receiving messages, or they are children of a singleton, or under some other conditions?.. Because, there is nothing that creates this child c1
actor on the user side. And I failed to find anything like such child actors creation in the akka.net source in reasonable time.
@kimbyungeun So far, it seems to me like Petabridge.cmd is working fine, and it's just some temporary "service" actor created by Akka.NET under the hood, that is displayed in the hierarchy from time to time.
@IgorFedchenko Excuse, me.
temporary "service" actor
.akka.tcp://Sys@localhost:90/remote/akka/tcp/localhost:89/$a
is temporary "service" actor, isn't it?
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
is akka.tcp://test-cluster-system@127.0.0.1:8728/remote/akka.tcp/test-cluster-system@127.0.0.1:8722/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1#939527172
TestClusterApi
's actor hierarchy
...
[akka://test-cluster-system/user]
[akka://test-cluster-system/user/ApiMasterActor#226114292]
[akka://test-cluster-system/user/ApiMasterActor/singleton#1061564729]
[akka://test-cluster-system/user/ApiMasterActor/singleton/TestClusterRole1HelloActor#487008015]
[akka.tcp://test-cluster-system@127.0.0.1:8728/remote/akka.tcp/test-cluster-system@127.0.0.1:8722/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1#939527172]
[akka://test-cluster-system/user/ApiMasterActor/singleton/TestClusterRole2HelloActor#25528566]
[akka.tcp://test-cluster-system@127.0.0.1:8112/remote/akka.tcp/test-cluster-system@127.0.0.1:8722/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1#737878211]
[akka://test-cluster-system/user/ApiMasterActor/singleton/TestClusterRole3HelloActor#35980901]
[akka.tcp://test-cluster-system@127.0.0.1:8113/remote/akka.tcp/test-cluster-system@127.0.0.1:8722/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1#544185678]
[akka://test-cluster-system/user/ApiMasterActor/$a#1800049128]
[akka://test-cluster-system/user/petabridge.cmd#1953829908]
[akka://test-cluster-system/user/petabridge.cmd-log-memorizer#302235048]
[akka://test-cluster-system/user/pbm-uptime#1968095170]
@kimbyungeun
@seungyongshim
You said, akka.tcp://Sys@localhost:90/remote/akka/tcp/localhost:89/$a is temporary "service" actor, isn't it?
No, my question is about c1
actor. There is no doubt actors with names starting with $a
, $b
are some internal Akka.NET actors, and there is no problem with them in this issue. The problem is, how I see it right now, that sometimes c1
actors are printed out in the hierarchy, and sometimes they are not.
My current theory is that this c1
actors might not be live all the time - they may be activated on demand to send a remote message for example, or due to some other stuff required for Akka.Remote. Living for a short period of time. And that's why they may be included / excluded from the cmd
logs. We do not want petabridge.cmd to show actors that are not working at the moment, right?
And my question is that, I could not find piece of code in Akka that is responsible for such remote reference representation of remote actor link. If I would found it, I could see if this is normal that this reference is not displayed. Or, as @Aaronontheweb said, they might not be discovered due to the timeout of some kind.
Actually, let me try to increase command timeout and see if we will get c1
actors of all three nodes displayed stable.
All right guys, @Aaronontheweb was absolutely right. Current timeout per layer is 40ms
, and when changed to 400ms
utility gives all c1
actors on all three nodes, all the time. Not sure why this is happening with this c1
actors only, but anyway, increasing the timeout does the trick.
Now the bad news is that this timeout is hardcoded as a readonly static field of internal class, so there is no way to configure this on your side. I am going to update our tool to support -t
parameter, like actor hierarchy -d 10 -t 400
for example, to make this timeout configurable when someone is getting issues like that.
And we will push this update with 0.8.2
version of the Petabridge.Cmd.Host
package. Does it sound good @kimbyungeun ?
@kimbyungeun We have just released v0.8.2 of Petabridge.Cmd
package pack.
Try to update and use actor hierarchy -d 10 -t 400
and let us know if the problem is resolved this way.
We will also update tool documentation soon, but the -t
parameter stands for timeout-per-layer
and allows to increase default 40ms
timeout to 400ms
for example.
Resolved via Petabridge.Cmd v0.8.2
@IgorFedchenko @Aaronontheweb
It will be very helpful. Thank you very much.
actor hierarchy -d 10 -t 90
All actors are printed.
PS C:\Users\be.kim> pbm 127.0.0.1:9111
____ _ _ _ _
| _ \ ___| |_ __ _| |__ _ __(_) __| | __ _ ___
| |_) / _ | __/ _` | '_ \| '__| |/ _` |/ _` |/ _ \
| __| __| || (_| | |_) | | | | (_| | (_| | __/
|_| \___|\__\__,_|_.__/|_| |_|\__,_|\__, |\___|
|___/
petabridge.cmd (0.8.2.0)
Copyright 2017 - 2020, Petabridge®.
successfully connected to [::ffff:127.0.0.1]:9111
Commands downloaded from server. type `help` to see what's available
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$a
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$a/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$b
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$b/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm> actor hierarchy -d 10 -t 90
/user
/user/ApiMasterActor
/user/ApiMasterActor/$a
/user/ApiMasterActor/singleton
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor
/user/ApiMasterActor/singleton/TestClusterRole1HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor
/user/ApiMasterActor/singleton/TestClusterRole2HelloActor/c1
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor
/user/ApiMasterActor/singleton/TestClusterRole3HelloActor/c1
/user/pbm-uptime
/user/petabridge.cmd
/user/petabridge.cmd/127.0.0.1%3A12435
/user/petabridge.cmd/127.0.0.1%3A12435/actor
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$c
/user/petabridge.cmd/127.0.0.1%3A12435/actor/handler/$c/$a
/user/petabridge.cmd-log-memorizer
[127.0.0.1:9111] pbm>
@seungyongshim
Glad to hear it - please let us know if there's anything else we can do to help
https://github.com/akkadotnet/akka.net/issues/4585
Environment
ClusterSingletonManager
.Question
actor hierarchy
prints different results each time.our service
actor hierarchy -d 8
1st runReceived unexpected ActorIdentity
Warning messageactor hierarchy -d 8
2nd runReceived unexpected ActorIdentity
Warning message