search

typesafehub / activator-akka-distributed-workers-java

Typesafe Activator template for distributed workers with Akka cluster in Java.
Other
47 stars 34 forks source link

WorkExecutor stops getting invoked after putting some thread.sleep on it #11

Open anant1525 opened 6 years ago

anant1525 commented 6 years ago

Hi, I am facing some gap when I am testing the blocking call on Work Executor. in the example.

the modified code. `public class WorkExecutor extends AbstractActor {

private LoggingAdapter log = Logging.getLogger(getContext().system(), this);

@Override public Receive createReceive() { return receiveBuilder() .match(Work.class, message -> {

      Integer n = (Integer) message.job;

      log.info("inside WorkExecutor processing the job : {}",message);

      //introducing the blocking
      if (n == 3){

          try {
            Thread.sleep(10000);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
      }

      int n2 = n.intValue() * n.intValue();
      String result = n + " * " + n + " = " + n2;
      log.info("Produced result {}", result);
      getSender().tell(new Worker.WorkComplete(result), getSelf());

  })
  .build();

} `

and the corresponding logs: (see the work executor...)

[INFO] [10/06/2018 22:53:43.065] [main] [akka.remote.Remoting] Starting remoting [INFO] [10/06/2018 22:53:43.313] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://ClusterSystem@127.0.0.1:2551] [INFO] [10/06/2018 22:53:43.315] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://ClusterSystem@127.0.0.1:2551] [INFO] [10/06/2018 22:53:43.330] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Starting up... [INFO] [10/06/2018 22:53:43.450] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [10/06/2018 22:53:43.450] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Started up successfully [WARN] [10/06/2018 22:53:43.539] [ClusterSystem-akka.actor.default-dispatcher-14] [akka.tcp://ClusterSystem@127.0.0.1:2551/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation. [INFO] [10/06/2018 22:53:43.560] [ClusterSystem-akka.actor.default-dispatcher-10] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Node [akka.tcp://ClusterSystem@127.0.0.1:2551] is JOINING, roles [backend, dc-default] [INFO] [10/06/2018 22:53:43.578] [ClusterSystem-akka.actor.default-dispatcher-10] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Leader is moving node [akka.tcp://ClusterSystem@127.0.0.1:2551] to [Up] [INFO] [10/06/2018 22:53:43.646] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://ClusterSystem@127.0.0.1:2551/user/master] Singleton manager starting singleton actor [akka://ClusterSystem/user/master/singleton] [INFO] [10/06/2018 22:53:43.647] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://ClusterSystem@127.0.0.1:2551/user/master] ClusterSingletonManager state change [Start -> Oldest] [INFO] [10/06/2018 22:53:48.582] [main] [akka.remote.Remoting] Starting remoting [INFO] [10/06/2018 22:53:48.651] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://ClusterSystem@127.0.0.1:2552] [INFO] [10/06/2018 22:53:48.652] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://ClusterSystem@127.0.0.1:2552] [INFO] [10/06/2018 22:53:48.657] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2552] - Starting up... [WARN] [10/06/2018 22:53:48.663] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config [INFO] [10/06/2018 22:53:48.663] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2552] - Started up successfully [WARN] [10/06/2018 22:53:48.667] [ClusterSystem-akka.actor.default-dispatcher-2] [akka.tcp://ClusterSystem@127.0.0.1:2552/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation. [INFO] [10/06/2018 22:53:48.712] [main] [akka.remote.Remoting] Starting remoting [INFO] [10/06/2018 22:53:48.786] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://WorkerSystem@127.0.0.1:58365] [INFO] [10/06/2018 22:53:48.787] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://WorkerSystem@127.0.0.1:58365] [INFO] [10/06/2018 22:53:48.846] [ClusterSystem-akka.actor.default-dispatcher-6] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Received InitJoin message from [Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#1357334455]] to [akka.tcp://ClusterSystem@127.0.0.1:2551] [INFO] [10/06/2018 22:53:48.846] [ClusterSystem-akka.actor.default-dispatcher-6] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Sending InitJoinAck message from node [akka.tcp://ClusterSystem@127.0.0.1:2551] to [Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#1357334455]] [INFO] [10/06/2018 22:53:48.871] [WorkerSystem-akka.actor.default-dispatcher-2] [akka.tcp://WorkerSystem@127.0.0.1:58365/user/clusterClient] Connected to [akka.tcp://ClusterSystem@127.0.0.1:2551/system/receptionist] [WARN] [SECURITY][10/06/2018 22:53:48.876] [WorkerSystem-akka.remote.default-remote-dispatcher-6] [akka.serialization.Serialization(akka://WorkerSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$RegisterWorker] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:48.881] [ClusterSystem-akka.actor.default-dispatcher-7] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Node [akka.tcp://ClusterSystem@127.0.0.1:2552] is JOINING, roles [backend, dc-default] [INFO] [10/06/2018 22:53:48.896] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Worker registered: ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:53:48.991] [ClusterSystem-akka.actor.default-dispatcher-22] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2552] - Welcome from [akka.tcp://ClusterSystem@127.0.0.1:2551] [INFO] [10/06/2018 22:53:49.351] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:53:49.541] [ClusterSystem-akka.actor.default-dispatcher-5] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Leader is moving node [akka.tcp://ClusterSystem@127.0.0.1:2552] to [Up] [INFO] [10/06/2018 22:53:49.564] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.tcp://ClusterSystem@127.0.0.1:2552/user/master] ClusterSingletonManager state change [Start -> Younger] [INFO] [10/06/2018 22:53:53.826] [main] [akka.remote.Remoting] Starting remoting [INFO] [10/06/2018 22:53:53.888] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://ClusterSystem@127.0.0.1:58379] [INFO] [10/06/2018 22:53:53.888] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://ClusterSystem@127.0.0.1:58379] [INFO] [10/06/2018 22:53:53.890] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:58379] - Starting up... [WARN] [10/06/2018 22:53:53.892] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config [INFO] [10/06/2018 22:53:53.893] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:58379] - Started up successfully [WARN] [10/06/2018 22:53:53.897] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.tcp://ClusterSystem@127.0.0.1:58379/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation. [INFO] [10/06/2018 22:53:53.914] [ClusterSystem-akka.actor.default-dispatcher-12] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Received InitJoin message from [Actor[akka.tcp://ClusterSystem@127.0.0.1:58379/system/cluster/core/daemon/joinSeedNodeProcess-1#1918361624]] to [akka.tcp://ClusterSystem@127.0.0.1:2551] [INFO] [10/06/2018 22:53:53.914] [ClusterSystem-akka.actor.default-dispatcher-12] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Sending InitJoinAck message from node [akka.tcp://ClusterSystem@127.0.0.1:2551] to [Actor[akka.tcp://ClusterSystem@127.0.0.1:58379/system/cluster/core/daemon/joinSeedNodeProcess-1#1918361624]] [INFO] [10/06/2018 22:53:53.918] [ClusterSystem-akka.actor.default-dispatcher-10] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Node [akka.tcp://ClusterSystem@127.0.0.1:58379] is JOINING, roles [dc-default] [INFO] [10/06/2018 22:53:53.922] [ClusterSystem-akka.actor.default-dispatcher-6] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:58379] - Welcome from [akka.tcp://ClusterSystem@127.0.0.1:2551] [INFO] [10/06/2018 22:53:53.939] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.tcp://ClusterSystem@127.0.0.1:58379/user/frontend/masterProxy] Singleton identified at [akka.tcp://ClusterSystem@127.0.0.1:2551/user/master/singleton] [INFO] [10/06/2018 22:53:54.338] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:53:54.927] [ClusterSystem-akka.actor.default-dispatcher-22] [akka://ClusterSystem/user/producer] Produced work: 1 [WARN] [SECURITY][10/06/2018 22:53:54.931] [ClusterSystem-akka.remote.default-remote-dispatcher-13] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.Master$Work] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:54.939] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 1e4fdb85-b746-4f1d-ad7d-192b48684e31 [WARN] [SECURITY][10/06/2018 22:53:54.962] [ClusterSystem-akka.actor.default-dispatcher-5] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.WorkState$WorkAccepted] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:54.974] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:53:54.974] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:53:54.974] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [WARN] [SECURITY][10/06/2018 22:53:54.975] [ClusterSystem-akka.remote.default-remote-dispatcher-26] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.Master$Ack] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [WARN] [SECURITY][10/06/2018 22:53:54.977] [ClusterSystem-akka.remote.default-remote-dispatcher-26] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$WorkIsReady] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:54.980] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] Inside Worker Work is ready : worker idba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:53:55.541] [ClusterSystem-akka.actor.default-dispatcher-12] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Leader is moving node [akka.tcp://ClusterSystem@127.0.0.1:58379] to [Up] [INFO] [10/06/2018 22:53:57.988] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/producer] Produced work: 2 [INFO] [10/06/2018 22:53:57.991] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 80b26a93-b809-43bb-a221-e26a30c8681c [INFO] [10/06/2018 22:53:58.045] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:53:58.045] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:53:58.045] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:53:58.047] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] Inside Worker idle state : worker idba438def-2074-42b9-a522-9147c828ecdf [WARN] [SECURITY][10/06/2018 22:53:58.048] [WorkerSystem-akka.remote.default-remote-dispatcher-6] [akka.serialization.Serialization(akka://WorkerSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$WorkerRequestsWork] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:58.053] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkerRequestsWork with worker ba438def-2074-42b9-a522-9147c828ecdf [WARN] [SECURITY][10/06/2018 22:53:58.061] [ClusterSystem-akka.actor.default-dispatcher-8] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.WorkState$WorkStarted] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:58.065] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] 1111111 Giving worker ba438def-2074-42b9-a522-9147c828ecdf some work 1e4fdb85-b746-4f1d-ad7d-192b48684e31 [WARN] [SECURITY][10/06/2018 22:53:58.066] [ClusterSystem-akka.remote.default-remote-dispatcher-26] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.Master$Work] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:58.068] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] Inside idle state changing state to working.. Got work: 1 [INFO] [10/06/2018 22:53:58.068] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker/exec] inside WorkExecutor processing the job : Work{workId='1e4fdb85-b746-4f1d-ad7d-192b48684e31', job=1} [INFO] [10/06/2018 22:53:58.068] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker/exec] Produced result 1 1 = 1 [INFO] [10/06/2018 22:53:58.068] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] inside Working state. Work is complete. Result 1 1 = 1. [WARN] [SECURITY][10/06/2018 22:53:58.069] [WorkerSystem-akka.remote.default-remote-dispatcher-7] [akka.serialization.Serialization(akka://WorkerSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$WorkIsDone] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:58.073] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work 1e4fdb85-b746-4f1d-ad7d-192b48684e31 [INFO] [10/06/2018 22:53:58.073] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Work 1e4fdb85-b746-4f1d-ad7d-192b48684e31 is done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:53:58.074] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Work state of the worker ba438def-2074-42b9-a522-9147c828ecdf is Busy{work=1e4fdb85-b746-4f1d-ad7d-192b48684e31, deadline=Deadline(305859126391953 nanoseconds)} [INFO] [10/06/2018 22:53:58.074] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] changing worker state to idle [WARN] [SECURITY][10/06/2018 22:53:58.079] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.WorkState$WorkCompleted] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [WARN] [SECURITY][10/06/2018 22:53:58.085] [ClusterSystem-akka.remote.default-remote-dispatcher-26] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.Master$WorkResult] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:53:58.087] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] Got Ack from Master inside waitForWorkIsDoneAck, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> 1e4fdb85-b746-4f1d-ad7d-192b48684e31) [INFO] [10/06/2018 22:53:58.090] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/consumer] Consumed result: 1 1 = 1 [INFO] [10/06/2018 22:53:58.091] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkerRequestsWork with worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:53:58.095] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] 1111111 Giving worker ba438def-2074-42b9-a522-9147c828ecdf some work 80b26a93-b809-43bb-a221-e26a30c8681c [INFO] [10/06/2018 22:53:58.097] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker] Inside idle state changing state to working.. Got work: 2 [INFO] [10/06/2018 22:53:58.097] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker/exec] inside WorkExecutor processing the job : Work{workId='80b26a93-b809-43bb-a221-e26a30c8681c', job=2} [INFO] [10/06/2018 22:53:58.097] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker/exec] Produced result 2 2 = 4 [INFO] [10/06/2018 22:53:58.097] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker] inside Working state. Work is complete. Result 2 2 = 4. [INFO] [10/06/2018 22:53:58.099] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work 80b26a93-b809-43bb-a221-e26a30c8681c [INFO] [10/06/2018 22:53:58.099] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Work 80b26a93-b809-43bb-a221-e26a30c8681c is done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:53:58.100] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Work state of the worker ba438def-2074-42b9-a522-9147c828ecdf is Busy{work=80b26a93-b809-43bb-a221-e26a30c8681c, deadline=Deadline(305859155830547 nanoseconds)} [INFO] [10/06/2018 22:53:58.100] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] changing worker state to idle [INFO] [10/06/2018 22:53:58.104] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] Got Ack from Master inside waitForWorkIsDoneAck, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> 80b26a93-b809-43bb-a221-e26a30c8681c) [INFO] [10/06/2018 22:53:58.105] [ClusterSystem-akka.actor.default-dispatcher-22] [akka://ClusterSystem/user/consumer] Consumed result: 2 2 = 4 [INFO] [10/06/2018 22:53:58.830] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:53:59.349] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:01.064] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/producer] Produced work: 3 [INFO] [10/06/2018 22:54:01.066] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:01.070] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:01.070] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:01.070] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:01.075] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] Inside Worker idle state : worker idba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:01.078] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkerRequestsWork with worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:01.082] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] 1111111 Giving worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:01.085] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] Inside idle state changing state to working.. Got work: 3 [INFO] [10/06/2018 22:54:01.085] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker/exec] inside WorkExecutor processing the job : Work{workId='dd37150b-276d-4165-8f95-e1d389d34bda', job=3} [INFO] [10/06/2018 22:54:04.351] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:05.093] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/producer] Produced work: 4 [INFO] [10/06/2018 22:54:05.096] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: cfdde0a5-0890-4b29-826b-8fc0a98987ce [INFO] [10/06/2018 22:54:05.101] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:05.101] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:06.101] [WorkerSystem-akka.actor.default-dispatcher-5] [akka://WorkerSystem/user/worker] Time out in the working state for worker id: (ba438def-2074-42b9-a522-9147c828ecdf doing work -> dd37150b-276d-4165-8f95-e1d389d34bda) [WARN] [SECURITY][10/06/2018 22:54:06.102] [WorkerSystem-akka.remote.default-remote-dispatcher-6] [akka.serialization.Serialization(akka://WorkerSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$WorkFailed] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkFailed with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda failed by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Work state of the worker ba438def-2074-42b9-a522-9147c828ecdf is Busy{work=dd37150b-276d-4165-8f95-e1d389d34bda, deadline=Deadline(305862142751089 nanoseconds)} [INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] changing worker state to idle [INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] inside WorkFailed -- changing work state to idle for worker dd37150b-276d-4165-8f95-e1d389d34bda WARNING arguments left: 1 [WARN] [SECURITY][10/06/2018 22:54:06.110] [ClusterSystem-akka.actor.default-dispatcher-8] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.WorkState$WorkerFailed] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [INFO] [10/06/2018 22:54:06.114] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:06.114] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:06.114] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:08.833] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:09.107] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/producer] Produced work: 5 [INFO] [10/06/2018 22:54:09.109] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 8727e25a-12e9-4fa4-b1a3-d391549196a6 [INFO] [10/06/2018 22:54:09.253] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:09.253] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:09.253] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:09.338] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:11.085] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker/exec] Produced result 3 * 3 = 9 [INFO] [10/06/2018 22:54:12.268] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/producer] Produced work: 6 [INFO] [10/06/2018 22:54:12.273] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: cf0cfcd7-c16d-4bea-a316-3a7b3f69d5b7 [INFO] [10/06/2018 22:54:12.280] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:12.280] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:12.280] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:14.339] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:16.296] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/producer] Produced work: 7 [INFO] [10/06/2018 22:54:16.300] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 900b4ee1-cf61-4c7e-813e-4959604584b8 [INFO] [10/06/2018 22:54:16.306] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:16.306] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:16.306] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:18.830] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:19.348] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:21.318] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:54:21.323] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:21.323] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:21.327] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/producer] Produced work: 8 [INFO] [10/06/2018 22:54:21.329] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 2052afb2-e19f-4fd3-b474-8a8fdd73ede2 [INFO] [10/06/2018 22:54:21.332] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:21.332] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:21.332] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:24.342] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:25.349] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/producer] Produced work: 9 [INFO] [10/06/2018 22:54:25.351] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 98ac1ecf-223b-4f3d-a56b-8d59fa5ce544 [INFO] [10/06/2018 22:54:25.354] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:25.354] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:25.354] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:28.821] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:29.340] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:30.369] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:54:30.371] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:30.371] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:33.367] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/producer] Produced work: 10 [INFO] [10/06/2018 22:54:33.369] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: b3db93cb-521a-4f9f-ab35-5ad0d7895b32 [INFO] [10/06/2018 22:54:33.372] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:33.372] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:33.372] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:34.352] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:38.388] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:54:38.390] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:38.390] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:38.830] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:39.342] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:40.388] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/producer] Produced work: 11 [INFO] [10/06/2018 22:54:40.391] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 89984b02-19d9-404b-945d-0e7a8d3529ed [INFO] [10/06/2018 22:54:40.402] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:40.402] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:40.402] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:44.340] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:45.420] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:54:45.423] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:45.423] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:48.419] [ClusterSystem-akka.actor.default-dispatcher-22] [akka://ClusterSystem/user/producer] Produced work: 12 [INFO] [10/06/2018 22:54:48.422] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 3bce5c92-7e5c-46bf-b551-dd6905294c9f [INFO] [10/06/2018 22:54:48.436] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:48.436] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:48.436] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:48.830] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:49.353] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:53.448] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:54:53.450] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:53.450] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:54.339] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:54.455] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/producer] Produced work: 13 [INFO] [10/06/2018 22:54:54.458] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 44558722-46c9-446e-ad9d-65a783e4ff86 [INFO] [10/06/2018 22:54:54.463] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:54.463] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:54.463] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:54:58.824] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:54:59.340] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:54:59.482] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:54:59.484] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:54:59.484] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:01.475] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/producer] Produced work: 14 [INFO] [10/06/2018 22:55:01.477] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 164c35e5-d857-4b95-8c2d-f64deb707280 [INFO] [10/06/2018 22:55:01.481] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:01.481] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:01.481] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:04.353] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:06.499] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:55:06.501] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:55:06.501] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:08.509] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/producer] Produced work: 15 [INFO] [10/06/2018 22:55:08.511] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: d87b9664-98bf-4c11-ac24-47e57579552c [INFO] [10/06/2018 22:55:08.515] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:08.515] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:08.515] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:08.834] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:09.341] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:13.530] [WorkerSystem-akka.actor.default-dispatcher-5] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:55:13.532] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:55:13.532] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:14.351] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:15.527] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/producer] Produced work: 16 [INFO] [10/06/2018 22:55:15.532] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 38f5deec-35cf-47fa-b8e4-1fd33884d7d4 [INFO] [10/06/2018 22:55:15.540] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:15.541] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:15.541] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:18.560] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/producer] Produced work: 17 [INFO] [10/06/2018 22:55:18.562] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 1144ec1f-e8e0-4fba-83f3-21ca2f31839f [INFO] [10/06/2018 22:55:18.567] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:18.567] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:18.567] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:18.831] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:19.350] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:22.580] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/producer] Produced work: 18 [INFO] [10/06/2018 22:55:22.582] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 22810256-d265-4b9f-bd99-edce97e60cd1 [INFO] [10/06/2018 22:55:22.650] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:22.650] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:22.650] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:24.340] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:27.662] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:55:27.664] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:55:27.664] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:28.824] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:29.345] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:30.669] [ClusterSystem-akka.actor.default-dispatcher-21] [akka://ClusterSystem/user/producer] Produced work: 19 [INFO] [10/06/2018 22:55:30.671] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 86bca828-4ec3-498e-8323-442e79085666 [INFO] [10/06/2018 22:55:30.674] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:30.674] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:30.674] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:34.341] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:35.690] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:55:35.691] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:55:35.691] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:38.833] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:39.353] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:39.690] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/producer] Produced work: 20 [INFO] [10/06/2018 22:55:39.693] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 47ad1a5b-d869-4f8a-a91d-e047288601a9 [INFO] [10/06/2018 22:55:39.697] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:39.697] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:39.697] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:44.343] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:44.710] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:55:44.712] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:55:44.712] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:46.710] [ClusterSystem-akka.actor.default-dispatcher-21] [akka://ClusterSystem/user/producer] Produced work: 21 [INFO] [10/06/2018 22:55:46.712] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: df36abc0-004a-4fab-9e98-c13e371e0035 [INFO] [10/06/2018 22:55:46.816] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:46.816] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:46.816] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:48.835] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:49.335] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:51.822] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:55:51.825] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:55:51.825] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:54.342] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:54.829] [ClusterSystem-akka.actor.default-dispatcher-21] [akka://ClusterSystem/user/producer] Produced work: 22 [INFO] [10/06/2018 22:55:54.831] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 211ab4fd-e57f-41f5-b492-7b9b0627d785 [INFO] [10/06/2018 22:55:54.834] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:54.834] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:54.834] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:55:58.835] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:55:59.356] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:55:59.854] [WorkerSystem-akka.actor.default-dispatcher-5] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:55:59.856] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:55:59.856] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:56:02.847] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/producer] Produced work: 23 [INFO] [10/06/2018 22:56:02.849] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: a63d4fe7-1d75-495b-9a23-47e9e8b45bd4 [INFO] [10/06/2018 22:56:02.913] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:02.913] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:02.913] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:04.342] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:56:07.934] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:56:07.936] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:56:07.936] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:56:08.836] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:56:09.355] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:56:10.938] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/producer] Produced work: 24 [INFO] [10/06/2018 22:56:10.941] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 8a1dbac1-7458-4b8e-9d6e-a8c606b8e7ce [INFO] [10/06/2018 22:56:11.061] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:11.061] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:11.061] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:14.335] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:56:16.085] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:56:16.087] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:56:16.087] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:56:18.078] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/producer] Produced work: 25 [INFO] [10/06/2018 22:56:18.081] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 2ae5a703-f979-40cc-b1b9-34a8ad80b56f [INFO] [10/06/2018 22:56:18.086] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:18.086] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:18.086] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:18.825] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:56:19.345] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:56:23.103] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:56:23.105] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:56:23.105] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:56:24.342] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:56:25.089] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/producer] Produced work: 26 [INFO] [10/06/2018 22:56:25.091] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 4d6e53f1-d6f8-4478-a86d-4c706c0c819d [INFO] [10/06/2018 22:56:25.096] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:25.096] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:25.096] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:28.834] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:56:29.112] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/producer] Produced work: 27 [INFO] [10/06/2018 22:56:29.115] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 71e6b052-fd4a-4732-89cc-d4e7b99d7e5e [INFO] [10/06/2018 22:56:29.118] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:29.118] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:29.118] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:29.346] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick [INFO] [10/06/2018 22:56:34.132] [WorkerSystem-akka.actor.default-dispatcher-5] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda) [INFO] [10/06/2018 22:56:34.133] [ClusterSystem-akka.actor.default-dispatcher-21] [akka://ClusterSystem/user/producer] Produced work: 28 [INFO] [10/06/2018 22:56:34.135] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda [INFO] [10/06/2018 22:56:34.135] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf [INFO] [10/06/2018 22:56:34.136] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 8a5340d2-f05a-4d9c-8d54-4d0531c3570c [INFO] [10/06/2018 22:56:34.139] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:34.139] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:34.139] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf] [INFO] [10/06/2018 22:56:34.346] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick

thanks in advance.

patriknw commented 6 years ago

sleep should never be used inside actors

anant1525 commented 6 years ago

@patriknw yes I agree its just used to simulate the blocking call. say if I am using a DB related task and it is taking time after some period of time say after 6-7 hours depending upon DB load.

If my design has above flow then how will I avoid such scenarios.