search

derekkraan / horde

Horde is a distributed Supervisor and Registry backed by DeltaCrdt
MIT License
1.32k stars 106 forks source link

Example dynamic cluster membership with tests. #155

Open mgwidmann opened 5 years ago

mgwidmann commented 5 years ago

Currently the tests fail due to the issue #154

@derekkraan Can you get the crashes to stop happening?

I know its a lot of output, but this is everything output from running mix test. There are only 2 tests.

✗ mix test

22:06:02.822 [info]  Application logger started at :nonode@nohost

22:06:02.825 [info]  Application jason started at :nonode@nohost

22:06:02.825 [info]  Application libcluster started at :nonode@nohost

22:06:02.826 [info]  Application local_cluster started at :nonode@nohost

22:06:02.833 [info]  Child :telemetry_handler_table of Supervisor :telemetry_sup started
Pid: #PID<0.192.0>
Start Call: :telemetry_handler_table.start_link()
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.833 [info]  Application telemetry started at :nonode@nohost

22:06:02.837 [info]  Child :telemetry_poller_default of Supervisor :telemetry_poller_sup started
Pid: #PID<0.197.0>
Start Call: :telemetry_poller.start_link([measurements: [:memory, :total_run_queue_lengths], name: :telemetry_poller_default])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.837 [info]  Application telemetry_poller started at :nonode@nohost

22:06:02.842 [info]  Application merkle_map started at :nonode@nohost

22:06:02.842 [info]  Application delta_crdt started at :nonode@nohost

22:06:02.844 [info]  Application libring started at :nonode@nohost

22:06:02.844 [info]  Application horde started at :nonode@nohost

22:06:02.863 [info]  Child Dynamic.EntityRegistry.Crdt of Supervisor Dynamic.EntityRegistry.Supervisor started
Pid: #PID<0.210.0>
Start Call: DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, [sync_interval: 300, max_sync_size: :infinite, shutdown: 30000, crdt: DeltaCrdt.AWLWWMap, on_diffs: #Function<0.13952030/1 in Horde.RegistrySupervisor.delta_crdt_options/1>, name: Dynamic.EntityRegistry.Crdt])
Restart: :permanent
Shutdown: 30000
Type: :worker

22:06:02.863 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:02.870 [info]  Child Dynamic.EntityRegistry of Supervisor Dynamic.EntityRegistry.Supervisor started
Pid: #PID<0.211.0>
Start Call: Horde.RegistryImpl.start_link([name: Dynamic.EntityRegistry, members: [Dynamic.EntityRegistry]])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.870 [info]  Child Horde.Registry of Supervisor Dynamic.Supervisor started
Pid: #PID<0.209.0>
Start Call: Horde.Registry.start_link([id: Horde.Registry, name: Dynamic.EntityRegistry, keys: :unique])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:02.877 [info]  Child Dynamic.EntitySupervisor.Crdt of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.213.0>
Start Call: DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, [sync_interval: 300, max_sync_size: :infinite, shutdown: 30000, crdt: DeltaCrdt.AWLWWMap, on_diffs: #Function<0.83009929/1 in Horde.SupervisorSupervisor.delta_crdt_options/1>, name: Dynamic.EntitySupervisor.Crdt])
Restart: :permanent
Shutdown: 30000
Type: :worker

22:06:02.877 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:02.877 [info]  Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.214.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.877 [info]  Child Horde.GracefulShutdownManager of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.215.0>
Start Call: GenServer.start_link(Horde.GracefulShutdownManager, Dynamic.EntitySupervisor.Crdt, [name: Dynamic.EntitySupervisor.GracefulShutdownManager])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.878 [info]  Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.216.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:02.878 [info]  Child Horde.SignalShutdown of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.217.0>
Start Call: GenServer.start_link(Horde.SignalShutdown, [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.880 [info]  Child Dynamic.EntitySupervisor_telemetry_poller of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.218.0>
Start Call: :telemetry_poller.start_link([measurements: [{:process_info, [name: Dynamic.EntitySupervisor, event: [:horde, :supervisor], keys: [:message_queue_len]]}, {Horde.SupervisorTelemetryPoller, :poll, [Dynamic.EntitySupervisor]}], period: 5000, name: Dynamic.EntitySupervisor_telemetry_poller])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.880 [info]  Child Horde.Supervisor of Supervisor Dynamic.Supervisor started
Pid: #PID<0.212.0>
Start Call: Horde.Supervisor.start_link([name: Dynamic.EntitySupervisor, strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:02.880 [info]  Child NodeListener of Supervisor Dynamic.Supervisor started
Pid: #PID<0.219.0>
Start Call: NodeListener.start_link([])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.886 [info]  Child Dynamic.ClusterConnector of Supervisor Dynamic.Supervisor started
Pid: #PID<0.220.0>
Start Call: Task.start_link(#Function<0.1039468/0 in Dynamic.Application.start/2>)
Restart: :transient
Shutdown: 5000
Type: :worker

22:06:02.886 [info]  Application dynamic started at :nonode@nohost

22:06:02.901 [info]  Child :erl_epmd of Supervisor :net_sup started
Pid: #PID<0.224.0>
Start Call: :erl_epmd.start_link()
Restart: :permanent
Shutdown: 2000
Type: :worker

22:06:02.904 [info]  Child :auth of Supervisor :net_sup started
Pid: #PID<0.225.0>
Start Call: :auth.start_link()
Restart: :permanent
Shutdown: 2000
Type: :worker

22:06:02.907 [info]  Child :net_kernel of Supervisor :net_sup started
Pid: #PID<0.226.0>
Start Call: :net_kernel.start_link([:"manager@127.0.0.1"], false)
Restart: :permanent
Shutdown: 2000
Type: :worker

22:06:02.907 [info]  Child :net_sup_dynamic of Supervisor :kernel_sup started
Pid: #PID<0.223.0>
Start Call: :erl_distribution.start_link([:"manager@127.0.0.1"], false)
Restart: :permanent
Shutdown: 1000
Type: :supervisor

22:06:02.921 [info]  Child ExUnit.Server of Supervisor ExUnit.Supervisor started
Pid: #PID<0.236.0>
Start Call: ExUnit.Server.start_link([])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.921 [info]  Child ExUnit.CaptureServer of Supervisor ExUnit.Supervisor started
Pid: #PID<0.237.0>
Start Call: ExUnit.CaptureServer.start_link([])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.921 [info]  Child ExUnit.OnExitHandler of Supervisor ExUnit.Supervisor started
Pid: #PID<0.238.0>
Start Call: ExUnit.OnExitHandler.start_link([])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.921 [info]  Application ex_unit started at :"manager@127.0.0.1"
warning: variable "node1_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:58

warning: variable "node2_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:62

22:06:03.009 [info]  Child of Supervisor :inet_gethost_native_sup started
Pid: #PID<0.252.0>
Start Call: :inet_gethost_native.init([])

22:06:03.009 [info]  Child :inet_gethost_native_sup of Supervisor :kernel_safe_sup started
Pid: #PID<0.251.0>
Start Call: :inet_gethost_native.start_link()
Restart: :temporary
Shutdown: 1000
Type: :worker

22:06:05.240 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:05.240 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:05.278 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:05.278 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor
warning: redefining module Dynamic.EntityTest (current version defined in memory)
  test/dynamic_test.exs:1

warning: variable "node1_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:58

warning: variable "node2_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:62

Test nodes that are now online: [:"test-dynamic-entity1@127.0.0.1", :"test-dynamic-entity2@127.0.0.1"]

  1) test multiple nodes a node goes down (Dynamic.EntityTest)
     test/dynamic_test.exs:85
     ** (CaseClauseError) no case clause matching: []
     code: assert Dynamic.get_entity(node1_entity_str) == node1_entity_data
     stacktrace:
       (dynamic) lib/dynamic.ex:17: Dynamic.get_entity_pid/1
       (dynamic) lib/dynamic.ex:11: Dynamic.get_entity/1
       test/dynamic_test.exs:99: (test)

22:06:06.351 [error] Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor terminated
** (exit) normal
Pid: #PID<0.216.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:06.361 [error] GenServer Dynamic.EntitySupervisor terminating
** (stop) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:581: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:358: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member, {Dynamic.EntitySupervisor, :"test-dynamic-entity1@127.0.0.1"}}, 1}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, local_process_count: 4, members: %{{Dynamic.EntitySupervisor, :"manager@127.0.0.1"} => 1, {Dynamic.EntitySupervisor, :"test-dynamic-entity1@127.0.0.1"} => 1}, members_info: %{{Dynamic.EntitySupervisor, :"manager@127.0.0.1"} => %Horde.Supervisor.Member{name: {Dynamic.EntitySupervisor, :"manager@127.0.0.1"}, status: :alive}, {Dynamic.EntitySupervisor, :"test-dynamic-entity1@127.0.0.1"} => %Horde.Supervisor.Member{name: {Dynamic.EntitySupervisor, :"test-dynamic-entity1@127.0.0.1"}, status: :uninitialized}}, name: Dynamic.EntitySupervisor, name_to_supervisor_ref: %{{Dynamic.EntitySupervisor, :"manager@127.0.0.1"} => #Reference<0.2045498330.1376518145.173581>, {Dynamic.EntitySupervisor, :nonode@nohost} => #Reference<0.2045498330.1376518145.173478>}, process_pid_to_id: %{#PID<17550.251.0> => 203344144639827332175378368881323364659, #PID<17550.252.0> => 336577234915540442433944746061623993228, #PID<17550.253.0> => 230434561701108535677139095368514670192, #PID<0.283.0> => 1308682357906732534861808327661391099, #PID<0.294.0> => 203344144639827332175378368881323364659, #PID<0.297.0> => 230434561701108535677139095368514670192, #PID<0.298.0> => 336577234915540442433944746061623993228}, processes_by_id: %{1308682357906732534861808327661391099 => {{Dynamic.EntitySupervisor, :"manager@127.0.0.1"}, %{id: 1308682357906732534861808327661391099, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["one", %{one: "data"}]}}, #PID<0.283.0>}, 203344144639827332175378368881323364659 => {{Dynamic.EntitySupervisor, :"manager@127.0.0.1"}, %{id: 203344144639827332175378368881323364659, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["two", %{two: "data"}]}}, #PID<0.294.0>}, 230434561701108535677139095368514670192 => {{Dynamic.EntitySupervisor, :"manager@127.0.0.1"}, %{id: 230434561701108535677139095368514670192, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["four", %{four: "data"}]}}, #PID<0.297.0>}, 336577234915540442433944746061623993228 => {{Dynamic.EntitySupervisor, :"manager@127.0.0.1"}, %{id: 336577234915540442433944746061623993228, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["three", %{three: "data"}]}}, #PID<0.298.0>}}, shutting_down: false, supervisor_options: [name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.2045498330.1376518145.173478> => {Dynamic.EntitySupervisor, :nonode@nohost}, #Reference<0.2045498330.1376518145.173581> => {Dynamic.EntitySupervisor, :"manager@127.0.0.1"}}, waiting_for_quorum: []}

22:06:06.361 [error] Process Dynamic.EntitySupervisor (#PID<0.214.0>) terminating
** (exit) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:581: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:358: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Initial Call: Horde.SupervisorImpl.init/1
Ancestors: [Dynamic.EntitySupervisor.Supervisor, Dynamic.Supervisor, #PID<0.207.0>]
Message Queue Length: 1
Messages: [{:"$gen_call", {#PID<0.217.0>, #Reference<0.2045498330.1376518145.174133>}, :horde_shutting_down}]
Links: [#PID<0.212.0>]
Dictionary: [rand_seed: {%{bits: 58, jump: #Function<8.10897371/1 in :rand.mk_alg/1>, next: #Function<5.10897371/1 in :rand.mk_alg/1>, type: :exrop, uniform: #Function<6.10897371/1 in :rand.mk_alg/1>, uniform_n: #Function<7.10897371/2 in :rand.mk_alg/1>, weak_low_bits: 1}, [31528902473244349 | 842880235163533]}]
Trapping Exits: true
Status: :running
Heap Size: 6772
Stack Size: 27
Reductions: 93712

22:06:06.362 [error] GenServer #PID<0.217.0> terminating
** (stop) exited in: GenServer.call(Dynamic.EntitySupervisor, :horde_shutting_down, 5000)
    ** (EXIT) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:989: GenServer.call/3
    (horde) lib/horde/signal_shutdown.ex:21: anonymous fn/1 in Horde.SignalShutdown.terminate/2
    (elixir) lib/enum.ex:769: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:769: Enum.each/2
    (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
    (stdlib) gen_server.erl:858: :gen_server.terminate/10
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.212.0>, :shutdown}
State: [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor]

22:06:06.362 [error] Process #PID<0.217.0> terminating
** (exit) exited in: GenServer.call(Dynamic.EntitySupervisor, :horde_shutting_down, 5000)
    ** (EXIT) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:989: GenServer.call/3
    (horde) lib/horde/signal_shutdown.ex:21: anonymous fn/1 in Horde.SignalShutdown.terminate/2
    (elixir) lib/enum.ex:769: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:769: Enum.each/2
    (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
    (stdlib) gen_server.erl:858: :gen_server.terminate/10
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Initial Call: Horde.SignalShutdown.init/1
Ancestors: [Dynamic.EntitySupervisor.Supervisor, Dynamic.Supervisor, #PID<0.207.0>]
Message Queue Length: 0
Messages: []
Links: []
Dictionary: []
Trapping Exits: true
Status: :running
Heap Size: 6772
Stack Size: 27
Reductions: 4727

22:06:06.362 [error] Child Horde.SignalShutdown of Supervisor Dynamic.EntitySupervisor.Supervisor shut down abnormally
** (exit) exited in: GenServer.call(Dynamic.EntitySupervisor, :horde_shutting_down, 5000)
    ** (EXIT) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
Pid: #PID<0.217.0>
Start Call: GenServer.start_link(Horde.SignalShutdown, [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.362 [error] Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor shut down abnormally
** (exit) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
Pid: #PID<0.214.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.372 [info]  Child Dynamic.EntitySupervisor.Crdt of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.307.0>
Start Call: DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, [sync_interval: 300, max_sync_size: :infinite, shutdown: 30000, crdt: DeltaCrdt.AWLWWMap, on_diffs: #Function<0.83009929/1 in Horde.SupervisorSupervisor.delta_crdt_options/1>, name: Dynamic.EntitySupervisor.Crdt])
Restart: :permanent
Shutdown: 30000
Type: :worker

22:06:06.372 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:06.372 [info]  Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.308.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.372 [info]  Child Horde.GracefulShutdownManager of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.309.0>
Start Call: GenServer.start_link(Horde.GracefulShutdownManager, Dynamic.EntitySupervisor.Crdt, [name: Dynamic.EntitySupervisor.GracefulShutdownManager])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.372 [info]  Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.310.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:06.372 [info]  Child Horde.SignalShutdown of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.311.0>
Start Call: GenServer.start_link(Horde.SignalShutdown, [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.373 [info]  Child Dynamic.EntitySupervisor_telemetry_poller of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.312.0>
Start Call: :telemetry_poller.start_link([measurements: [{:process_info, [name: Dynamic.EntitySupervisor, event: [:horde, :supervisor], keys: [:message_queue_len]]}, {Horde.SupervisorTelemetryPoller, :poll, [Dynamic.EntitySupervisor]}], period: 5000, name: Dynamic.EntitySupervisor_telemetry_poller])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:07.868 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:07.868 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:07.903 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:07.903 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor
warning: redefining module Dynamic.EntityTest (current version defined in memory)
  test/dynamic_test.exs:1

warning: variable "node1_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:58

warning: variable "node2_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:62

Test nodes that are now online: [:"test-dynamic-entity1@127.0.0.1", :"test-dynamic-entity2@127.0.0.1"]

22:06:08.327 [error] Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor terminated
** (exit) normal
Pid: #PID<0.310.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:08.328 [error] GenServer Dynamic.EntitySupervisor terminating
** (stop) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:581: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:358: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member_node_info, {Dynamic.EntitySupervisor, :"manager@127.0.0.1"}}, %Horde.Supervisor.Member{name: {Dynamic.EntitySupervisor, :"manager@127.0.0.1"}, status: :shutting_down}}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, local_process_count: 1, members: %{{Dynamic.EntitySupervisor, :"manager@127.0.0.1"} => 1}, members_info: %{{Dynamic.EntitySupervisor, :"manager@127.0.0.1"} => %Horde.Supervisor.Member{name: {Dynamic.EntitySupervisor, :"manager@127.0.0.1"}, status: :alive}}, name: Dynamic.EntitySupervisor, name_to_supervisor_ref: %{{Dynamic.EntitySupervisor, :"manager@127.0.0.1"} => #Reference<0.2045498330.1376518146.173040>, {Dynamic.EntitySupervisor, :"test-dynamic-entity2@127.0.0.1"} => #Reference<0.2045498330.1376518145.174676>}, process_pid_to_id: %{#PID<0.337.0> => 30956948180949210312828241188152995010}, processes_by_id: %{30956948180949210312828241188152995010 => {{Dynamic.EntitySupervisor, :"manager@127.0.0.1"}, %{id: 30956948180949210312828241188152995010, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["one", %{one: "data"}]}}, #PID<0.337.0>}}, shutting_down: true, supervisor_options: [name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.2045498330.1376518145.174676> => {Dynamic.EntitySupervisor, :"test-dynamic-entity2@127.0.0.1"}, #Reference<0.2045498330.1376518146.173040> => {Dynamic.EntitySupervisor, :"manager@127.0.0.1"}}, waiting_for_quorum: []}

22:06:08.329 [error] Process Dynamic.EntitySupervisor (#PID<0.308.0>) terminating
** (exit) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:581: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:358: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Initial Call: Horde.SupervisorImpl.init/1
Ancestors: [Dynamic.EntitySupervisor.Supervisor, Dynamic.Supervisor, #PID<0.207.0>]
Message Queue Length: 0
Messages: []
Links: [#PID<0.212.0>]
Dictionary: [rand_seed: {%{bits: 58, jump: #Function<8.10897371/1 in :rand.mk_alg/1>, next: #Function<5.10897371/1 in :rand.mk_alg/1>, type: :exrop, uniform: #Function<6.10897371/1 in :rand.mk_alg/1>, uniform_n: #Function<7.10897371/2 in :rand.mk_alg/1>, weak_low_bits: 1}, [260826826645588017 | 17108248010956880]}]
Trapping Exits: true
Status: :running
Heap Size: 4185
Stack Size: 27
Reductions: 65646

22:06:08.329 [error] Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor shut down abnormally
** (exit) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
Pid: #PID<0.308.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:08.329 [info]  Child Dynamic.EntitySupervisor.Crdt of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.341.0>
Start Call: DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, [sync_interval: 300, max_sync_size: :infinite, shutdown: 30000, crdt: DeltaCrdt.AWLWWMap, on_diffs: #Function<0.83009929/1 in Horde.SupervisorSupervisor.delta_crdt_options/1>, name: Dynamic.EntitySupervisor.Crdt])
Restart: :permanent
Shutdown: 30000
Type: :worker

22:06:08.329 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:08.329 [info]  Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.342.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:08.330 [info]  Child Horde.GracefulShutdownManager of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.345.0>
Start Call: GenServer.start_link(Horde.GracefulShutdownManager, Dynamic.EntitySupervisor.Crdt, [name: Dynamic.EntitySupervisor.GracefulShutdownManager])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:08.330 [info]  Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.346.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:08.330 [info]  Child Horde.SignalShutdown of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.347.0>
Start Call: GenServer.start_link(Horde.SignalShutdown, [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:08.330 [info]  Child Dynamic.EntitySupervisor_telemetry_poller of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.348.0>
Start Call: :telemetry_poller.start_link([measurements: [{:process_info, [name: Dynamic.EntitySupervisor, event: [:horde, :supervisor], keys: [:message_queue_len]]}, {Horde.SupervisorTelemetryPoller, :poll, [Dynamic.EntitySupervisor]}], period: 5000, name: Dynamic.EntitySupervisor_telemetry_poller])
Restart: :permanent
Shutdown: 5000
Type: :worker

  2) test multiple nodes entity data fetch from other nodes (Dynamic.EntityTest)
     test/dynamic_test.exs:56
     ** (CaseClauseError) no case clause matching: []
     stacktrace:
       (dynamic) lib/dynamic.ex:17: Dynamic.get_entity_pid/1
       test/dynamic_test.exs:28: Dynamic.EntityTest.__ex_unit_setup_0/1
       test/dynamic_test.exs:1: Dynamic.EntityTest.__ex_unit__/2

Finished in 5.3 seconds
2 tests, 2 failures

Randomized with seed 925875
mgwidmann commented 5 years ago

Have you had a chance to look at this test? I'm seeing shutdowns occur while my system is running at full load, not sure if its related but it certainly causes a lot of noise when it does since it all I can find is this same error message. Perhaps there is a further root cause I cannot find on why the system is shutting down...

derekkraan commented 5 years ago

I haven't had a chance to look at this yet. Work is busy these days.