skirino / raft_kv

An Elixir library to store key-value pairs in a distributed, fault-tolerant, self-adjusting data structure
MIT License
18 stars 1 forks source link

Crash, if some short-lived nodes connects to a cluster. #5

Open liveforeverx opened 6 years ago

liveforeverx commented 6 years ago

Crash, if some short-lived nodes connects to a cluster.

2018-08-30 13:38:01.847 [error] Child RaftKV.WorkflowExecutor of Supervisor #PID<0.2539.0> (Supervisor.Default) terminated
** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: {["app@ip": :ok], [:"remote_shell@ip"]}
        (raft_kv) lib/raft_kv/ets_record_manager.ex:79: RaftKV.EtsRecordManager.ensure_record_changed_in_all_relevant_nodes!/1
        (raft_kv) lib/raft_kv/workflow.ex:301: RaftKV.Workflow.execute/1
        (raft_kv) lib/raft_kv/workflow_executor.ex:22: RaftKV.WorkflowExecutor.handle_info/2
        (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Do we really need to send update to all non raft relevant nodes?

liveforeverx commented 6 years ago

Moved from https://github.com/skirino/raft_fleet/issues/6

skirino commented 6 years ago

The “relevant nodes” here include nodes that only work as clients (i.e. nodes that don’t have replicated key-value pairs but do send queries/commands). I mean, to send queries/commands to correct nodes, updates in ETS records are necessary. Could you add your remote shell node as “hidden” one?

(Now I can’t investigate more as I’m on my holidays)

liveforeverx commented 6 years ago

It should be fixed in distillery first for all node types: https://github.com/bitwalker/swarm/blob/master/config/config.exs#L7-L16 (how swarm handles it, because of this behaviour)

skirino commented 6 years ago

I'm not familiar with distillery but looking at https://github.com/bitwalker/distillery/blob/master/priv/libexec/commands/remote_console.sh it seems that remote_console nodes start as hidden ones. How do you start your remote shell?

liveforeverx commented 6 years ago

@skirino Via remote_console command...