search

whatyouhide / xandra

Fast, simple, and robust Cassandra/ScyllaDB driver for Elixir.
ISC License
402 stars 54 forks source link

Xandra crashes when a newly added node sends `:up` message #371

Open harunzengin opened 1 week ago

harunzengin commented 1 week ago

Recently, we added a new node to Xandra which caused some issues. The trigger was this:

** (stop) exited in: :gen_statem.call(#PID<0.152339.0>, :checkout, :infinity)
** (EXIT) an exception was raised:
    ** (FunctionClauseError) no function clause matching in anonymous fn/1 in Xandra.Cluster.Pool.handle_event/4
        (xandra 0.19.0) lib/xandra/cluster/pool.ex:245: anonymous fn(nil) in Xandra.Cluster.Pool.handle_event/4
        (elixir 1.16.3) lib/map.ex:957: Map.get_and_update/3
        (elixir 1.16.3) lib/map.ex:999: Map.get_and_update!/3
        (xandra 0.19.0) lib/xandra/cluster/pool.ex:245: Xandra.Cluster.Pool.handle_event/4
        (stdlib 5.2) gen_statem.erl:1397: :gen_statem.loop_state_callback/11
        (stdlib 5.2) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
(stdlib 5.2) gen.erl:246: :gen.do_call/4
(stdlib 5.2) gen_statem.erl:923: :gen_statem.call/3
(xandra 0.19.0) lib/xandra/cluster.ex:548: Xandra.Cluster.with_conn_and_retrying/3

Looking closer, this is where we get the error: https://github.com/whatyouhide/xandra/blob/main/lib/xandra/cluster/pool.ex#L245

So since we get a HOST_UP event for a machine that we don't know about, Xandra crashes. Adding nodes to the cassandra cluster shouldn't cause crashes though.

whatyouhide commented 6 days ago

@harunzengin yep totally. Wanna work on fixing this? I won't have bandwidth for a while 😢