Crash during riak_test - Githubissues

martinsumner commented 2 years ago

08:38:18.443 [info] <0.226.0> Finding tombstones from node 'dev1@127.0.0.1'.
08:38:18.870 [info] <0.226.0> Reaping 10000 keys from node 'dev1@127.0.0.1'.
08:38:20.011 [warning] <0.9.0> nextgenrepl_deletemodes failed: {badarg,[{ets,lookup_element,[riak_kv_get_fsm_sj_2,full,2],[]},{sidejob,is_available,3,[{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/sidejob/src/sidejob.erl"},{line,146}]},{sidejob,available,1,[{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/sidejob/src/sidejob.erl"},{line,125}]},{sidejob,call,3,[{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/sidejob/src/sidejob.erl"},{line,68}]},{sidejob_supervisor,start_child,4,[{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/sidejob/src/sidejob_supervisor.erl"},{line,53}]},{riak_kv_get_fsm,start,4,[{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/riak_kv/src/riak_kv_get_fsm.erl"},{line,123}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
08:38:20.011 [error] <0.9.0> 
================ nextgenrepl_deletemodes failure stack trace =====================
{badarg,[{ets,lookup_element,[riak_kv_get_fsm_sj_2,full,2],[]},
         {sidejob,is_available,3,
                  [{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/sidejob/src/sidejob.erl"},
                   {line,146}]},
         {sidejob,available,1,
                  [{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/sidejob/src/sidejob.erl"},
                   {line,125}]},
         {sidejob,call,3,
                  [{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/sidejob/src/sidejob.erl"},
                   {line,68}]},
         {sidejob_supervisor,start_child,4,
                             [{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/sidejob/src/sidejob_supervisor.erl"},
                              {line,53}]},
         {riak_kv_get_fsm,start,4,
                          [{file,"/Users/martinsumner/dbroot/basho/riak/_build/default/lib/riak_kv/src/riak_kv_get_fsm.erl"},
                           {line,123}]},
         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
==================================================================================

Presumably the badarg is that the table is not defined?

The riak_test was being run as part of a group of tests - so there was no halt on failure to examine logs further. This part of test is a significant time after startup (so presumably this is not a race at startup).

The failure occurred after a refactoring of the reaper process - perhaps this is now reaping at a faster rate?

martinsumner commented 2 years ago

Testing of https://github.com/basho/riak_kv/pull/1809 - baselined change from riak 3.0.9

martinsumner commented 2 years ago

ets table brought down by a crashing node - issue not related to sidejob

basho / sidejob

Crash during riak_test #21