Example snippet does not work

[ostinelli]

I'm curious to give this a try but the lack of docs makes it hard to get started with. So I'm trying to use the snippet provided in the readme with the added ETS options from the kvstore example:

(watest@127.0.0.1)1> Spec = wa_raft_sup:child_spec([#{table => watest, partition => 1, nodes => [node()], log_module => wa_raft_log_ets, storage_module => wa_raft_storage_ets}]).
** exception error: no match of right hand side value undefined
     in function  wa_raft_sup:child_spec/1 (/Users/roberto/workspace/watest/_build/default/lib/waraft/src/wa_raft_sup.erl, line 74)

As I'm running this in a console per the example, but apparently the method needs to know the application name, I pass in the app name:

(watest@127.0.0.1)2> Spec = wa_raft_sup:child_spec(watest, [#{table => watest, partition => 1, nodes => [node()]}]).
#{id => wa_raft_sup,restart => permanent,shutdown => infinity,
  start =>
      {wa_raft_sup,start_link,
                   [watest,
                    [#{table => watest,nodes => ['watest@127.0.0.1'],partition => 1}],
                    #{}]},
  type => supervisor,
  modules => [wa_raft_sup]}

Then start the supervisor:

(watest@127.0.0.1)3> supervisor:start_child(kernel_sup, Spec).
{error,{{'EXIT',{{'EXIT',{{no_configured_database_path,watest},
                          [{wa_raft_env,database_path,1,
                                        [{file,"/Users/roberto/workspace/watest/_build/default/lib/waraft/src/wa_raft_env.erl"},
                                         {line,38}]},
                           {wa_raft_part_sup,normalize_spec,2,
                                             [{file,"/Users/roberto/workspace/watest/_build/default/lib/waraft/src/wa_raft_part_sup.erl"},
                                              {line,124}]},
                           {wa_raft_part_sup,start_link,2,
                                             [{file,"/Users/roberto/workspace/watest/_build/default/lib/waraft/src/wa_raft_part_sup.erl"},
                                              {line,69}]},
                           {supervisor,do_start_child_i,3,
                                       [{file,"supervisor.erl"},{line,420}]},
                           {supervisor,handle_call,3,
                                       [{file,"supervisor.erl"},{line,445}]},
                           {gen_server,try_handle_call,4,
                                       [{file,"gen_server.erl"},{line,1113}]},
                           {gen_server,handle_msg,6,
                                       [{file,"gen_server.erl"},{line,1142}]},
                           {proc_lib,init_p_do_apply,3,
                                     [{file,"proc_lib.erl"},{line,241}]}]}},
                 [{wa_raft_sup,'-start_link/3-lc$^0/1-0-',2,
                               [{file,"/Users/roberto/workspace/watest/_build/default/lib/waraft/src/wa_raft_sup.erl"},
                                {line,102}]},
                  {wa_raft_sup,start_link,3,
                               [{file,"/Users/roberto/workspace/watest/_build/default/lib/waraft/src/wa_raft_sup.erl"},
                                {line,104}]},
                  {supervisor,do_start_child_i,3,
                              [{file,"supervisor.erl"},{line,420}]},
                  {supervisor,do_start_child,2,
                              [{file,"supervisor.erl"},{line,406}]},
                  {supervisor,handle_start_child,2,
                              [{file,"supervisor.erl"},{line,712}]},
                  {supervisor,handle_call,3,
                              [{file,"supervisor.erl"},{line,461}]},
                  {gen_server,try_handle_call,4,
                              [{file,"gen_server.erl"},{line,1113}]},
                  {gen_server,handle_msg,6,
                              [{file,"gen_server.erl"},{line,1142}]}]}},
        {child,undefined,wa_raft_sup,
               {wa_raft_sup,start_link,
                            [watest,
                             [#{table => watest,nodes => ['watest@127.0.0.1'],partition => 1}],
                             #{}]},
               permanent,false,infinity,supervisor,
               [wa_raft_sup]}}}

Browsing through code, I set the application env key raft_database to "/Users/roberto/workspace" (even though I am setting the ETS in-memory only options for log and storage) and then I get:

=ERROR REPORT==== 11-Sep-2023::18:03:47.182676 ===
** State machine raft_server_watest_1 terminating
** Last event = {internal,init_state}
** When server state  = {stalled,
                            {raft_state,watest,raft_server_watest_1,
                                {raft_identity,raft_server_watest_1,
                                    'watest@127.0.0.1'},
                                watest,1,"/Users/roberto/workspace/watest.1",
                                {log_view,watest,raft_log_watest_1,wa_raft_log_ets,
                                    0,0,[],undefined,undefined},
                                wa_raft_distribution,raft_storage_watest_1,
                                raft_log_catchup_watest_1,0,0,undefined,
                                undefined,0,undefined,#{},undefined,
                                -576460751864,#{},#{},#{},#{},0,undefined,
                                undefined,false}}
** Reason for termination = error:badarg
** Callback modules = [wa_raft_server]
** Callback mode = [state_functions,state_enter]
** Stacktrace =
**  [{ets,delete,
          [wa_raft_info,{state,watest,1}],
          [{error_info,#{cause => id,module => erl_stdlib_errors}}]},
     {wa_raft_info,delete_state,2,
                   [{file,"/Users/roberto/workspace/watest/_build/default/lib/waraft/src/wa_raft_info.erl"},
                    {line,62}]},
     {wa_raft_server,terminate,3,
                     [{file,"/Users/roberto/workspace/watest/_build/default/lib/waraft/src/wa_raft_server.erl"},
                      {line,1502}]},
     {gen_statem,terminate,7,[{file,"gen_statem.erl"},{line,2553}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]

I can keep this going (and I'll try to) but is there any chance there's a proper working example on how to use waraft?

Thank you, r.

[eKristensen]

I struggle to get started as well. 😥

[eKristensen]

I managed to get the server running using the following commands in the shell:

wa_raft_info:init_tables().
wa_raft_log_catchup:init_tables().
application:set_env(watest, raft_database, '/tmp'). 
Spec = wa_raft_sup:child_spec(watest, [#{table => watest, partition => 1, nodes => [node()]}]).  supervisor:start_child(kernel_sup, Spec).
wa_raft_server:status(raft_server_watest_1).

There are a few initialization steps that are missing in the getting started tutorial.

My console for reference

$ rebar3 shell --sname yo
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling wa_raft
Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

Eshell V14.1.1 (press Ctrl+G to abort, type help(). for help)
(yo@fedora)1> wa_raft_info:init_tables().
ok
(yo@fedora)2> wa_raft_log_catchup:init_tables().
wa_raft_log_catchup
(yo@fedora)3> application:set_env(watest, raft_database, '/tmp'). 
ok
(yo@fedora)4> 
              Spec = wa_raft_sup:child_spec(watest, [#{table => watest, partition => 1, nodes => [node()]}]).  supervisor:start_child(kernel_sup, Spec).
#{id => wa_raft_sup,restart => permanent,shutdown => infinity,
  start =>
      {wa_raft_sup,start_link,
                   [watest,
                    [#{table => watest,nodes => [yo@fedora],partition => 1}],
                    #{}]},
  type => supervisor,
  modules => [wa_raft_sup]}
(yo@fedora)5>  supervisor:start_child(kernel_sup, Spec).
{ok,<0.219.0>}
(yo@fedora)6> wa_raft_server:status(raft_server_watest_1).
[{state,stalled},
 {id,yo@fedora},
 {table,watest},
 {partition,1},
 {data_dir,"/tmp/watest.1"},
 {current_term,0},
 {voted_for,undefined},
 {commit_index,0},
 {last_applied,0},
 {leader_id,undefined},
 {next_index,#{}},
 {match_index,#{}},
 {log_module,wa_raft_log_ets},
 {log_first,0},
 {log_last,0},
 {votes,#{}},
 {inflight_applies,0},
 {disable_reason,undefined},
 {config,#{version => 1}},
 {config_index,0},
 {witness,false}]

UPDATE: Promotion works just fine but I'm still missing something. I cannot read written values.

[eKristensen]

UPDATE: Solved. I was a bit too quick in writing here. I've updated the example and included that update in a pull request: #6 Hopefully this will help others get a slightly better start with waraft.

It is a "feature", the API has changed so the read command is: wa_raft_acceptor:read(raft_acceptor_watest_1, {read, test, key}).

---

When I call to read like it says on the get started page; wa_raft_acceptor:commit(raft_acceptor_watest_1, {make_ref(), {read, test, key}}).

The chain of function calls ends up calling wa_raft_storage_ets:storage_apply(...) (which has no function clause match for {read,...}) instead of what I think it should be calling: wa_raft_storage_ets:storage_read(...).

Is this a bug? Should the read command work?

Is this a feature? Has the API for reading from the cluster been changed?

ERROR REPORT:

(yo@fedora)11> wa_raft_acceptor:commit(raft_acceptor_watest_1, {make_ref(), {read, test, key}}).
=ERROR REPORT==== 23-Nov-2023::12:32:18.045863 ===
** Generic server raft_storage_watest_1 terminating 
** Last message in was {'$gen_cast',
                           {apply,
                               {3,
                                {1,
                                 {#Ref<0.2161551319.62652417.212751>,
                                  {read,test,key}}}},
                               1}}
** When Server state == {state,raft_storage_watest_1,watest,1,"/tmp/watest.1",
                               wa_raft_storage_ets,
                               {state,raft_storage_watest_1,watest,1,
                                      {raft_log_pos,2,1}},
                               {raft_log_pos,2,1}}
** Reason for termination ==
** {function_clause,
       [{wa_raft_storage_ets,storage_apply,
            [{read,test,key},
             {raft_log_pos,3,1},
             {state,raft_storage_watest_1,watest,1,{raft_log_pos,2,1}}],
            [{file,"/home/ek/code/waraft/src/wa_raft_storage_ets.erl"},
             {line,56}]},
        {wa_raft_storage,execute,3,
            [{file,"/home/ek/code/waraft/src/wa_raft_storage.erl"},
             {line,550}]},
        {wa_raft_storage,apply_impl,3,
            [{file,"/home/ek/code/waraft/src/wa_raft_storage.erl"},
             {line,518}]},
        {wa_raft_storage,handle_cast,2,
            [{file,"/home/ek/code/waraft/src/wa_raft_storage.erl"},
             {line,478}]},
        {gen_server,try_handle_cast,3,[{file,"gen_server.erl"},{line,1103}]},
        {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1165}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}

=CRASH REPORT==== 23-Nov-2023::12:32:18.046083 ===
  crasher:
    initial call: wa_raft_storage:init/1
    pid: <0.222.0>
    registered_name: raft_storage_watest_1
    exception error: no function clause matching 
                     wa_raft_storage_ets:storage_apply({read,test,key},
                                                       {raft_log_pos,3,1},
                                                       {state,
                                                        raft_storage_watest_1,
                                                        watest,1,
                                                        {raft_log_pos,2,1}}) (/home/ek/code/waraft/src/wa_raft_storage_ets.erl, line 56)
      in function  wa_raft_storage:execute/3 (/home/ek/code/waraft/src/wa_raft_storage.erl, line 550)
      in call from wa_raft_storage:apply_impl/3 (/home/ek/code/waraft/src/wa_raft_storage.erl, line 518)
      in call from wa_raft_storage:handle_cast/2 (/home/ek/code/waraft/src/wa_raft_storage.erl, line 478)
      in call from gen_server:try_handle_cast/3 (gen_server.erl, line 1103)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1165)
    ancestors: [raft_sup_watest_1,raft_sup_watest,kernel_sup,<0.47.0>]
    message_queue_len: 0
    messages: []
    links: [<0.220.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 28
    reductions: 12281
  neighbours:

{error,{call_error,shutdown}}
=SUPERVISOR REPORT==== 23-Nov-2023::12:32:18.047463 ===
    supervisor: {local,raft_sup_watest_1}
    errorContext: child_terminated
    reason: {function_clause,
                [{wa_raft_storage_ets,storage_apply,
                     [{read,test,key},
                      {raft_log_pos,3,1},
                      {state,raft_storage_watest_1,watest,1,
                          {raft_log_pos,2,1}}],
                     [{file,
                          "/home/ek/code/waraft/src/wa_raft_storage_ets.erl"},
                      {line,56}]},
                 {wa_raft_storage,execute,3,
                     [{file,"/home/ek/code/waraft/src/wa_raft_storage.erl"},
                      {line,550}]},
                 {wa_raft_storage,apply_impl,3,
                     [{file,"/home/ek/code/waraft/src/wa_raft_storage.erl"},
                      {line,518}]},
                 {wa_raft_storage,handle_cast,2,
                     [{file,"/home/ek/code/waraft/src/wa_raft_storage.erl"},
                      {line,478}]},
                 {gen_server,try_handle_cast,3,
                     [{file,"gen_server.erl"},{line,1103}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1165}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.222.0>},
               {id,wa_raft_storage},
               {mfargs,
                   {wa_raft_storage,start_link,
                       [{raft_options,watest,watest,1,false,
                            {raft_identity,raft_server_watest_1,yo@fedora},
                            "/tmp/watest.1",raft_acceptor_watest_1,
                            wa_raft_distribution,raft_log_watest_1,
                            wa_raft_log_ets,raft_log_catchup_watest_1,
                            raft_queue_watest_1,
                            {atomics,#Ref<0.2161551319.62783496.212820>},
                            raft_commit_queue_watest_1,
                            raft_read_queue_watest_1,raft_server_watest_1,
                            raft_storage_watest_1,wa_raft_storage_ets,
                            raft_sup_watest_1,raft_transport_cleanup_watest_1,
                            "/tmp/watest.1/transport",
                            wa_raft_dist_transport}]}},
               {restart_type,transient},
               {significant,false},
               {shutdown,30000},
               {child_type,worker}]