Can't bootstrap Hibari cluster in multi admin node config

[tatsuya6502]

• Hibari dev branch (pre 0.1.11)
• total 6 nodes (3 admin nodes and 6 brick nodes)

In a multi admin node configuration, bootstrapping will fail due to quorum_error in brick_admin:add_table/4 call. QuickCheck squorum test is also failing with the same error (https://github.com/hibari/gdss-admin/issues/4).

3 admin nodes

[clus@hibaribench ~]$ grep ADMIN hibari.config
ADMIN_NODES=(hibari1 hibari2 hibari3)
[clus@hibaribench ~]$ ./clus/priv/clus-hibari.sh -f bootstrap hibari hibari.config
RPC(bootstrap) to hibari@hibari1 failed: {'EXIT',
                                          {{{badmatch,quorum_error},
                                            [{brick_admin,
                                              bootstrap_existing_schema2,1,
                                              [{file,"src/brick_admin.erl"},
                                               {line,999}]},
                                             {brick_admin,handle_info,2,
                                              [{file,"src/brick_admin.erl"},
                                               {line,850}]},
                                             {gen_server,try_dispatch,4,
                                              [{file,"gen_server.erl"},
                                               {line,593}]},
                                             {gen_server,handle_msg,5,
                                              [{file,"gen_server.erl"},
                                               {line,659}]},
                                             {proc_lib,init_p_do_apply,3,
                                              [{file,"proc_lib.erl"},
                                               {line,237}]}]},
                                           {gen_server,call,
                                            [brick_admin,
                                             {add_table,tab1,
                                              [{tab1_ch1,
                                                [{tab1_ch1_b1,hibari@hibari1},
                                                 {tab1_ch1_b2,hibari@hibari2},
                                                 {tab1_ch1_b3,
                                                  hibari@hibari3}]},
                                               {tab1_ch2,
                                                [{tab1_ch2_b1,hibari@hibari2},
                                                 {tab1_ch2_b2,hibari@hibari3},
                                                 {tab1_ch2_b3,
                                                  hibari@hibari4}]},
                                               {tab1_ch3,
                                                [{tab1_ch3_b1,hibari@hibari3},
                                                 {tab1_ch3_b2,hibari@hibari4},
                                                 {tab1_ch3_b3,
                                                  hibari@hibari5}]},
                                               {tab1_ch4,
                                                [{tab1_ch4_b1,hibari@hibari4},
                                                 {tab1_ch4_b2,hibari@hibari5},
                                                 {tab1_ch4_b3,
                                                  hibari@hibari6}]},
                                               {tab1_ch5,
                                                [{tab1_ch5_b1,hibari@hibari5},
                                                 {tab1_ch5_b2,hibari@hibari6},
                                                 {tab1_ch5_b3,
                                                  hibari@hibari1}]},
                                               {tab1_ch6,
                                                [{tab1_ch6_b1,hibari@hibari6},
                                                 {tab1_ch6_b2,hibari@hibari1},
                                                 {tab1_ch6_b3,
                                                  hibari@hibari2}]}],
                                              [{hash_init,
                                                #Fun<brick_hash.chash_init.3>},
                                               {old_float_map,[]},
                                               {new_chainweights,
                                                [{tab1_ch1,100},
                                                 {tab1_ch2,100},
                                                 {tab1_ch3,100},
                                                 {tab1_ch4,100},
                                                 {tab1_ch5,100},
                                                 {tab1_ch6,100}]},
                                               {hash_init,
                                                #Fun<brick_hash.chash_init.3>},
                                               {prefix_method,var_prefix},
                                               {prefix_separator,'$/'},
                                               {num_separators,3},
                                               {bigdata_dir,"cwd"},
                                               {do_logging,true},
                                               {do_sync,true},
                                               {created_date,{2015,3,22}},
                                               {created_time,{13,56,4}}]},
                                             300000]}}}
node hibari1 bootstrap failed

run ./clus/priv/clus-hibari.sh without any arguments for usage

1 admin nodes

[clus@hibaribench ~]$ grep ADMIN hibari.config
ADMIN_NODES=(hibari1)
[clus@hibaribench ~]$ ./clus/priv/clus-hibari.sh -f bootstrap hibari hibari.config
ok
hibari@hibari1 => hibari@hibari1 hibari@hibari2 hibari@hibari3 hibari@hibari4 hibari@hibari5 hibari@hibari6

[tatsuya6502]

brick_squorum module needs to be update to avoid server-side timestamp. This will be done by a part of https://github.com/hibari/gdss-client/issues/2.

[tatsuya6502]

I resolved this issue via https://github.com/hibari/gdss-client/issues/2. It no longer fails to bootstrap 3-node admin server. Closing this issue.

[clus@hibaribench ~]$ grep ADMIN hibari.config 
ADMIN_NODES=(hibari1 hibari2 hibari3)
[clus@hibaribench ~]$ ./clus/priv/clus-hibari.sh bootstrap hibari hibari.config
ok
hibari@hibari1 => hibari@hibari1 hibari@hibari2 hibari@hibari3 hibari@hibari4 hibari@hibari5 hibari@hibari6
[clus@hibaribench ~]$ ./clus/priv/clus-hibari.sh ping hibari hibari.config