search

basho / riak_core

Distributed systems infrastructure used by Riak.
Apache License 2.0
1.23k stars 391 forks source link

Mailbox can grow very large when starting many nodes simultaneously with custom bucket properties #763

Open drewkerrigan opened 9 years ago

drewkerrigan commented 9 years ago

Cross-post from: https://github.com/basho/yokozuna/issues/389

The specific riak_core_broadcast ordsets:add_element/2 queue length problem occurs when setting a custom bucket property (allow_mult=false in this case) multiple times on every node in a large (32 nodes in this case) cluster. The result seems to be a race condition that ends up multiplying the actual change to the bucket metadata. While not related to Yokozuna, this is still a problem that is not difficult to trigger in large clusters.

Here is a dump of erlang:processes(Pid, messages) on one of the problem pids:

{messages,[{'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',15009},
                                    {'riak@a2.cluster',15507},
                                    {'riak@a3.cluster',15480},
                                    {'riak@a4.cluster',15126},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16416},
                                    {'riak@b3.cluster',17045},
                                    {'riak@b4.cluster',16938},
                                    {'riak@c1.cluster',16522},
                                    {'riak@c2.cluster',16058},
                                    {'riak@c3.cluster',16585},
                                    {'riak@c4.cluster',16446},
                                    {'riak@d1.cluster',15964},
                                    {'riak@d2.cluster',14990},
                                    {'riak@d3.cluster',16291},
                                    {'riak@d4.cluster',15585},
                                    {'riak@e1.cluster',14951},
                                    {'riak@e2.cluster',9998},
                                    {'riak@e3.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',15009,[]},
                                              {'riak@a2.cluster',15507,[]},
                                              {'riak@a3.cluster',15480,[]},
                                              {'riak@a4.cluster',15126,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16416,[]},
                                              {'riak@b3.cluster',17045,[]},
                                              {'riak@b4.cluster',16938,[]},
                                              {'riak@c1.cluster',16522,[]},
                                              {'riak@c2.cluster',16058,[]},
                                              {'riak@c3.cluster',16585,[]},
                                              {'riak@c4.cluster',16446,[]},
                                              {'riak@d1.cluster',15964,[]},
                                              {'riak@d2.cluster',14990,[{...}|...]},
                                              {'riak@d3.cluster',16291,[]},
                                              {'riak@d4.cluster',15585,...},
                                              {'riak@e1.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager,1,'riak@d2.cluster',
                                  'riak@f4.cluster'}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',16361},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',15497},
                                    {'riak@a4.cluster',15138},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16431},
                                    {'riak@b3.cluster',17372},
                                    {'riak@b4.cluster',16982},
                                    {'riak@c1.cluster',16523},
                                    {'riak@c2.cluster',16059},
                                    {'riak@c3.cluster',17353},
                                    {'riak@c4.cluster',16446},
                                    {'riak@d1.cluster',15964},
                                    {'riak@d2.cluster',14961},
                                    {'riak@d3.cluster',16303},
                                    {'riak@d4.cluster',15602},
                                    {'riak@e1.cluster',14951},
                                    {'riak@e2.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',16361,
                                                                          [{[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<"mytype">>},
                                                                              {rw,quorum},
                                                                              {r,quorum},
                                                                              {pw,0},
                                                                              {precommit,[]},
                                                                              {pr,0},
                                                                              {postcommit,...},
                                                                              {...}|...],
                                                                            {1433,794527,519128}},
                                                                            {[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<"mytype">>},
                                                                              {rw,quorum},
                                                                              {r,quorum},
                                                                              {pw,0},
                                                                              {precommit,[]},
                                                                              {pr,...},
                                                                              {...}|...],
                                                                            {1433,794527,485424}}]},
                                              {'riak@a2.cluster',15521,[]},
                                              {'riak@a3.cluster',15497,[]},
                                              {'riak@a4.cluster',15138,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16431,[]},
                                              {'riak@b3.cluster',17372,[]},
                                              {'riak@b4.cluster',16982,[]},
                                              {'riak@c1.cluster',16523,[]},
                                              {'riak@c2.cluster',16059,[]},
                                              {'riak@c3.cluster',17353,[]},
                                              {'riak@c4.cluster',16446,[]},
                                              {'riak@d1.cluster',15964,[]},
                                              {'riak@d2.cluster',14961,[]},
                                              {'riak@d3.cluster',16303,...},
                                              {'riak@d4.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',15009},
                                    {'riak@a2.cluster',15507},
                                    {'riak@a3.cluster',15480},
                                    {'riak@a4.cluster',15126},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16416},
                                    {'riak@b3.cluster',17045},
                                    {'riak@b4.cluster',16938},
                                    {'riak@c1.cluster',16522},
                                    {'riak@c2.cluster',16058},
                                    {'riak@c3.cluster',16585},
                                    {'riak@c4.cluster',16446},
                                    {'riak@d1.cluster',15964},
                                    {'riak@d2.cluster',14991},
                                    {'riak@d3.cluster',16291},
                                    {'riak@d4.cluster',15585},
                                    {'riak@e1.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',15009,[]},
                                              {'riak@a2.cluster',15507,[]},
                                              {'riak@a3.cluster',15480,[]},
                                              {'riak@a4.cluster',15126,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16416,[]},
                                              {'riak@b3.cluster',17045,[]},
                                              {'riak@b4.cluster',16938,[]},
                                              {'riak@c1.cluster',16522,[]},
                                              {'riak@c2.cluster',16058,[]},
                                              {'riak@c3.cluster',16585,[]},
                                              {'riak@c4.cluster',16446,[]},
                                              {'riak@d1.cluster',15964,[]},
                                              {'riak@d2.cluster',14991,...},
                                              {'riak@d3.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager,1,'riak@d2.cluster',
                                  'riak@f4.cluster'}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',16362},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',15497},
                                    {'riak@a4.cluster',15138},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16431},
                                    {'riak@b3.cluster',17372},
                                    {'riak@b4.cluster',16982},
                                    {'riak@c1.cluster',16523},
                                    {'riak@c2.cluster',16059},
                                    {'riak@c3.cluster',17353},
                                    {'riak@c4.cluster',16446},
                                    {'riak@d1.cluster',15964},
                                    {'riak@d2.cluster',14961},
                                    {'riak@d3.cluster',16303},
                                    {'riak@d4.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',16362,
                                                                          [{[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<"mytype">>},
                                                                              {rw,quorum},
                                                                              {r,quorum},
                                                                              {pw,0},
                                                                              {precommit,...},
                                                                              {...}|...],
                                                                            {1433,794527,536013}},
                                                                            {[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<"mytype">>},
                                                                              {rw,quorum},
                                                                              {r,quorum},
                                                                              {pw,...},
                                                                              {...}|...],
                                                                            {1433,794527,519128}},
                                                                            {[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<"mytype">>},
                                                                              {rw,quorum},
                                                                              {r,...},
                                                                              {...}|...],
                                                                            {1433,794527,485424}}]},
                                              {'riak@a2.cluster',15521,[]},
                                              {'riak@a3.cluster',15497,[]},
                                              {'riak@a4.cluster',15138,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16431,[]},
                                              {'riak@b3.cluster',17372,[]},
                                              {'riak@b4.cluster',16982,[]},
                                              {'riak@c1.cluster',16523,[]},
                                              {'riak@c2.cluster',16059,[]},
                                              {'riak@c3.cluster',17353,[]},
                                              {'riak@c4.cluster',16446,[]},
                                              {'riak@d1.cluster',15964,...},
                                              {'riak@d2.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',16363},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',15497},
                                    {'riak@a4.cluster',15138},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16431},
                                    {'riak@b3.cluster',17372},
                                    {'riak@b4.cluster',16982},
                                    {'riak@c1.cluster',16523},
                                    {'riak@c2.cluster',16059},
                                    {'riak@c3.cluster',17353},
                                    {'riak@c4.cluster',16446},
                                    {'riak@d1.cluster',15964},
                                    {'riak@d2.cluster',14961},
                                    {'riak@d3.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',16363,
                                                                          [{[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<"mytype">>},
                                                                              {rw,quorum},
                                                                              {r,quorum},
                                                                              {pw,...},
                                                                              {...}|...],
                                                                            {1433,794527,552944}},
                                                                            {[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<"mytype">>},
                                                                              {rw,quorum},
                                                                              {r,...},
                                                                              {...}|...],
                                                                            {1433,794527,536013}},
                                                                            {[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<...>>},
                                                                              {rw,...},
                                                                              {...}|...],
                                                                            {1433,794527,519128}}]},
                                              {'riak@a2.cluster',15521,[]},
                                              {'riak@a3.cluster',15497,[]},
                                              {'riak@a4.cluster',15138,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16431,[]},
                                              {'riak@b3.cluster',17372,[]},
                                              {'riak@b4.cluster',16982,[]},
                                              {'riak@c1.cluster',16523,[]},
                                              {'riak@c2.cluster',16059,[]},
                                              {'riak@c3.cluster',17353,[]},
                                              {'riak@c4.cluster',16446,...},
                                              {'riak@d1.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',15009},
                                    {'riak@a2.cluster',15507},
                                    {'riak@a3.cluster',15480},
                                    {'riak@a4.cluster',15126},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16416},
                                    {'riak@b3.cluster',17045},
                                    {'riak@b4.cluster',16938},
                                    {'riak@c1.cluster',16522},
                                    {'riak@c2.cluster',16058},
                                    {'riak@c3.cluster',16585},
                                    {'riak@c4.cluster',16446},
                                    {'riak@d1.cluster',15964},
                                    {'riak@d2.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',15009,[]},
                                              {'riak@a2.cluster',15507,[]},
                                              {'riak@a3.cluster',15480,[]},
                                              {'riak@a4.cluster',15126,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16416,[]},
                                              {'riak@b3.cluster',17045,[]},
                                              {'riak@b4.cluster',16938,[]},
                                              {'riak@c1.cluster',16522,[]},
                                              {'riak@c2.cluster',16058,[]},
                                              {'riak@c3.cluster',16585,...},
                                              {'riak@c4.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager,1,'riak@d2.cluster',
                                  'riak@f4.cluster'}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',16364},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',15497},
                                    {'riak@a4.cluster',15138},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16431},
                                    {'riak@b3.cluster',17372},
                                    {'riak@b4.cluster',16982},
                                    {'riak@c1.cluster',16523},
                                    {'riak@c2.cluster',16059},
                                    {'riak@c3.cluster',17353},
                                    {'riak@c4.cluster',16446},
                                    {'riak@d1.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',16364,
                                                                          [{[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,<<...>>},
                                                                              {rw,...},
                                                                              {...}|...],
                                                                            {1433,794527,569788}},
                                                                            {[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,...},
                                                                              {...}|...],
                                                                            {1433,794527,552944}},
                                                                            {[{young_vclock,20},{w,quorum},{small_vclock,...},{...}|...],
                                                                            {1433,794527,536013}},
                                                                            {[{young_vclock,20},{w,...},{...}|...],{1433,794527,...}}]},
                                              {'riak@a2.cluster',15521,[]},
                                              {'riak@a3.cluster',15497,[]},
                                              {'riak@a4.cluster',15138,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16431,[]},
                                              {'riak@b3.cluster',17372,[]},
                                              {'riak@b4.cluster',16982,[]},
                                              {'riak@c1.cluster',16523,[]},
                                              {'riak@c2.cluster',16059,...},
                                              {'riak@c3.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',16365},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',15497},
                                    {'riak@a4.cluster',15138},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16431},
                                    {'riak@b3.cluster',17372},
                                    {'riak@b4.cluster',16982},
                                    {'riak@c1.cluster',16523},
                                    {'riak@c2.cluster',16059},
                                    {'riak@c3.cluster',17353},
                                    {'riak@c4.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',16365,
                                                                          [{[{young_vclock,20},
                                                                              {w,quorum},
                                                                              {small_vclock,50},
                                                                              {search_index,...},
                                                                              {...}|...],
                                                                            {1433,794527,586608}},
                                                                            {[{young_vclock,20},{w,quorum},{small_vclock,...},{...}|...],
                                                                            {1433,794527,569788}},
                                                                            {[{young_vclock,20},{w,...},{...}|...],{1433,794527,...}},
                                                                            {[{young_vclock,...},{...}|...],{1433,...}},
                                                                            {[{...}|...],{...}}]},
                                              {'riak@a2.cluster',15521,[]},
                                              {'riak@a3.cluster',15497,[]},
                                              {'riak@a4.cluster',15138,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16431,[]},
                                              {'riak@b3.cluster',17372,[]},
                                              {'riak@b4.cluster',16982,[]},
                                              {'riak@c1.cluster',16523,...},
                                              {'riak@c2.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',15009},
                                    {'riak@a2.cluster',15507},
                                    {'riak@a3.cluster',15480},
                                    {'riak@a4.cluster',15126},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16416},
                                    {'riak@b3.cluster',17045},
                                    {'riak@b4.cluster',16938},
                                    {'riak@c1.cluster',16522},
                                    {'riak@c2.cluster',16058},
                                    {'riak@c3.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',15009,[]},
                                              {'riak@a2.cluster',15507,[]},
                                              {'riak@a3.cluster',15480,[]},
                                              {'riak@a4.cluster',15126,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16416,[]},
                                              {'riak@b3.cluster',17045,[]},
                                              {'riak@b4.cluster',16938,...},
                                              {'riak@c1.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager,1,'riak@d2.cluster',
                                  'riak@f4.cluster'}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',15009},
                                    {'riak@a2.cluster',15507},
                                    {'riak@a3.cluster',15480},
                                    {'riak@a4.cluster',15126},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16417},
                                    {'riak@b3.cluster',17045},
                                    {'riak@b4.cluster',16939},
                                    {'riak@c1.cluster',16522},
                                    {'riak@c2.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',15009,[]},
                                              {'riak@a2.cluster',15507,[]},
                                              {'riak@a3.cluster',15480,[]},
                                              {'riak@a4.cluster',15126,[]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16417,[]},
                                              {'riak@b3.cluster',17045,...},
                                              {'riak@b4.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager,0,'riak@f4.cluster',
                                  'riak@f4.cluster'}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',15010},
                                    {'riak@a2.cluster',15507},
                                    {'riak@a3.cluster',15480},
                                    {'riak@a4.cluster',15171},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16417},
                                    {'riak@b3.cluster',17045},
                                    {'riak@b4.cluster',16939},
                                    {'riak@c1.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',15010,[]},
                                              {'riak@a2.cluster',15507,[]},
                                              {'riak@a3.cluster',15480,[]},
                                              {'riak@a4.cluster',15171,[{...}|...]},
                                              {'riak@b1.cluster',15792,[]},
                                              {'riak@b2.cluster',16417,...},
                                              {'riak@b3.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager,1,'riak@a4.cluster',
                                  'riak@d4.cluster'}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',16366},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',15497},
                                    {'riak@a4.cluster',15138},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16431},
                                    {'riak@b3.cluster',17372},
                                    {'riak@b4.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',16366,
                                                                          [{[{...}|...],{...}},{[...],...},{...}|...]},
                                              {'riak@a2.cluster',15521,[]},
                                              {'riak@a3.cluster',15497,[]},
                                              {'riak@a4.cluster',15138,[]},
                                              {'riak@b1.cluster',15792,...},
                                              {'riak@b2.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',15010},
                                    {'riak@a2.cluster',15507},
                                    {'riak@a3.cluster',15480},
                                    {'riak@a4.cluster',15126},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',16446},
                                    {'riak@b3.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',15010,[]},
                                              {'riak@a2.cluster',15507,[]},
                                              {'riak@a3.cluster',15480,[]},
                                              {'riak@a4.cluster',15126,...},
                                              {'riak@b1.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager,1,'riak@b2.cluster',
                                  'riak@f4.cluster'}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',16367},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',15497},
                                    {'riak@a4.cluster',15138},
                                    {'riak@b1.cluster',15792},
                                    {'riak@b2.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',16367,[{...}|...]},
                                              {'riak@a2.cluster',15521,[]},
                                              {'riak@a3.cluster',15497,...},
                                              {'riak@a4.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',16368},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',15497},
                                    {'riak@a4.cluster',15138},
                                    {'riak@b1.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',16368,[...]},
                                              {'riak@a2.cluster',15521,...},
                                              {'riak@a3.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_test3">>}},
                                    [{'riak@a1.cluster',15010},
                                    {'riak@a2.cluster',15507},
                                    {'riak@a3.cluster',15480},
                                    {'riak@a4.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',15010,...},
                                              {'riak@a2.cluster',...},
                                              {...}|...],
                                              []}},
                                  riak_core_metadata_manager,1,'riak@b2.cluster',
                                  'riak@f4.cluster'}},
          {'$gen_cast',{broadcast,{{{core,buckets},
                                    {<<"mytype">>,<<"bb_t"...>>}},
                                    [{'riak@a1.cluster',16369},
                                    {'riak@a2.cluster',15521},
                                    {'riak@a3.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{'riak@a1.cluster',...},{...}|...],[]}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},{<<"mytype">>,<<...>>}},
                                    [{'riak@a1.cluster',16370},
                                    {'riak@a2.cluster',...},
                                    {...}|...]},
                                  {metadata,{[{...}|...],[]}},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{{core,buckets},{<<...>>,...}},
                                    [{'riak@a1.cluster',...},{...}|...]},
                                  {metadata,{[...],...}},
                                  riak_core_metadata_manager,0,'riak@h4.cluster',...}},
          {'$gen_cast',{broadcast,{{{core,...},{...}},[{...}|...]},
                                  {metadata,{...}},
                                  riak_core_metadata_manager,1,...}},
          {'$gen_cast',{broadcast,{{{...},...},[...]},
                                  {metadata,...},
                                  riak_core_metadata_manager}},
          {'$gen_cast',{broadcast,{{...},...},{...},...}},
          {'$gen_cast',{broadcast,{...},...}},
          {'$gen_cast',{broadcast,...}},
          {'$gen_cast',{...}},
          {'$gen_cast',...},
          {...}|...]}
drewkerrigan commented 9 years ago

cc: @wbrown

llelf commented 5 years ago

@drewkerrigan any more info on how to trigger it?

I’d really love to get a riak_test case for this. Current status: already getting ridiculous¹ — doesn’t help, riak is stable. Next thing to try is probably adding network delays.

¹) https://github.com/llelf/riak_test/blob/RCB/tests/rcb.erl#L47