search

basho / riak

Riak is a decentralized datastore from Basho Technologies.
http://docs.basho.com
Apache License 2.0
3.95k stars 537 forks source link

Riak real-time replication rebalancing (in 2.0.1) #625

Open jonasrichard opened 9 years ago

jonasrichard commented 9 years ago

When real-time replication is enabled source and sink cluster nodes form replication pairs, so one source node pushes replicated operations (PUT, DELETE) to the appropriate sink node. Let us imagine two 5-node clusters, one directional real-time replication in between.

If one node fails in sink cluster (node2_1 - the first node in the second cluster), its pair (node1_1) won't connect to that node but it will be directed to - let us say - node2_2. It is fine, real-time traffic will be handled by a living node in the sink cluster. But when node2_1 returns to a good state, nobody will connect to that node, so it will be sitting idle.

After playing with the nodes in the sink cluster I found that I needed to stop all nodes in the sink cluster in order that source nodes connect to that node2_1 node (since there weren't any other node in the sink cluster). But after staring the sink nodes they sat idle, since source nodes didn't reconnect to sink nodes.

There should be a redistribution and some kind of monitoring which can help rebalance the source-sink pairs, if the connections are made in a way that some nodes are useless (in terms of replication).

jonmeredith commented 9 years ago

Thanks for reporting, we're aware of the issue and have work scheduled against the 2.0.X series to improve.

(Tracked against RIAK-1182 in our internal work tracking system).

jonasrichard commented 9 years ago

Is it possible to apply that fix to 1.4.x product line?

jonmeredith commented 9 years ago

Unfortunately not - it has required refactoring the inter-cluster connection manager and at this time we view the changes as too significant to backport.

You can trigger a rebalance by stop/starting the realtime subsystem (pending writes will be queued as long as they are enabled). Additionally our support team have some workarounds to do more targeted rebalancing that require running some code in the erlang shell. Please open a ticket on ZenDesk at help.basho.com if you need assistance with that.

DSomogyi commented 9 years ago

Fixed by Riak-1127.