Some curious and unexpected cluster plans when running leave operations, prompted a review of the lead code.
This PR corrects a number of issues found:
The first part of a leave plan is to plan to handoff all vnodes on the leaving nodes to other members of the cluster, in a way that does not break the target_n_val. The logic of this attempt_simple_leave function was incorrect, as it would produce plans that would include tail violations (where the location was safe when the ring was viewed as a list, but not when it was considered as a ring where the head wraps round to the tail). This issue is corrected.
The attempt_simple_leave did not attempt to balance the vnodes in any way, leading in some cases to unbalanced plans (and leaves that might not be supportable as a receiving node may not have sufficient space). There is now a preference to make choices that lead to more balanced outcomes - and a workaround to skip this stage and move straight to a freshly balanced cluster plan (riak_core.full_rebalance_onleave).
When a full rebalance was forced, as it was not possible to produce a safe outcome to attempt_simple_transfer, then a full rebalance would be triggered. However a deprecated function was used, one which could lead to tail violations (ignoring the improvements made in Riak 2.2.5). The updated function is now used.
Some curious and unexpected cluster plans when running leave operations, prompted a review of the lead code.
This PR corrects a number of issues found:
The first part of a leave plan is to plan to handoff all vnodes on the leaving nodes to other members of the cluster, in a way that does not break the target_n_val. The logic of this attempt_simple_leave function was incorrect, as it would produce plans that would include tail violations (where the location was safe when the ring was viewed as a list, but not when it was considered as a ring where the head wraps round to the tail). This issue is corrected.
The attempt_simple_leave did not attempt to balance the vnodes in any way, leading in some cases to unbalanced plans (and leaves that might not be supportable as a receiving node may not have sufficient space). There is now a preference to make choices that lead to more balanced outcomes - and a workaround to skip this stage and move straight to a freshly balanced cluster plan (
riak_core.full_rebalance_onleave).When a full rebalance was forced, as it was not possible to produce a safe outcome to
attempt_simple_transfer, then a full rebalance would be triggered. However a deprecated function was used, one which could lead to tail violations (ignoring the improvements made in Riak 2.2.5). The updated function is now used.