Return status of set_priority. Ability to broadcast priority changes if there's a leader

[myrrc]

set_priority is a bit inconsistent with add_srv and remove_srv. If you invoke the latter two on a follower, the request will be forwarded to leader. You can wait on the request and check whether it was accepted or not. Unfortunately, there are no such guarantees for set_priority. If you invoke it on a follower, and there is a live leader, function would quit. I want to know the result of set_priority, so changes are proposed that change its return type.

[myrrc]

Hi @greensky00.

This PR is needed for my other PR regarding a ZooKeeper-like program. Each ZooKeeper-like replica has a NuRaft server inside, and it can be sent ZooKeeper-compatible commands. When a command gets to the replica, replica must process it or abort. For add/remove_srv we can check whether result was accepted. For set_priority we don't have this guarantee.

What should we do if we don't want to lose the priority update message? We can't rely on leadership and only process message at the current leader as:

• there is a chance there will be elections after returning from is_leader() and before appending new config to log in set_priority in which case we'll lose the update.
• there is a counter-example on {A, B, C} cluster where command going to A sees B as leader, then there are re-elections and command on B sees C as leader, re-elections again, and then command on C sees A as leader. In this case changes would also be lost.

Nor can we simply change our config via get_server(id)->set_priority as NuRaft has various invariants inside like uncommitted_config_ that would be broken. So, we need to set_priority on each replica explicitly. However, if there's a live leader, unlike add/remove_srv, the new config won't be forwarded (I tried implementing this but failed as it seems quite a big task) and we'll lose the update.

Thus, a new option is introduced that would send messages to replicas telling to update priority even when a leader is present. In my case there are no chances of going out of sync (as update will be propagated to every replica) but I agree it can happen in general. Please note, however, that invoking set_priority when there's no live leader currently can also lead to inconsistencies as a peer can reject the priority update message (is_bisy() case).

I will address formatting issues and documentation for set_priority today.

Kind regards, Mike.