Raft: Modify to generate deterministic raftIds, remove required `--raftjoinexisting $raftId` flag

[libby]

Currently, raftIds are variable, and sometime depended on the order of the static-nodes.json file, while other times they need to be obtained from an outside source. This means that a new node cannot determine its own $raftId without being explicitly passed the information via the geth startup param (--raftjoinexisting $raftId) at startup.

This is overly complex, and makes automating operations (adding / removing nodes) difficult for projects like ConsenSys qubernetes which supports easily creating, deploying and managing Consensys Quorum networks on Kubernetes. Related to add nodes PR in qubernetes / k8s https://github.com/ConsenSys/qubernetes/pull/60

Current RaftId Behavior

Starting a New Network

• The RaftId is obtained from the order of the static-nodes.json file when starting the network for the first time.
• issue: this means that every node in the network must have the same exact static-nodes.json file with the same exact order. This only holds at bootup time, after bootup the static-nodes.json is not no longer the raftId recorded of truth.
• update request: the raftId should not be dependent on the order of the static-nodes.json file, but should be obtained deterministically by the nodes.

Adding a New Node To Raft Network

1. Add the new node from a node in the current cluster by running raft.addPeer($enodeId) and obtain the raftId that is returned (this id is not deterministic but generated by the node the command was executed on, it is supposed to be the next monotonically increasing int of all known nodes in the cluster, the command can only be called once).

2. Take the $raftId returned in step 1 and add the --raftjoinexsiting $raftId to the geth startup params for the node that is being added.

3. Additionally when removing a node and re adding it, it will receive a new $raftId and the old $raftId can never be used again, this is a current requirement as the id are monotonically increasing, and need to ensure uniqueness.

Issues: This has many drawbacks for operations, and makes adding/removing raft nodes overly complex, e.g. deployment configs will be constantly in need of updating . The difficulty comes from managing the state outside of the Quorum node itself, and requiring different config params dependent on whether it is is an initial node in the network or a new node joining the network, or a node that has been added, removed and added again. A new node has no way of knowing its $raftId apart from obtaining it from the passed in flag --raftjoinexsiting $raftId.

Update request/proposal: If the raftId for every node was deterministic, e.g. based on nodeId, every node, including the new node would be able to determine its own raftId, this would remove the requirement of passing the raftId via a flag when starting a new node and would keep the geth startup params the same for initial and current nodes.

Additional Feature Request

• Currently, raft.addPeer($enodeid) will only return the $raftId once after which it returns an error.
• Update request: make this call idempotent. Calls to raft.addPeer($enodeid) should alway return the $raftId and can additionally return a bool var to indicate whether or not it was already added to the network.

System information

Quorum version: v2.7.0

[melihbirim]

@libby Is there any work in progress for this issue?