Support Witness role

[siddontang]

From Spanner https://cloud.google.com/spanner/docs/replication

Witness
Witness replicas don’t support reads but do participate in voting to commit writes. These replicas make it easier to achieve quorums for writes without the storage and compute resources that are required by read-write replicas to store a full copy of data and serve reads. Witness replicas:

Are only used in multi-region instances.
Do not maintain a full copy of data.
Do not serve reads.
Vote whether to commit writes.
Participate in leader election but are not eligible to become leader.

It is helpful for us to support crossing IDC replication too.

[siddontang]

Most of the time, the leader can only send Log meta (not including Log data) to the witness, but if the log entry contains ConfChange or Admin(like Split, Merge), we need to send these log data too.

We can check ConfChange with https://github.com/pingcap/raft-rs/blob/master/proto/eraftpb.proto#L6 directly, but for Admin, maybe we need to add a flag to https://github.com/pingcap/raft-rs/blob/master/proto/eraftpb.proto#L24.

IMO, we can only introduce the Witness role to the Raft lib, the Witness can't be promoted to Follower or Learner because it has no data.

The Raft module still sends whole messages to the Witness, and we will decide whether to filter the messages outside. This can make the Raft library simple. Another way is to introduce a send-hook and do the filter in the Raft send function. But I prefer the first way.

We can also support Witness in etcd if possible. /cc @xiang90

[xiang90]

Witness mode can be implemented outside raft by:

1. remove data in log entry
2. drop voting message

But I am fine adding it into raft though.

[hicqu]

Please also take a look at https://github.com/lni/dragonboat/issues/106.