Proposals never return in a <3 node cluster

[Hoverbear]

No that's not a heart!

It was noted while playing with the register example (in #67) that proposals seem to fail with cluster sizes under 3.

With 3:

 hoverbear  ~  git  raft  RUST_LOG=raft=debug cargo run --example register put foo 127.0.0.1:8080 127.0.0.1:8081 127.0.0.1:8082        replica_message 
     Running `target/debug/examples/register put foo 127.0.0.1:8080 127.0.0.1:8081 127.0.0.1:8082`
"127.0.0.1:8080"
"127.0.0.1:8081"
"127.0.0.1:8082"
{V4(127.0.0.1:8080), V4(127.0.0.1:8082), V4(127.0.0.1:8081)}
DEBUG:raft::client: connecting to potential leader 127.0.0.1:8080
DEBUG:raft::client: connected
DEBUG:raft::client: awaiting response from connection
DEBUG:raft::client: received response NotLeader
DEBUG:raft::client: had existing connection Ok(V4(127.0.0.1:8082))
DEBUG:raft::client: awaiting response from connection
DEBUG:raft::client: received response Success
[102, 111, 111]
 hoverbear  ~  git  raft  RUST_LOG=raft=debug cargo run --example register get 127.0.0.1:8080 127.0.0.1:8081 127.0.0.1:8082            replica_message 
     Running `target/debug/examples/register get 127.0.0.1:8080 127.0.0.1:8081 127.0.0.1:8082`
"127.0.0.1:8080"
"127.0.0.1:8081"
"127.0.0.1:8082"
{V4(127.0.0.1:8080), V4(127.0.0.1:8082), V4(127.0.0.1:8081)}
DEBUG:raft::client: connecting to potential leader 127.0.0.1:8080
DEBUG:raft::client: connected
DEBUG:raft::client: awaiting response from connection
DEBUG:raft::client: received response NotLeader
DEBUG:raft::client: had existing connection Ok(V4(127.0.0.1:8082))
DEBUG:raft::client: awaiting response from connection
DEBUG:raft::client: received response Success
[102, 111, 111]

With 2:

 hoverbear  ~  git  raft  RUST_LOG=raft=debug cargo run --example register put foobar 127.0.0.1:8080 127.0.0.1:8081             101   replica_message 
     Running `target/debug/examples/register put foobar 127.0.0.1:8080 127.0.0.1:8081`
"127.0.0.1:8080"
"127.0.0.1:8081"
{V4(127.0.0.1:8081), V4(127.0.0.1:8080)}
DEBUG:raft::client: connecting to potential leader 127.0.0.1:8081
DEBUG:raft::client: connected
DEBUG:raft::client: awaiting response from connection
DEBUG:raft::client: received response NotLeader
DEBUG:raft::client: had existing connection Ok(V4(127.0.0.1:8080))
DEBUG:raft::client: awaiting response from connection
^C

And one:

 hoverbear  ~  git  raft  RUST_LOG=raft=debug cargo run --example register put foobar 127.0.0.1:8080                            130   replica_message 
     Running `target/debug/examples/register put foobar 127.0.0.1:8080`
"127.0.0.1:8080"
{V4(127.0.0.1:8080)}
DEBUG:raft::client: connecting to potential leader 127.0.0.1:8080
DEBUG:raft::client: connected
DEBUG:raft::client: awaiting response from connection
^C

On the one case, here is the output of the server:

 hoverbear  ~  git  raft  RUST_LOG=raft=debug cargo run --example register server 1 127.0.0.1:8080                              130   replica_message 
     Running `target/debug/examples/register server 1 127.0.0.1:8080`
DEBUG:raft::server: executing actions: Actions { peer_messages: [], client_messages: [], clear_timeouts: false, timeouts: [Election] }
INFO:raft::replica: Server(1): Replica { id: 1, state: Follower, term: 0, index: 0 }: ElectionTimeout
INFO:raft::replica: Server(1): Replica { id: 1, state: Follower, term: 0, index: 0 }: transitioning to Leader
DEBUG:raft::server: Server(1): executing actions: Actions { peer_messages: [], client_messages: [], clear_timeouts: false, timeouts: [] }
DEBUG:raft::server: Server(1): new connection accepted: UnknownConnection(127.0.0.1:38425)
DEBUG:raft::server: Server(1): received new connection from ClientId(d4325b95dc3e41b38152f29df3ed1e3f)
DEBUG:raft::replica: Server(1): Replica { id: 1, state: Leader, term: 1, index: 0 }: proposal from Client(d4325b95dc3e41b38152f29df3ed1e3f)
DEBUG:raft::server: Server(1): executing actions: Actions { peer_messages: [], client_messages: [], clear_timeouts: false, timeouts: [] }

Checked:

• [x] Majority is being determined correctly.
• [x] It was noted that pub fn count_match_indexes(&self, index: LogIndex) -> usize looks suspicious. Was changed to reflect that the leader always has a match index. Fixed 2 node cluster.
• [x] self.advance_commit_index() only gets called when an append entries response is recieved, which obviously won't happen in a 1 node cluster.

[ongardie]

It's a heart.

[Hoverbear]

@ongardie It totally isn't!