server.rs occasionally resets the Tcp connection to remote peers, either in the case that the peer reconnects to the local Server, or the local Server becomes disconnected and initiates a reconnection. In both cases existing messages in the send/receive queue are cleared. Clearing the messages is the correct behavior in the long run, but we currently do not handle it correctly, since the local Replica assumes any message sent to a peer (through the Actions struct) will be delivered.
This is easily fixed by adding a new event type for Replica to handle: connection_reset. This would inform the Replica that any outstanding messages to the peer should be considered lost, and give the Replica a chance to return an Actions struct containing messages to send to the peer on the new connection.
In all cases it is possible for the Replica to determine what messages to send based on it's local state:
Leader - a leader should send an AppendEntries message to the reset peer containing the latest commits between the peer's next_index, and the current leaders index (or an empty 'heartbeat' if there is no difference). In the case that the peer missed messages for any reason (for example, because they were in the send queue), the normal Raft inconsistent previous entries mechanism will takeover (see Replica::append_entries_response).
Candidate - A candidate should resend the RequestVote request to the reset peer, if it has not already received a response.
Follower - A follower should not do anything, if the reset peer happens to be the leader, it will be responsible for sending a message.
server.rsoccasionally resets the Tcp connection to remote peers, either in the case that the peer reconnects to the local Server, or the local Server becomes disconnected and initiates a reconnection. In both cases existing messages in the send/receive queue are cleared. Clearing the messages is the correct behavior in the long run, but we currently do not handle it correctly, since the local Replica assumes any message sent to a peer (through theActionsstruct) will be delivered.This is easily fixed by adding a new event type for Replica to handle:
connection_reset. This would inform theReplicathat any outstanding messages to the peer should be considered lost, and give theReplicaa chance to return anActionsstruct containing messages to send to the peer on the new connection.In all cases it is possible for the
Replicato determine what messages to send based on it's local state:Leader- a leader should send anAppendEntriesmessage to the reset peer containing the latest commits between the peer'snext_index, and the current leaders index (or an empty 'heartbeat' if there is no difference). In the case that the peer missed messages for any reason (for example, because they were in the send queue), the normal Raft inconsistent previous entries mechanism will takeover (seeReplica::append_entries_response).Candidate- A candidate should resend theRequestVoterequest to the reset peer, if it has not already received a response.Follower- A follower should not do anything, if the reset peer happens to be the leader, it will be responsible for sending a message.