Delayed messages to self can cause leader to be elected without quorum

[colin-scott]

Upon receiving an ElectionTimeout message, the Candidate sends a BeginElection message to its clusterSelf.

It's possible that if the BeginElection message is delayed, the Candidate might vote for itself twice.

Specifically, consider the following scenario:

• Candidate receives ElectionTimeout message
• Candidate sends BeginElection message [which is delayed]
• Candidate receives ElectionTimeout message
• Candidate sends BeginElection message
• Akka dispatcher delivers first BeginElection message
• Akka dispatcher delivers second BeginElection message

Upon receiving the BeginElection message, the Candidate increments the number of votes it has received for the current Term (without checking whether it has already voted for itself).

However, the Candidate does not change its current Term without receiving an ElectionTimeout message. Consequently, in the scenario above, the Candidate would vote for itself twice in the same Term.

This scenario would admittedly be triggered very rarely in practice. But I do believe that it would possible to trigger it, especially if the election timeout value was set to a low value; akka's dispatcher doesn't AFAICT provide guarentees on when particular messages are delivered.

[colin-scott]

For what it's worth, I have a (non-pull-request-worthy) fix for this issue here:

https://github.com/NetSys/sts2-applications/commit/3367d2df722e95030b5e0ca634c6e9f04eaa0777