brianlong
commented
3 years ago I didn't discuss network topology above, but we need to take into account that the network delay for any leader's first slot will always be higher than slots 2-4. In the example above, the data for slot 591 traveled straight-ish from Ogden, UT to my node in Frankfurt, DE because that was 93P6's 4th slot.
For slot 592, I need to wait for the final shred of 591 to be sent from Ogden, UT to Tokyo, JP before FzPo can start building 592. Then the first shred of 592 is sent from Tokyo to Frankfurt. For slots 593-595, we don't have the extra hop because FzPo is building on its' own blocks.
For every leader transition, we need to insert NN milliseconds of 'sync time' when waiting for the first shred of the new leader's first slot.
For anyone who has rowed in an 8-person rowing shell, you will know intuitively how this works. There is a slight pause near 'the catch' for all of the rowers to get in sync before the next power stroke.
The rhythm of the cluster looks like 1-2-3-4-sync-1-2-3-4-sync-1-2-3-4 and so on.
jon-chuang
ryoqun
sakridge
carllin
Problem
The race between the current leader "L0" and the next leader in line "L1", described in https://github.com/solana-labs/solana/blob/22a18a68e3ee68ae013d647e62e12128433d7230/docs/src/implemented-proposals/leader-leader-transition.md, is causing skipped slots. See the "Supporting Information" section below for a view of four skipped slots from the perspective of three different validators.
I believe that the race is intended to shorten the gaps in transaction processing when L0 is offline. Instead, with slow network propagation, we see situations where L1 creates a new fork for their 4 slots and then loses out when the voters vote for L0. In cases like this, the ONLY WAY to resolve the conflict is to skip 4 slots. Ironically, the policy intended to minimize the impact of skipped slots is actually causing them.
I see an inconsistency between the relative patience of L1 and the voters. L1 is acting in an impatient manner by jumping ahead to create a new fork. In contrast, the voters will patiently wait for L0's work to arrive before voting. We are asking/begging for skipped slots/forks when L1 and the voters are working from a different set of rules.
Supporting Information
Thanks to @nyet | Blockdaemon & @Zantetsu | Shinobi Systems for sharing logs. I see some inconsistent behavior related to event timing between the leader and the validators. Here are some details for discussion. Relevant slots & leaders:
For all three of us (Block Daemon, BlockLogic, & Shinobi), the transition from 591 => 592 looked normal. Here are some events from Zantetsu's log (timestamps are seconds):
You'll notice that DDnA started to send shreds for 596 shortly after FzPo sent shreds for 592.
After both banks were frozen, Shinobi voted on 592 about 2.26 seconds after the first shreds arrived and 2.46 seconds after voting for 591. Keep those times in mind as we go along. I think they matter.
For Block Daemon, we also see a smooth handoff from 591 => 592. They vote for 592 about 1.30 seconds after receiving the first shred and 1.74 seconds after voting for 591.
On my server, the hand-off from 591-592 took a little longer (1.14 seconds). However, it looks like DDnA didn't wait for a reasonable grace period before starting a new fork. The fork for 596 was created 0.32 seconds after the first shred from 592 and 1.47 seconds after voting for 591. The frozen bank for 592 was delivered 1.02 seconds after the first shred and 2.17 seconds after voting for 591.
So, here is where I see the inconsistent behavior, Shinobi was comfortable waiting over 2 seconds before voting on 592. Yet, DDnA created a new fork less than 1.5 seconds after voting on 591.
If DDnA had waited a similar period, it would have seen the frozen bank for 592 and NOT created the fork at 596.
Proposed Solution Ideas
To resolve the conflict, we need L1 and the voters to share the same behavior. Either the cluster is patient or impatient, not both at the same time. Here are a couple of solution ideas. One solution for a patient cluster and another for the impatient fast-twitch crowd.
1) Patient Cluster: L1 should only jump ahead when L0 is known to be delinquent. Otherwise, wait for the shreds to arrive and proceed normally. Please note that L0 in the example above is located in Tokyo and it will have a longer propagation time to most of the nodes in the cluster. The cluster will be much more tolerant of network propagation delays. The cluster will also have higher transactional throughput because we are not skipping as many slots. (There will still be a maximum wait period to avoid stalls. Both L1 and the voters will use the same wait period.)
2) The Impatient Cluster: The behavior of the voters will be consistent with L1, not L0. The voters will recognize that L0 was slow to send data, and will prefer to vote for L1 in these cases. This solution may have a further unintended consequence of forcing nodes out of locations like Tokyo. We can't be too impatient if we want to see validator nodes all over the world. This solution also means that we are allowing more skipped slots which lowers the transactional throughput of the cluster. Sometimes faster isn't actually faster.
We'll need to debate the Patient v. Impatient cluster before arriving at a final solution. Once decided, the path forward will be clear.