This PR simply adds an additional grace period for the nodes to wait before attempting to call try_block_sync.
Ideally, try_block_sync isn't called until the first PrimaryPing message is received, however in the initial bootup of validators, the nodes need to consider themselves synced to send the PrimaryPing in the first place. Hence, the heuristic wait approach.
This is needed because we've observed that during bootup, nodes were (occasionally) incorrectly setting themselves as synced and when they were in fact much further behind than the other validators. This caused them to incorrectly start processing proposals and certificates from the other validators instead of actually syncing.
Here is an example of an observed sequence of logs that should not happen:
DEBUG Skipping batch proposal (node is syncing)
INFO No connected validators
DEBUG Fetched {} missing previous certificates for round {} from '{}'
DEBUG Primary is not ready to propose the next round
DEBUG Primary is safely skipping a batch proposal (please connect to more validators)
Upon further debugging, this seems to happen when the validator node call try_block_sync before receiving block locators from peers via a Event::PrimaryPing. The call to try_block_sync makes the node think it's synced since there are no block locators to sync from, and thus starts processing proposals/certificates from peers.
Motivation
This PR simply adds an additional grace period for the nodes to wait before attempting to call
try_block_sync.Ideally,
try_block_syncisn't called until the firstPrimaryPingmessage is received, however in the initial bootup of validators, the nodes need to consider themselves synced to send thePrimaryPingin the first place. Hence, the heuristic wait approach.This is needed because we've observed that during bootup, nodes were (occasionally) incorrectly setting themselves as synced and when they were in fact much further behind than the other validators. This caused them to incorrectly start processing proposals and certificates from the other validators instead of actually syncing.
Here is an example of an observed sequence of logs that should not happen:
Upon further debugging, this seems to happen when the validator node call
try_block_syncbefore receiving block locators from peers via aEvent::PrimaryPing. The call totry_block_syncmakes the node think it's synced since there are no block locators to sync from, and thus starts processing proposals/certificates from peers.