Optimize consensus follower to avoid lagging 1 view when observing finality

[AlexHentschel]

As part of updating the consensus follower to Jolteon, we front-loaded the all validity checks into the compliance layer. The consensus follower now only stores valid blocks on disk. Furthermore, it only forwards blocks to its local HotStuff follower, whose validity has been confirmed. The follower does not validate blocks in full. Instead, it mainly validates QCs (and a few other details of the block header) and considers blocks B valid only if there is a QC for B known. This means that the consensus follower waits for a child block, before it processes B.

Details:

• assume we receive a child C of block B, i.e.

  A <- B <- C

  where A.View + 1 == B.View, i.e. (A <- B) forms a direct one-chain.

• When receiving C, we the follower validates the QC contained in it. Assuming the QC is valid, B is then certified, i.e. valid and the block is submitted to the HotStuff follower.

• The HotStuff follower determines finality.

  • Formally, A is already finalized, because it has a certified child B, which satisfies B.View == A.View + 1. At the moment,
  • At the moment, the follower only sees B but does not know that a QC for B already exists. From the perspective of the finalization logic, B is treated as an uncertified block.

  Therefore, the follower lags one view behind when observing finality

• The Follower uses Forks internally. At the moment, Forks assumes all input blocks to be Proposals. This is because Forks is used by the full consensus logic that mainly deals with proposals (not yet certified) on the happy path.

  The follower has no concept of 'certified blocks'

Definition of done:

On the happy path, we desire:

• [x] Part 1 implemented in PR #4155:

  At the moment, Forks works only with Proposals. Proposals include optionally a TC, which is completely irrelevant for the follower. For the long-term, it would be beneficial if forks would ingest only Validated Block and Certified Block. More information is not necessary for determining finality (the purpose of Forks). The only reason Forks currently uses Proposals is for crash-recovery of the pacemaker. However, this adds additional requirements to Forks and various code in the follower to work with Proposal everywhere.

  • Change PaceMaker to accept recovery information in the constructor, so we don't depend on Forks for recovering the PaceMaker.
  • Note: currently the PaceMaker emits notifications to the telemetry consumer. When the pacemaker is initialized the telemetry consumer has not been started and should not receive events. Therefore, we should be separating the notification logic from the business logic.

• [x] Part 2 implemented in PR #4190:

  Forks can ingest Proposals as well as CertifiedBlocks. The finality logic in Forks accounts for the certifying QC.

• [x] Part 3 implemented in PR #4234:

  The follower's compliance layer supplies certified blocks to Forks (via the pipeline Compliance → FollowerLoop → ~FollowerLogic~ → Forks)

[AlexHentschel]

Furthermore, I would suggest to remove the FollowerLogic interface and implementation and have the FollowerLoop directly interact with Forks.

Thereby, we would be removing the validation step from the HotStuffFollower, which I think is fine, as correctness of the proposals is already guaranteed by the follower's compliance layer.

[jordanschalm]

The follower's compliance layer forwards certified blocks to Forks

Forks is not thread-safe. Currently only the single-threaded event loop forwards data to Forks for processing. If the compliance layer also forwards certified blocks to Forks, then we will need data synchronization. Alternatively the compliance layer could forward to the event loop first, which in turn would forward to Forks.

Flagging in case this hasn't come up yet -- did you have a preference for how to approach this?

[AlexHentschel]

Re Jordan's comment: Basically, my intention is that Forks can work within the consensus follower as well as the consensus participant.

• Currently, we have Compliance -> FollowerLoop -> FollowerLogic -> Forks While there are some abstraction layers in between, the input blocks for Forks are originally supplied by the compliance layer.
• I want to keep this pipeline (maybe minus some simplifications), where the FollowerLoop queues the blocks from the compliance layer and feeds them single-threaded into Forks
• The central aspect is to extend Forks so it can consume validated proposals (consensus participants) as well as certified blocks (consensus follower) 👉 PR #4190

[Update] issue description to reduce ambiguity

[AlexHentschel]

last part implemented by https://github.com/onflow/flow-go/pull/4234