[State sync engine] Design sunsetting protocol

[synzhu]

We will need a sunset boundary (e.g. once a day / week) past which nodes are synced to a state checkpoint rather than a sequence of state diffs.

We will need to figure out the details of how this will be implemented:

• Over time, state diffs can be accumulated into batches of n starting from the state checkpoint, which is how nodes can catchup from the checkpoint to the latest state.
• How does a newly spun up node discover what the CID of the latest checkpoint is? Some ideas:
  • We could store a block id -> CID mapping in the DHT. But is this reliable / BFT?
  • Each execution result could contain the hash for the latest checkpoint as of that block. Thus, anyone following consensus can find out the hash of the latest checkpoint, and therefore start requesting that checkpoint.
  • There could be a separate channel that is used for requesting this information (and if so, we will need to create a new engine on execution nodes and access nodes to respond to these requests):
    • A newly spun up node could request the latest checkpoint hash on this channel.
    • We also would need a separate channel where the node could find out about the aggregated state diffs from the checkpoint to latest state.

[peterargue]

A requirement of the state sync is that DPS is able to generate its index for all blocks within the spork. Any checkpoint we use will either need to be based on a DPS index checkpoint, or retain enough information for DPS to build its index for blocks contained within the checkpoint.

[synzhu]

Hey team! Please add your planning poker estimate with ZenHub @peterargue @vishalchangrani @m4ksio @franklywatson