In order to allow the live indexer to index live data before it is sealed, the DPS needs to change its indexing process to allow multiple versions of an index at a given height, in the case of a fork. In order to do this, we can change our indexes to include the block hash.
This will reduce the latency between the time at which events happen and they are available in our API, which is essential for the DPS to be relevant in the Flow ecosystem.
To double check: The block order is very important in the execution tracker, since we need to read trie updates sequentially. Since trie updates contain the previous state commitment they apply to, as long as for each "fork" we get updates in the right order, the current implementation should work.
TODO: Once this is all confirmed, create sub tasks and start implementation.
Changes
Change the consensus follower so that it can warn us about executed blocks
Instead of downloading finalized blocks only, we need to download block data as soon as they get optimistically executed.
Since we download block data for blocks that haven't been sealed yet, we need to add a sanity check to ensure that our index matches with the sealed execution state as we go.
Additional Notes
This is a major breaking change, and will render previously built indexes obsolete. -> DPS v2.0.0 already coming up 😄
Same goes for the Rosetta and Access API repos, they need a new release that makes it clear the change is breaking.
This means the DPS API also needs to be updated to use block hashes for its queries.
This will impact the DPS Access API and DPS Rosetta API repositories, which will no longer be compatible with the previous version of the DPS.
Description
In order to allow the live indexer to index live data before it is sealed, the DPS needs to change its indexing process to allow multiple versions of an index at a given height, in the case of a fork. In order to do this, we can change our indexes to include the block hash.
This will reduce the latency between the time at which events happen and they are available in our API, which is essential for the DPS to be relevant in the Flow ecosystem.
The consensus tracker currently already accesses block data as soon as they are finalized, while the execution tracker has access to all execution data that is uploaded onto its GCP bucket. All finalized blocks get pushed onto the execution tracker's queue, to be downloaded in the same order as they were finalized.
To double check: The consensus tracker follows the access nodes to bind its
OnBlockFinalizedso I assume that should be the case and we don't need to change anything.To double check: The block order is very important in the execution tracker, since we need to read trie updates sequentially. Since trie updates contain the previous state commitment they apply to, as long as for each "fork" we get updates in the right order, the current implementation should work.
TODO: Once this is all confirmed, create sub tasks and start implementation.
Changes
Additional Notes
v2.0.0already coming up 😄