refactor(consensus)!: block diffs, exec on propose and locks improvement

[sdbondi]

Description

• Generate and persist block diff for an accepted block
• generate a ProposedBlockChangeSet and persist for an accepted block
• State store read transactions no longer require a mutable reference
• Use PendingSubstateStore to track block changes and locks
• Remove previous locking code
• Allows previously conflicting local-only transactions to be sequenced using uncommitted outputs from other local only transactions as inputs (Local-Only-Rules)
• Fix multiple issues resulting in test flakiness
• Execute deferred transactions on propose and on prepare
• Adds deferred transaction test (currently for local-only case only)
• decide_what_to_vote is read-only
• Unignored some previously-flaky consensus tests
• Removed BlockTransactionExecutorBuilder trait
• Updated block sync

Motivation and Context

This PR improves locking, substate management and allows local-only transaction concurrency.

It adds the following concepts: BlockDiff - contains the substate changes (UP or DOWN) for a block. Is used to commit state in a commit block PendingSubstateStore - tracks and validates changes to the substate store and decouples state from execution. It is used to generate a BlockDiff, track locks in the current block context, and to generate the state tree diff. ProposedBlockChangeSet - contains all the pending changes to be committed when deciding to ACCEPT a new block TransactionExecution - decouples execution results from a transaction record. These represent an execution within the context of a block. Once a transaction is finalized, the execution result is added to the final result in the transaction.

Possible issues: Currently to fetch the correct transaction executions and locks for a block we need to limit the results to those that match the chain. This query will grow slower as the chain progresses. Bugs may still occur in the unhappy/malicious path.

How Has This Been Tested?

New deferred transaction consensus test, new unit tests, existing tests, removed ignore attribute from several consensus tests.

What process can a PR reviewer use to test or verify this change?

Submit transactions without specifying the version of the input. Submit local-only transactions concurrently that spend previous outputs from other local only transactions.

Breaking Changes

• [ ] None
• [x] Requires data directory to be deleted
• [ ] Other - Please specify

BREAKING CHANGE: database structure changed

[github-actions[bot]]

Test Results (CI)

541 tests  +9   541 :white_check_mark: +9   1h 36m 13s :stopwatch: - 20m 0s  63 suites ±0     0 :zzz: ±0    2 files   ±0     0 :x: ±0 

Results for commit 766f8a0e. ± Comparison against base commit 198e6ec6.

This pull request removes 2 and adds 11 tests. Note that renamed tests count towards both.

``` consensus_tests ‑ consensus::leader_failure_output_conflict tari_dan_common_types ‑ substate_address::tests::to_committee_shard ``` ``` consensus_tests ‑ consensus::deferred_execution consensus_tests ‑ consensus::foreign_shard_decides_to_abort consensus_tests ‑ consensus::multi_shard_propose_blocks_with_new_transactions_until_all_committed consensus_tests ‑ consensus::output_conflict_abort consensus_tests ‑ substate_store::it_allows_down_then_up consensus_tests ‑ substate_store::it_allows_locks_within_one_transaction consensus_tests ‑ substate_store::it_allows_substate_up_for_v0 consensus_tests ‑ substate_store::it_disallows_more_than_one_write_lock_non_local_only consensus_tests ‑ substate_store::it_fails_if_previous_version_is_not_down tari_dan_common_types ‑ substate_address::tests::max_committees … ```

:recycle: This comment has been updated with latest results.