[SPIKE] State Sync Explorations w/ LibP2P and w/o the FSM

[Olshansk]

Objective

Simplify the architecture around state sync.

Origin Document

In #352, we added the first functional (but basic) implementation of state sync touching many different parts of the codebase.

FSM Dependencies

For example, fsm_handler in consensus is dependant on fsm.go to implement consensus/doc/PROTOCOL_STATE_SYNC.md, but the E2E codeflow is hard to follow.

There may be ways to encapsulate 100% of the logic in consensus/state_sync consensus/state_sync

Lack of P2P Dependencies Dependencies

The business logic for the state sync client and server requests is spread across different places and is dependent on the consensus.server_mode_enabled config and consensus/types/proto/state_sync.proto proto. As explained in the deliverable below, there might be a way to leverage libp2p gossip to simply an asynchronous process of requesting/receiving metadata & blocks over these channels.

Goals

• Look into way to simplify the E2E state sync implementation & architecture
• Research whether using LibP2P as a pubsub is viable (and efficient) for state sync
• Research whether the FSM (see state_machine/fsm.go) is needed for state sync.

Deliverable

• [ ] Look into removing fsm_handler altogether and encapsulating the state sync logic entirely in the StateSyncModule
• [ ] Look into using the messages in consensus/types/proto/state_sync.proto over libp2p pubsub channels as we do for gossip and routine (see background topic here) to simplify the amount of code managed by statesync itself

Non-goals / Non-deliverables

• This ticket does not capture the reimplementation of state sync

General issue deliverables

• [ ] Update the appropriate CHANGELOG(s)
• [ ] Update any relevant local/global README(s)
• [ ] Update relevant source code tree explanations
• [ ] Add or update any relevant or supporting mermaid diagrams

Testing Methodology

• [ ] Task specific tests or benchmarks: make ...
• [ ] New tests or benchmarks: make ...
• [ ] All tests: make test_all
• [ ] LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md
• [ ] k8s LocalNet: verify a k8s LocalNet is still functioning correctly by following the instructions here

---

Creator: @Olshansk Co-Owners: @bryanchriswhite