[WIP] Block by Block Synchronous Sate Sync

Olshansk commented 1 year ago

Description

Summary generated by Reviewpad on 07 Aug 23 18:22 UTC

This pull request includes changes in multiple files. Here is a summary of the modifications:

The "consensus_module's state synchronization" file was completely deleted.
The "bus_module.go" file added a new type and a new method.
The "state_machine/module.go" file had comments added and a line modified.
The "state_machine/fsm.go" file had comments added and new states added.
The "hotstuff.proto" file added a comment related to a circular dependency.
The "charts/pocket/README.md" file added a new configuration option.
The "PROTOCOL_STATE_SYNC.md" file had several updates to the content.
The "hotstuff_leader.go" file had modifications in various functions.
The "actor.go" file had changes related to the Output Address design.
The "events.proto" file added two new messages.
The "consensus_module.go" file had several changes and suggestions in the code.
The "state_sync_handler.go" file had modifications in two functions.
The "types.go" file added a new function.
The "config.validator1.json" file had changes in the "servicer" and "ibc" sections.
The "charts/pocket/values.yaml" file added a new key-value pair.
The "p2p/utils_test.go" file had a constant commented as technical debt.
The "defaults.go" file added a new constant.
The "events.go" file had updates in comments.
The "Tiltfile" had changes in the dependencies list.
The "consensus/state_sync/helpers.go" file had changes in three functions.
The "event_handler.go" file had a comment removed.
The "block.proto" file had modifications in the "BlockHeader" message.

Please review these changes and let me know if you need further assistance.

Issue

Fixes #579

Type of change

Please mark the relevant option(s):

[ ] New feature, functionality or library
[ ] Bug fix
[ ] Code health or cleanup
[ ] Major breaking change
[ ] Documentation
[ ] Other

List of changes

What is this PR in its current form?

Basic synchronous block-by-block state sync
Not the perfect/cleanest code, but something that's somewhat understandable
Some code cleanup related to consensus tended to along the way

What is this PR NOT intended to be

Fully asynchronous background state sync
State sync independent of FSM
Any sort of optimization in state sync

Blockers / further improvements needed in this PR

Tickets to track future work (tech debt, improvements, cleanup, optimizations)
An E2E test as part of this PR
A video showing that it works on localnet

Testing

[x] make develop_test; if any code changes were made
[x] make test_e2e on k8s LocalNet; if any code changes were made
[ ] e2e-devnet-test passes tests on DevNet; if any code was changed
[ ] Docker Compose LocalNet; if any major functionality was changed or introduced
[ ] k8s LocalNet; if any infrastructure or configuration changes were made

Required Checklist

[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added, or updated, godoc format comments on touched members (see: tip.golang.org/doc/comment)
[ ] I have tested my changes using the available tooling
[ ] I have updated the corresponding CHANGELOG

If Applicable Checklist

[ ] I have updated the corresponding README(s); local and/or global
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added, or updated, mermaid.js diagrams in the corresponding README(s)
[ ] I have added, or updated, documentation and mermaid.js diagrams in shared/docs/* if I updated shared/*README(s)

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 2.04% and project coverage change: -0.21 :warning:

Comparison is base (2074a1b) 31.52% compared to head (e9fd550) 31.32%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #812 +/- ## ========================================== - Coverage 31.52% 31.32% -0.21% ========================================== Files 107 107 Lines 9034 9096 +62 ========================================== + Hits 2848 2849 +1 - Misses 5846 5906 +60 - Partials 340 341 +1 ``` | [Impacted Files](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network) | Coverage Δ | | |---|---|---| | [app/client/cli/debug.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-YXBwL2NsaWVudC9jbGkvZGVidWcuZ28=) | `11.37% <ø> (ø)` | | | [consensus/block.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL2Jsb2NrLmdv) | `0.00% <0.00%> (ø)` | | | [consensus/debugging.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL2RlYnVnZ2luZy5nbw==) | `0.00% <0.00%> (ø)` | | | [consensus/event\_handler.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL2V2ZW50X2hhbmRsZXIuZ28=) | `0.00% <0.00%> (ø)` | | | [consensus/helpers.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL2hlbHBlcnMuZ28=) | `0.00% <0.00%> (ø)` | | | [consensus/hotstuff\_leader.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL2hvdHN0dWZmX2xlYWRlci5nbw==) | `0.00% <0.00%> (ø)` | | | [consensus/hotstuff\_replica.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL2hvdHN0dWZmX3JlcGxpY2EuZ28=) | `0.00% <0.00%> (ø)` | | | [consensus/module.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL21vZHVsZS5nbw==) | `0.00% <0.00%> (ø)` | | | [consensus/module\_consensus\_debugging.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL21vZHVsZV9jb25zZW5zdXNfZGVidWdnaW5nLmdv) | `0.00% <ø> (ø)` | | | [consensus/state\_sync\_handler.go](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network#diff-Y29uc2Vuc3VzL3N0YXRlX3N5bmNfaGFuZGxlci5nbw==) | `0.00% <0.00%> (ø)` | | | ... and [6 more](https://app.codecov.io/gh/pokt-network/pocket/pull/812?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pokt-network) | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Olshansk commented 1 year ago

@bryanchriswhite This is still a WIP but would appreciate if you can spend some time reviewing and providing feedback, or at least ideas for future improvements. Would be nice to have one other person have some context into how this works and will evolve. Please note that my goal with this PR is to get some sort of state sync into main to unblock other features and we will then improve it. consensus/e2e_tests/state_sync_test.go might be a good start and please ignore the commit history as it will only add confusion rather than help.

Olshansk commented 1 year ago

@bryanchriswhite Just FYI that I have paused work on this PR and am working on the E2E test in #829.

PTAL at this when you have a chance

gitguardian[bot] commented 1 year ago

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

| GitGuardian id | Secret | Commit | Filename | | | -------------- | ------------------------- | ---------------- | --------------- | -------------------- | | [5841025](https://dashboard.gitguardian.com/incidents/5841025?occurrence=99835954) | Generic High Entropy Secret | 5a4ec3bbc58313aca0157a91dc7517624782d9ea | charts/pocket/templates/configmap-genesis.yaml | [View secret](https://github.com/pokt-network/pocket/commit/5a4ec3bbc58313aca0157a91dc7517624782d9ea#diff-33d68fb410c6b83957f59a2c6fcdc96d62a90d236e6a2a65bf24b205baa46f5dR1799) |

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code. 2. Replace and store your secret safely. [Learn here](https://blog.gitguardian.com/secrets-api-management?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) the best practices. 3. Revoke and [rotate this secret](https://docs.gitguardian.com/secrets-detection/detectors/generics/generic_high_entropy_secret#revoke-the-secret?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). 4. If possible, [rewrite git history](https://blog.gitguardian.com/rewriting-git-history-cheatsheet?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data. To avoid such incidents in the future consider - following these [best practices](https://blog.gitguardian.com/secrets-api-management/?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) for managing and storing secrets including API keys and other credentials - install [secret detection on pre-commit](https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

Olshansk commented 1 year ago

@bryanchriswhite I'm going to keep the WIP tag and keep working on it in #874 but have moved it into the ready-for-review stage in order to get more 👀 on it

Olshansk commented 1 year ago

@red-0ne Will look into the broken tests tomorrow.

pokt-network / pocket