Longarithm commented 5 months ago

Goals

Stateless Validation is alternative design for phase 2 of sharding that does not require state rollback to mainnet. It involves significant changes to different parts of the protocol. The goal is to deliver it to mainnet, using StatelessNet to iterate on design.

All open issues for stateless validation are listed here.

Tasks

Ordered by priority. With all this completed, stateless validation can be released. Also the work includes monitoring and fixing issues appearing in StatelessNet.

Timeline view

https://github.com/orgs/near/projects/140/views/1

Task list

[x] https://github.com/near/nearcore/issues/10552
[ ] Single shard tracking + https://github.com/near/nearcore/issues/10965
[x] https://github.com/near/nearcore/issues/10556
- [ ] Determine new hardware Requirements for validators and communicate
[ ] https://github.com/near/nearcore/issues/10259
- [x] https://github.com/near/nearcore/pull/10703
- [x] https://github.com/near/nearcore/issues/10890
- launch blocking parts are implemented
- [x] https://github.com/near/nearcore/issues/11193
- [x] https://github.com/near/nearcore/issues/10780
- [x] https://github.com/near/nearcore/issues/11019
[ ] https://github.com/near/near-one-project-tracking/issues/61
[x] https://github.com/near/nearcore/issues/10565
[ ] https://github.com/near/nearcore/issues/10555
[x] https://github.com/near/nearcore/issues/10557
[x] https://github.com/near/nearcore/issues/10578
[ ] Fix issues and bugs
- [x] stack overflow https://github.com/near/nearcore/pull/10663
- [x] investigate network one shard stalling under mainnet load zulip - localised to network.
- [ ] https://github.com/near/nearcore/issues/10954
- [ ] https://github.com/near/nearcore/issues/10964
- [x] https://github.com/near/nearcore/issues/11065
- [ ] https://github.com/near/nearcore/issues/11099
[ ] #66
- Need to equip ourselves with enough number of tests to stay confident before MainNet launch
- Test chain stays stable when state witness size grows
[ ] #65
[ ] https://github.com/near/nearcore/issues/11142

Side tasks

[ ] Component code refactoring for better development environment @pugachAG
[ ] Data dependency bomb
- [ ] slow distribution of large chunks
[ ] NEP approval
[ ] Write documentation (general + how to test locally and in adversenet)
[ ] Benchmarking - e.g. https://github.com/near/nearcore/issues/10543

Links to external documentations and discussions

Estimated effort

30 weeks on Core team side

Assumptions

Community validators participate in StatelessNet which helps to ensure that stateless validation works properly.

Pre-requisites

https://github.com/near/near-one-project-tracking/issues/5
StatelessNet launch

Release order

Stateless validation is anticipated to be the most significant launch since the Near MainNet rollout, and it's crucial to keep it isolated from other feature releases. To ensure a smooth rollout, I'd like to gather input from the team on what absolutely needs to be deployed to MainNet prior to Stateless validation.

Currently, these are the projects that better be launched in MainNet prior to Stateless validation (ref), as they require changes in trie structure

Congestion control
Yield/resume

Decisions made

More than 2/3 of endorsement is required for a chunk to be included ref @mooori
State witness size hard limit is: TBD @jancionear
State witness size soft limit is: TBD @jancionear
Top 100 validators will be both BP and CP. The remaining validators will be chunk validators. @birchmd
Shard shuffling will be disabled during the initial release ref @bowenwang1996
Cloud based state sync will be used with initial launch ref] @tayfunelmas

Out of scope

[ ] https://github.com/near/nearcore/issues/9982
[ ] https://github.com/near/nearcore/issues/10584
[ ] improving performance of loading memtrie: link
[x] https://github.com/near/nearcore/issues/10991
[x] https://github.com/near/nearcore/issues/11064

walnut-the-cat commented 4 months ago

Need to update the format to make sure it follows the tracking issue template

walnut-the-cat commented 4 months ago

February 26th status update

[Note] The team's effort is spread between this issue and 20. Items listed in this issue focus more on readiness for Stateless validation MainNet release, where as items listed in 20 focus more on success of Stake wars program.

[Completed] Orphan state witness pool
[Completed] StateTransitionData disk usage optimization
[Completed] Handle production of state witness for first chunk after genesis
[Completed] Fix improper panic when a Client receives ChunkStateWitness with invalid shard_id
[WIP] Network cost analysis
[Upcoming] 2x latency in critical path analysis
[Upcoming] State witness size limit
[Upcoming] Stateless validation NEP draft completion and schedule SME review

Longarithm commented 3 months ago

March 8th status update

[Completed] State witness soft size limit
[Completed] Started state witness compression testing, 30-70% size reduction achievable
[Completed] Benchmarking showing 600 TPS on single shard with memtrie and no caches
[WIP] Proposal for validator roles and rewards - waiting to discuss
[WIP] TestLoop framework prepared and draft of memtrie + stateless validation test is made.
[Upcoming] Discussion on contract cache (large values cache)
[Upcoming] State witness hard size limit
[Upcoming] Stack overflow & memtrie internal failure investigation
[Upcoming] Stateless validation NEP draft completion and schedule SME review

Longarithm commented 3 months ago

March 22nd status update

[Completed] Discussed validator roles and rewards.
[Completed] Conclusion on reducing state witness size - will apply compression for now
[Completed] Investigation of losing all chunks on statelessnet - related to network.
[WIP] Enabling memtrie on statelessnet. Working on state sync integration & missing trie value fix
[WIP] Investigating how to resolve network concerns: buffer overflows, low bandwidth for distant nodes.
[Upcoming] Stateless validation NEP draft completion and schedule SME review
[Upcoming] Implementation of validator roles and rewards
[Upcoming] State witness hard size limit - including validation

walnut-the-cat commented 2 months ago

April 9th status update

[Completed] In-memory trie implementation needed before enabling
[Completed] Refactoring needed for state witness compression
[Completed] baseline softlimit except corner case: ref @jancionear
[WIP] Network optimization prototyping @shreyan-gupta @saketh-are
- Prototype two different approaches so we can compare their performance with simulation tool
- Reed-solomon encoding
- Fan-out distribution of state witness
- While looking into this, @saketh-are found out the issue with network latency where it takes more than 2 sec (Europe <-> SouthEast Asia) for neard to exchange even a tiny data (link). Investigation is in progress.
[WIP] Soft limit corner case handling + hard limit implementation: https://github.com/near/nearcore/issues/10890 @jancionear
[WIP] validator seat assignment @birchmd , @mooori
- Simulation is running to verify Michael's calculation: link
[WIP] validator reward calculation (@birchmd )
- Aligned on MVP approach we will take and Michael will share LoE on Thursday: https://github.com/near/nearcore/issues/10556
- Further improvement will be revisited once MVP is done: https://github.com/near/nearcore-private/issues/106
[WIP] In-memory trie test with empty state and fix discovered issues (@staffik)
[Upcoming] In-memory trie test with mainnet state (@Longarithm , @staffik )
- waiting for Forknet to be ready

walnut-the-cat commented 2 months ago

Stateless validation April 11th status update

Forknet is running with in-memtrie enabled, but without any traffic. Once @marcelo-gonzalez fixes traffic generator, we will start sending mainnet traffic and see what breaks (@staffik , @Longarithm , @robin-near )
@jancionear is currently taking the approach number 1 to address corner case that can happen during node deletion for soft limit and hard limit (ref). @jancionear will update with more analysis
Timeline view will be used to track of stateless validation MVP. The team should add any newly created issues that are required before MainNet launch to the view
With resharding, compression seems to have less impact (40% -> 15%) in general, but it's still significant for max witness size. Max witness size with compression in the last 24 hours is ~2.2MB
For Network optimization, we are investigating ways to improve node - node latency by using other than TCP (e.g. UDP, QUIC). At the same time, we will implement reed-solomon coding for state witness. (@shreyan-gupta , @saketh-are )
Reward calculation implementation requires 1 full engineering week to implement. @birchmd will drive it.
[P1] With the current assignment model, top validators are likely to validate/track multiple chunks. We may want to see how things can be optimized post MainNet launch.

walnut-the-cat commented 2 months ago

Stateless validation April 16th update

[DONE]Fix is made for incorrect read of non-existent key, which caused chunk misses in ForkNet
[DONE] State witness compression work is completed
[WIP] TCP optimization is being tested in Forknet.
[WIP] PRs for State witness hard limit and soft limit implementation are out
[WIP] State witness reed solomon encoding
[WIP] Address two security concerns
- Sending witnesses for old epochs
- Replaying same state witness to overwhelm validators
Forknet status
- Forknet ran for more than 4 hours with reasonable performance with MainNet traffic. Shard 2 was suffering with 20-30% chunk misses and we observed several block misses. However, the chain never experienced 100% chunk misses.
- TODOs
- Integrate with State witness size limit and compression
- Better chunk tracing system
- Forknet-control & forknet-test set up for performance comparison
- Merge code to send x% of mainnet traffic

walnut-the-cat commented 2 months ago

Stateless validation April 19th update

Tasks

[DONE] Replaying same state witness to overwhelm validators @Anton Puhach (pugachag)
[Code complete] State witness hard limit and soft limit implementation @Jan Ciołek (jancionear)
[In test] Improve network connection performance on burst traffic @Saketh Are
[WIP] State witness reed solomon encoding @Shreyan Gupta (shreyan-gupta)
[WIP] Missing main transition state proof for old block during shard shuffling @Adam Chudaś (staffik)
[PR review] Sending witnesses for old epochs @Anton Puhach (pugachag)
[TODO] State witness size limit for incoming receipts @Alex Logunov (Longarithm)
[TODO] Make contract code inclusion in state witness deterministic @Alex Logunov (Longarithm)

ForkNet

Shard shuffling is now enabled
Set maximum number of stored log files to 5 from 20 to avoid out of disk space issue
@Razvan Barbascu (vanbarbascu) is exploring ways for continuous back up of ForkNet ref
@Marcelo Diop-Gonzalez shared PR to set interval between transactions to control traffic volume ref
@Marcelo Diop-Gonzalez will work on the way to send synthetic traffic to Forknet
@Adam Chudaś (staffik) will verify the following features are working as expected:
- TCP optimization
- State witness size limit
- State witness compression
Working with @Andrei Mustuc (andrei-near) to explore a way to completely turn off all ForkNet nodes outside of business hours ref
@Shreyan Gupta (shreyan-gupta) will investigate why we ended up with 5 shards instead of 6 shards in ForkNet

cc. @Tayfun Elmas (tayfunelmas) , @Bowen Wang

walnut-the-cat commented 2 months ago

Stateless validation April 23rd update

Tasks

[DONE] Replaying same state witness to overwhelm validators @Anton Puhach (pugachag)
[DONE] State witness hard limit and soft limit implementation @Jan Ciołek (jancionear)
[In test] Improve network connection performance on burst traffic @Saketh Are
[WIP] State witness reed solomon encoding @Shreyan Gupta (shreyan-gupta)
[ETA: April 26th] Missing main transition state proof for old block during shard shuffling @Adam Chudaś (staffik)
[Pending until Anton is back] Sending witnesses for old epochs @Anton Puhach (pugachag)
[WIP - part of congestion control] State witness size limit for incoming receipts @wacban
[TODO] Make contract code inclusion in state witness deterministic @Alex Logunov (Longarithm)

ForkNet

Shard shuffling is turned off while investigation in progress.
@Razvan Barbascu (vanbarbascu) is exploring ways for continuous back up of ForkNet ref
@Marcelo Diop-Gonzalez is working on
- Better MainNet traffic pace control
- longer MainNet traffic replay period
- MainNet traffic replay period from congested hours
@Adam Chudaś (staffik) will verify the following features are working as expected:
- TCP optimization
@jancionear will verify the following features are working as expected:
- State witness size limit
- State witness compression
@jancionear and @staffik to test a way to send synthetic traffic along with MainNet traffic
Working with @Andrei Mustuc (andrei-near) to explore a way to completely turn off all ForkNet nodes outside of business hours ref
@Shreyan Gupta (shreyan-gupta) will investigate why we ended up with 5 shards instead of 6 shards in ForkNet

cc. @Tayfun Elmas (tayfunelmas) , @Bowen Wang

walnut-the-cat commented 2 months ago

Stateless validation April 25rd update

Tasks

[DONE] Test improvement made with TCP optimization. (link)
[DONE] ChunkExtra missing due to lazy Flat head catch up (link)
[WIP] State witness reed solomon encoding @Shreyan Gupta (shreyan-gupta)
[WIP] Missing main transition state proof for old block during shard shuffling @Adam Chudaś (staffik)
[Pending until Anton is back] Sending witnesses for old epochs @Anton Puhach (pugachag)
[WIP - part of congestion control] State witness size limit for incoming receipts @wacban
[TODO] Make contract code inclusion in state witness deterministic @Alex Logunov (Longarithm)

ForkNet

Forknet ran from April 2tth 19:00 UTC to April 26th 01:00 UTC, then got stuck. @robin-near suspects its due to undeleted dump data during reset, but confirmation is needed. Once chain stall had happened, we stopped the nodes and stopped the traffic generator.
This week, we have spent significant amount of engineering time investigating that's less relevant to stateless validation itself, e.g. incorrect ForkNet set up, State sync failure, etc. To address issue, we plan to do the followings:
- @VanBarbascu and @marcelo-gonzalez will look into simple automation of ForkNet so we can reliably start and tear down ForkNet without human error
- Core team will focus on working on known issues and confirm the fix with more scoped tests, such as AdverseNet and Nayduck. ForkNet will be used only during the last step for final confirmation with more expansive network

cc. @tayfunelmas , @bowenwang1996

walnut-the-cat commented 2 months ago

Stateless validation April 30th update

Task

[DONE] Verify state witness compression works as expected @jancionear
[DONE] Store N latest invalid state witnesses @jancionear
[WIP] limit size of transaction @jancionear
[WIP] Missing transition state proof for old block @staffik
[WIP] More integration tests using Nayduck and testloop @pugachAG , @staffik , @robin-near
[WIP] security concern - sending state witness for old epoch @pugachAG
[WIP] Congestion control for stateless validation @wacban
[WIP] Make contract code inclusion deterministic @Longarithm
[Code complete, Testing] Reed solomon encoding for state witness @shreyan-gupta

ForkNet

There were confusion on what needs to be delivered first with automation effort. @tayfunelmas and @walnut-the-cat came together to come up with TODO list to clarify what to focus on first: ref @VanBarbascu , @marcelo-gonzalez
Currently, ForkNet is down and will be soon replaced with the next version using Marcelo's script for fast construction

cc. @bowenwang1996 , @tayfunelmas

walnut-the-cat commented 1 month ago

May 7th update

[Deprioritized] @staffik Missing main transition state proof for old block (#10964)
[DONE] @saketh-are TCP optimization (https://github.com/near/near-one-project-tracking/issues/68)
[DONE] @shreyan-gupta State witness distribution by parts (https://github.com/near/near-one-project-tracking/issues/67)
- Metrics need to be added for proper tracking
[WIP] Node team, ForkNet rerunning
[WIP] @jancionear Txn size limit #11103
[WIP] @robin-near Stateful -> stateless migration prep (https://github.com/near/near-one-project-tracking/issues/65)
[WIP] @staffik Fix Nayduck tests
[WIP] @Longarithm Update CP assignment logic + tracking shard assignment logic #11202, #11213
[WIP] @pugachAG Migrate chunk state witness pre validation logic from orphan witness handler to partial witness actor #11193

cc. @bowenwang1996 , @tayfunelmas

near / near-one-project-tracking

[ProjectTracking]: Stateless validation Mainnet Release #46

Goals

Tasks

Timeline view

Task list

Side tasks

Links to external documentations and discussions

Estimated effort

Assumptions

Pre-requisites

Release order

Decisions made

Out of scope

February 26th status update

March 8th status update

March 22nd status update

April 9th status update

Stateless validation April 11th status update

Stateless validation April 16th update

Stateless validation April 19th update

Tasks

ForkNet

Stateless validation April 23rd update

Tasks

ForkNet

Stateless validation April 25rd update

Tasks

ForkNet

Stateless validation April 30th update

Task

ForkNet

May 7th update