search

near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.33k stars 627 forks source link

Flat storage MVP #7327

Closed exalate-issue-sync[bot] closed 1 year ago

exalate-issue-sync[bot] commented 2 years ago

Owner: Alex L

Requires completion of READ only flat storage. Tracking issue for flat storage https://github.com/near/nearcore/issues/7327

LoE for Q1 includes:

Picture: https://miro.com/app/board/uXjVP3ziycI=/

Implementation roadmap for Flat Storage (FS)

This section is kept up-to-date to provide a high-level progress overview for anyone not directly involved in the implementation. We used it when we didn't have github tracking issues for subtask. Detailed subtasks are also tracked on Jira.

  1. Basic flat storage (DONE)

    • [x] implement state lookup that reads from flat state
    • [x] keep flat state in sync when we write to state
    • [x] implement deltas to deal with forks

  2. Data migration (DONE)

    • [x] implement migration code
    • [x] do migration in the background without hindering block processing #8053
    • [x] test and handle corner cases in migration, see #8500

  3. Fully operational read-only FS (DONE)

    • [x] Support state sync
    • [x] Support catch-up mode, that needs to download a non-tracked shard
    • [x] Naively remove non-tracked shards' flat state
    • [x] Add metrics and make long-term observations on canary nodes doing FS and old lookup in parallel
    • [x] #8436
    • [x] unit tests
    • [x] integration tests, see #8500
    • [x] estimate and define new gas costs #8006
    • [x] implement flat state cache optimizations
    • [x] debug tools (inital ideas listed here: https://pagodaplatform.atlassian.net/browse/STR-85)

  4. Add docs and update existing documentation to include flat storage (DONE)

There is also a blocker for shipping: https://github.com/near/nearcore/issues/8032. Without it, undercharging coming from non-charged TTNs can't be controlled, but TTN parameter = 10 would help.

Links

Existing documentation:

https://nearinc.atlassian.net/wiki/spaces/EAP/pages/58556568/Flat+Storage

https://docs.google.com/document/d/1Tla2JtHHc6rJwZPUOF0skoaLiExrcV1PW2F65llfHG8/edit#heading=h.2r15ioo581ig

Related works and discussions:

Main flat storage discussion channel

https://nearinc.atlassian.net/browse/CRT-8 and subsequently linked tasks

https://nearinc.atlassian.net/browse/CRT-86

Picture of flat storage structure

https://near.zulipchat.com/#narrow/stream/329092-pagoda.2Fstorage.2Fprivate/topic/Action.20receipt.20undercharging

bowenwang1996 commented 1 year ago

@Longarithm is the progress in this tracking issue up to date?

Longarithm commented 1 year ago

Yes. The PR for catchups needs some polishing to be merged https://github.com/near/nearcore/pull/8193, tests for migration and proper flat state removal are TODO, other items weren't started yet.

pugachAG commented 1 year ago

Progress update (created together with @Longarithm):

Total progress for Q1: 45%

cc @walnut-the-cat

walnut-the-cat commented 1 year ago

8531 is now merged. Does it complete #8398?

Longarithm commented 1 year ago

Yes, closing

jakmeier commented 1 year ago

A new issue related to FS: https://github.com/near/nearcore/issues/8791

I don't think we should block FS on it, since it functionally does not depend on it. We just have to make sure #8791 also gets done before the next release cut, which should be more than enough time.

Longarithm commented 1 year ago

We completed the work for flat storage reads. There are some remaining tasks to release it in 1.34, which I listed in #8827. We plan to spend some time on it in Q2, as 1.34 is expected to happen in Q2.