Block data for flat storage head may be GC-d

near / nearcore

Reference client for NEAR Protocol

https://near.org

GNU General Public License v3.0

2.31k stars 619 forks source link

Block data for flat storage head may be GC-d #11583

Closed Longarithm closed 3 months ago

Longarithm commented 3 months ago

...if the shard has 0 validators and 0 tx&receipts load. We discussed this issue around here: https://near.zulipchat.com/#narrow/stream/308695-nearone.2Fprivate/topic/Statelessnet.20new.20release/near/444673576. We observed specifically that ChunkExtra is missing, which is required to load memtrie.

Two possible solutions:

on clear_data, when gc stop height is determined, all flat storage heads must be moved above that height. If this is not possible for some reason, clear_data should be no-op at this iteration;
if flat storage head is < 3 epochs behind current epoch, we can enforce flat storage head update, as FS head lag is needed only for snapshotting previous (or prev prev?) epoch data.

cc @wacban

wacban commented 3 months ago

I would propose the following:

Introduce an invariant that TAIL <= flat head. This can be enforced by not allowing garbage collection to go beyond the flat head.
Lock the flat head for the duration of state snapshot (this is already done, no changes here)
Move the flat head to the final block, unless it is locked. This means that it should also be moved on empty chunks.
Add a warning if the tail is more than GC_PERIOD + 2 behind head.

Longarithm commented 3 months ago

Move the flat head to the final block, unless it is locked

Strictly speaking, flat head can be moved too far before snapshotting is triggered. This is because arbitrary number of chunks in the end of epoch can be missing.

wacban commented 3 months ago

Strictly speaking, flat head can be moved too far before snapshotting is triggered. This is because arbitrary number of chunks in the end of epoch can be missing.

Ah I see. What do we do about it today? Is this the reason why we don't move the flat head on empty chunks? Would it make sense to always move the flat head to the latest new chunk (barring the other invariants)?

Longarithm commented 3 months ago

fixed in #11599