search

onflow / flow-go

A fast, secure, and developer-friendly blockchain built to support the next generation of games, apps, and the digital assets that power them.
GNU Affero General Public License v3.0
531 stars 176 forks source link

Chunk Data pack Pruner #6516

Open j1010001 opened 1 week ago

j1010001 commented 1 week ago

Why

In general, see Epic: #6515

Specifically for this issue:

  1. Chunks data packs are stored on EN and use the most storage storage (growing ~1TB in 2 weeks, for first 2 weeks of MN26 - traffic dependent).
  2. Pruning requires EN downtime, so requires manual intervention every time the storage gets close to a limit.

How will we measure success ?

Current status

Chunk data packs are already stored in Pebble DB on EN on Miannet. We haven’t implemented pruning yet, current solution requires downtime on EN to remove old data. Scope: We have a POC PR based on v0.33, it needs to be ported to master, design and implementation reviewed, required refactoring of the POC PR finished. No HCU required to deploy, only rolling upgrade.

Estimate

5 dev/weeks to be ready to deploy.

DACI

Role Assigned
Driver Technical: @zhangchiqing , EM: @j1010001
Approver @Kay-Zee
Consulted @AlexHentschel, @franklywatson
Informed Flow protocol Engineering team
zhangchiqing commented 1 week ago

It's been 7 days since the latest spork (Sep 25), and the mainnet EN produced 480GB chunk data packs data.