Consensys / teku

Open-source Ethereum consensus client written in Java
https://consensys.io/teku
Apache License 2.0
660 stars 278 forks source link

Allow the option to store blobs to filesystem as they're being removed from database #8513

Open rolfyone opened 1 month ago

rolfyone commented 1 month ago

Blobs are currently stored for the mandatory time in database and then deleted.

Rather than simply deleting them, it'd be helpful to be able to write the blobs to filesystem.

I could imagine keeping this relatively simple and just specifying something like --data-storage-store-old-blobs-path=/data and we'd then start writing all blobs to the filesystem before they get archived.

Theres a tool already in use https://github.com/base-org/blob-archiver that basically dumps the blobs api output per block, we could do similar re. file naming etc potentially if its not too crazy.. Remembering there'll be 7200 slots per day so we probably want output in some kind of structure.

Work is split into following tasks:

david-ry4n commented 5 days ago

Moved plan to main description.

tbenr commented 5 days ago

The plan makes sense to me! bare in mind that you used fixes xxx in the first PR description, so once you merge that it will automatically close this one, which instead seems like an "epic" tracking several steps. I'd move the checkboxes in the main issue description so we can keep it open and track the status.