search

zeta-chain / node

ZetaChain’s blockchain node and an observer validator client
https://zetachain.com
MIT License
164 stars 105 forks source link

Snapshot Restore could benefit from download and untar in a single step #2915

Open CryptoFewka opened 2 days ago

CryptoFewka commented 2 days ago

Rather than downloading the archive snapshot in one step, and then unpacking it in another step, we should consider streaming the download in to tar through the pipeline so that the overall process completes more quickly.

https://github.com/zeta-chain/node/blob/3eba70102a12be9ab55f0939b1a5260251ef2d5c/contrib/docker-scripts/start.sh#L140-L144

CryptoFewka commented 2 days ago

To be explicit, it will also cut the free disk space required in half, because it won't need to download first, and then unpack.

Note: this change would cause the download to be unable to resume if interrupted. I do not believe that the current process will use an already downloaded, or partially downloaded snapshot currently though.

We would need to decide which is better:

  1. A process that is slower and requires roughly 9 TB of free disk space.
  2. A process that is faster, requires 4.5 TB of free disk space, but cannot tolerate download interruptions