stops uploading raw snapshots to R2

filecoin-project / filecoin-chain-archiver

Filecoin snapshot / chain export software

Other

5 stars 3 forks source link

stops uploading raw snapshots to R2 #69

Closed ognots closed 1 year ago

ognots commented 1 year ago

lotus now support directly importing zstd compressed car files. stop uploading raw snapshots, as they are too big, and no longer needed

ognots commented 1 year ago

note: I have left the multi-writer functionality in, as we may want to upload to several object stores in the future

ognots commented 1 year ago

Preliminary results from lightweight snapshot service raw car deprecation Testing https://github.com/filecoin-project/filecoin-chain-archiver/pull/69

job execution time reduced

~2h to 1.3h
we will be able to scale down from 5 nodes to 3 nodes again, with these execution times

snapshot artifacts size goes from 1.4TiB to 0.5TiB

Conclusion: we expect significant cost savings from this effort on bandwidth and compute time.

ognots commented 1 year ago