search

filecoin-project / filecoin-chain-archiver

Filecoin snapshot / chain export software
Other
5 stars 3 forks source link

Add support for zstd compression #40

Closed travisperson closed 1 year ago

travisperson commented 2 years ago

zstd compression reduces the size of snapshots quite significantly we should explore how we can incorporate this compression. I have not used it before so unsure if it will work well in our current streaming model, or it this will need to be a post-processing to create a zstd compressed file and add it back.

ribasushi commented 2 years ago

@travisperson yup, it works with streaming. zstd compresses faster than lotus spits things out, it takes ~4 cores to keep up. This is literally what I do today:

lotus chain export --tipset @$height --recent-stateroots $span_epochs --skip-old-msgs \
    /dev/stdout 2> >( grep --line-buffered -v "rpc output message buffer" 1>&2 ) \
| tee >( sha256sum > "$expdir/$datedprefix.sha256sum" ) \
| tee "$expdir/$datedprefix.car" \
| zstd -T0 --long -15 > "$expdir/$datedprefix.car.zst"
travisperson commented 2 years ago

Awesome this should be really easy to add then.

ribasushi commented 1 year ago

This can be closed as it has been resolved by https://github.com/filecoin-project/filecoin-chain-archiver/pull/50 However see request for significant improvement at https://github.com/filecoin-project/filecoin-chain-archiver/issues/55