search

filecoin-project / filecoin-chain-archiver

Filecoin snapshot / chain export software
Other
5 stars 3 forks source link

Suboptimal zstd compression #55

Closed ribasushi closed 1 year ago

ribasushi commented 1 year ago

The zstd compressor used here has unfortunate defaults / might not be the best compressor. It produces significantly larger files:

~$ curl -LI https://snapshots.mainnet.filops.net/minimal/latest.zst
content-length: 57102403792
content-disposition: attachment; filename="2382480_2022_11_30T04_00_00Z.car.zst"

vs the original zstd -T0 --long -15

~$ curl -LI https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car.zst 
Content-Length: 37840040354
x-amz-website-redirect-location: https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_2382481_2022-11-30_04-00-30.car.zst

cc @magik6k

ribasushi commented 1 year ago

@travisperson ping on this? Snapshots are still way larger than they could be.

ognots commented 1 year ago

We're using the maximum compression currently implemented in the zstd lib