search

commoncrawl / cc-index-table

Index Common Crawl archives in tabular format
Apache License 2.0
106 stars 9 forks source link

Explore Zstandard compression #11

Open sebastian-nagel opened 2 years ago

sebastian-nagel commented 2 years ago

(depends on #10)

Zstandard compression is directly integrated into Parquet (PARQUET-1866). Also Athena now supports Zstd as Parquet compression. Time to explore whether switching from gzip to zstd brings improvements in storage size or query processing speed. In addition to Athena and Spark, support for zstd should be verified also Hive, Arrow, Pandas and other common tools to read/process Parquet files.

sebastian-nagel commented 2 years ago

By passing --outputCompression=zstd the Parquet files are compressed using Zstandard compression. However, the zstd-compressed files are slightly bigger compared to gzip. There is no effect on the Parquet file size visible by adjusting the zstd compression level (eg. --conf spark.io.compression.zstd.level=22) - this option might only affect other output formats.