[pixels-core] add support for general data compression

pixelsdb / pixels

An efficient storage and compute engine for both on-prem and cloud-native data analytics.

GNU Affero General Public License v3.0

133 stars 34 forks source link

[pixels-core] add support for general data compression #16

Open ray6080 opened 5 years ago

ray6080 commented 5 years ago

Although we have implemented basic encodings (like RLE) in pixels, it's also common to use general compressions (like gzip) along with encoding in columnar storage. However, we haven't implemented this yet.

This is a highly likely feature in the near future.

ray6080 commented 5 years ago

Shall we consider lz4 and zstd? lz4 has a very fast decompression speed (but lower compression ratio) while zstd has a very high compression ratio and reasonable compression/decompression speed. They are both under BSD License and supported in Parquet.

Yes, you're right. We should look into these two first when compression is on our plan table.