Open Limess opened 8 months ago
Redshift now supports zstd compression for COPY operations from s3 (https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-file-compression.html#copy-zstd).
ZSTD is considerably cheaper to compress/decompress for the same compression.
A few python implementations are readily available:
Python (full) | Gregory Szorc | https://pypi.python.org/pypi/zstandard Python (simple) | Sergey Dryabzhinsky | https://github.com/sergey-dryabzhinsky/python-zstd Python (bz2 api) | Ma Lin | https://pypi.org/project/pyzstd
Redshift now supports zstd compression for COPY operations from s3 (https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-file-compression.html#copy-zstd).
ZSTD is considerably cheaper to compress/decompress for the same compression.
A few python implementations are readily available:
Python (full) | Gregory Szorc | https://pypi.python.org/pypi/zstandard Python (simple) | Sergey Dryabzhinsky | https://github.com/sergey-dryabzhinsky/python-zstd Python (bz2 api) | Ma Lin | https://pypi.org/project/pyzstd