search

apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
6.36k stars 2.2k forks source link

Support ZSTD compression codec for Avro files #3206

Closed kekwan closed 1 month ago

kekwan commented 3 years ago

Currently, only SNAPPY and GZIP compression codec are supported for Avro files. Are there plans to support ZSTD? It is several times faster than GZIP.

https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/avro/Avro.java#L77

amogh-jahagirdar commented 2 years ago

I am happy to look into raising a PR for support ZSTD Compression for Avro. We should define a sane default compression level and should allow configurable compression level options as well. What are your thoughts?

@jackye1995 @rdblue @kekwan

rdblue commented 2 years ago

Yes, Zstd for Avro would be great if Avro supports it. I'd go with whatever the default is in the Zstd library.

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

github-actions[bot] commented 1 month ago

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'