search

apache / parquet-java

Apache Parquet Java
https://parquet.apache.org/
Apache License 2.0
2.56k stars 1.4k forks source link

Provide pluggable APIs to support user customized compression codec #2463

Open asfimport opened 4 years ago

asfimport commented 4 years ago

In demand of better performance, quite some end users want to leverage accelerators (e.g. FPGA, Intel QAT) to offload compression computation. Parquet, as a well adopted data format, should allow compression codec implementation customization via a pluggable mechanism for those standard compression codec. This JIRA is intended for introducing this set of APIs.

Reporter: XinDong / @XinDongSh

PRs and other links:

Note: This issue was originally created as PARQUET-1804. Please see the migration documentation for further details.

asfimport commented 9 months ago

Atour Mousavi Gourabi / @amousavigourabi: After PARQUET-2385 and PARQUET-2282, there is now support for using custom CodecFactory implementations. This way users could implement and use their own (hardware accelerated) codecs while reading and writing Parquet.