search

apache / iceberg-rust

Apache Iceberg
https://rust.iceberg.apache.org/
Apache License 2.0
646 stars 154 forks source link

feat: Support Parquet modular encryption #686

Open adamreeve opened 1 week ago

adamreeve commented 1 week ago

Hi

The Java Iceberg implementation is adding support for using native Parquet modular encryption, which is being developed as part of version 3 of the Iceberg specification: https://github.com/orgs/apache/projects/393/views/1.

This encrypts data in Parquet files, but also requires encryption of manifests and manifest lists due to these containing potentially sensitive data like column statistics.

I'm currently working with @rok on adding support for Parquet modular encryption to arrow-rs in https://github.com/apache/arrow-rs/issues/3511, which would be a prerequisite for supporting Parquet encryption in iceberg-rust.

We're considering using Iceberg for a project but would need encryption support. Is there any plan already to follow the Java version and add encryption support, or is this something that would be likely to be accepted if we contributed it?

liurenjie1024 commented 1 week ago

Hi, @adamreeve thanks for raising this! I'm not aware of encryption yet, and of course we are always welcoming to contributions!

adamreeve commented 1 week ago

OK great, we're not sure what our plans are yet but I'll update here if we decide to start working on this.