search

apache / iceberg-rust

Apache Iceberg
https://rust.iceberg.apache.org/
Apache License 2.0
673 stars 159 forks source link

feat: Support Parquet modular encryption #686

Open adamreeve opened 3 weeks ago

adamreeve commented 3 weeks ago

Hi

The Java Iceberg implementation is adding support for using native Parquet modular encryption, which is being developed as part of version 3 of the Iceberg specification: https://github.com/orgs/apache/projects/393/views/1.

This encrypts data in Parquet files, but also requires encryption of manifests and manifest lists due to these containing potentially sensitive data like column statistics.

I'm currently working with @rok on adding support for Parquet modular encryption to arrow-rs in https://github.com/apache/arrow-rs/issues/3511, which would be a prerequisite for supporting Parquet encryption in iceberg-rust.

We're considering using Iceberg for a project but would need encryption support. Is there any plan already to follow the Java version and add encryption support, or is this something that would be likely to be accepted if we contributed it?

liurenjie1024 commented 2 weeks ago

Hi, @adamreeve thanks for raising this! I'm not aware of encryption yet, and of course we are always welcoming to contributions!

adamreeve commented 2 weeks ago

OK great, we're not sure what our plans are yet but I'll update here if we decide to start working on this.