Add modular parquet de-/encryption

brainslush commented 3 days ago

Currently there are efforts to implement parquet de-/encryption into arrow-rs: https://github.com/apache/arrow-rs/pull/6637

I suggest to extend the existing scan,read,sink and write interfaces to handle encrypted parquet.

I offer to implement it.

coastalwhite commented 3 days ago

I quite agree with most of what is being said in the arrow-r/s thread:

We are interested but unless someone steps up to drive this effort resource or time-wise, I am not sure we have the bandwidth at the moment.
Any implementation of this should use one of the pure Rut cryptography efforts (ring or RustCrypto).
I am not 100% aware of the standard surrounding this, but I think this should be a relatively transparent step, maybe in PageReader or just above.

If you want to take this up, I suggest you first made a PR with a rough draft.

rok commented 1 day ago

Glad to see this! https://github.com/apache/arrow-rs/pull/6637 will most likely be ring based. I'd be happy to help with reviews etc.

pola-rs / polars