search

pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
27.72k stars 1.7k forks source link

[Question] Any way to control the compression level when writing Parquet files? #3473

Open dclong opened 2 years ago

dclong commented 2 years ago

https://stackoverflow.com/questions/72340767/control-the-compression-level-when-write-parquet-files-using-polars-in-rust

ritchie46 commented 2 years ago

This is not possible in rust polars. You could make a feature request for this upstream in arrow2.

If you want that functionality in python polars you can leverage pyarrow for this purpose. polars has zero copy interop with pyarrow.

jorgecarleitao commented 2 years ago

We have this already in latest parquet2 main, it will soon be available in arrow2 - needs a backward incompatible release