search

pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
27.83k stars 1.7k forks source link

Rust API for editing metadata when creating Parquet and IPC/Arrow output #15921

Open thinkingfish opened 2 months ago

thinkingfish commented 2 months ago

Description

Currently there doesn't seem to be a way to add or modify metadata (as key-val map) either at the file or column level when exporting dataframe/series to parquet/arrow formats. This could be useful for preserving data lineage and key attributes during data transformation, as Polars gets used as part of a much more extensive data pipeline.

ritchie46 commented 2 months ago

Yeap, would accept a PR on this.

deanm0000 commented 2 months ago

Are we talking about doing this with polars natively?