I have added a streaming option for the to_csv/3 that uses the Polars sink_csv function under the hood. Polars does not have a sink_csv_cloud function similar to sink_parquet_cloud, so for now, I implemented it only for a LazyFrame using a local entry.
Notes
As far as my implementation goes, I've kept the DataFrame API clean since I'm really just adding an option, but it does beg the question of how we'd want to expose some other Polars serialization options for users of the to_csv function. For now I won't worry about that, though.
philss
commented
7 months ago
Closes #888
Description
I have added a streaming option for the
to_csv/3that uses the Polarssink_csvfunction under the hood. Polars does not have asink_csv_cloudfunction similar tosink_parquet_cloud, so for now, I implemented it only for a LazyFrame using a local entry.Notes
As far as my implementation goes, I've kept the DataFrame API clean since I'm really just adding an option, but it does beg the question of how we'd want to expose some other Polars serialization options for users of the
to_csvfunction. For now I won't worry about that, though.