Add :streaming option to DataFrame.to_csv/3

elixir-explorer / explorer

Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir

https://hexdocs.pm/explorer

MIT License

1.13k stars 123 forks source link

Add :streaming option to DataFrame.to_csv/3 #888

Closed ryancurtin closed 8 months ago

ryancurtin commented 8 months ago

I have a use case where I would like to stream a large lazy DataFrame to a CSV without having to load the entire dataset in memory, and I was looking to implement a streaming option to the to_csv/3 function, similar to DataFrame.to_parquet/3.

I've already been able to make this work in a fork using the Polars sink_csv function under the hood. Please let me know what you think and I will submit my PR. Thanks!

josevalim commented 8 months ago

Yes, a pull request would be welcome to any of our API‘s that do not support streaming yet