search

apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
6.4k stars 1.21k forks source link

Support Configuring arrow_ipc::FileWriter via ArrowWriterOptions #8635

Open devinjdangelo opened 11 months ago

devinjdangelo commented 11 months ago

Is your feature request related to a problem or challenge?

ArrowWriterOptions is already a variant of FileTypeWriterOptions, but it does not yet contain any configurable options. Now that we support writing out arrow files #8608 we should add support for the options that arrow_ipc::FileWriter supports.

Describe the solution you'd like

Update ArrowWriterOptions to parse options and pass through to arrow_ipc::FileWriter.

Describe alternatives you've considered

We could also add session level configuration options, as we have for parquet.

Additional context

No response

devinjdangelo commented 11 months ago

Also filed https://github.com/apache/arrow-rs/issues/5236 which would enable completing this ticket in a similar way to parquet and csv.