apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine

https://datafusion.apache.org/ballista

Apache License 2.0

1.39k stars 181 forks source link

Fix regression in DataFrame.write_xxx #945

Closed andygrove closed 6 months ago

andygrove commented 6 months ago

Which issue does this PR close?

Closes https://github.com/apache/arrow-ballista/issues/894

Status

[x] Add failing test
[x] Upgrade to DataFusion version that supports CopyTo serde (https://github.com/apache/arrow-datafusion/pull/8618)
[x] Upgrade to DataFusion version that supports CopyTo serde with support for WriterProperties (https://github.com/apache/arrow-datafusion/pull/8627 and https://github.com/apache/arrow-datafusion/pull/8641)
[x] Upgrade to DataFusion version that supports serde for Csv and Parquet FileSinkExec (https://github.com/apache/arrow-datafusion/pull/8646)
[x] Test passes
[x] Upgrade to DataFusion main branch version
[x] Clippy fixes for latest Rust version

Rationale for this change

Fix a major regression where we can no longer save the results of a query.

What changes are included in this PR?

Upgrade to more recent DataFusion
Add test

Are there any user-facing changes?