chore: upgrade to Datafusion 38

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python

https://delta-io.github.io/delta-rs/

Apache License 2.0

1.98k stars 365 forks source link

chore: upgrade to Datafusion 38 #2499

Closed rtyler closed 3 weeks ago

ion-elgreco commented 1 month ago

With v38, I think we can revert this temp fix: https://github.com/delta-io/delta-rs/pull/2395/files#diff-60c94c425cf655602ada7636a35fb8e25391bc92f5b9e35462562604449be324R1055

echai58 commented 1 month ago

datafusion 38 should also close this issue: https://github.com/delta-io/delta-rs/issues/2188 with https://github.com/apache/datafusion/pull/10095

abhiaagarwal commented 3 weeks ago

It could be nice to skip directly to DataFusion 39, which just dropped an RC — it depends on Arrow 52, which released a few days ago, and pyo3 0.21.1, meaning all three can be bumped together :) working on that PR rn