search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
2.3k stars 404 forks source link

Optimize z-order doesn't work if contains struct column with large/normal version of it #1927

Closed ion-elgreco closed 2 months ago

ion-elgreco commented 11 months ago

Z-order read doesn't work when table has mixed large and normal arrow types in multiple files. This is because arrow-compute can_cast_types does not yet support structs.

https://github.com/apache/arrow-datafusion/issues/8372#issuecomment-1833682591

https://github.com/delta-io/delta-rs/actions/runs/7046215574/job/19177551302#step:9:697

ion-elgreco commented 11 months ago

Created also an issue in arrow-rs: https://github.com/apache/arrow-rs/issues/5151#issuecomment-1833891822

ion-elgreco commented 2 months ago

Not relevant anymore since we read the data now in the correct schema casted