delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python

https://delta-io.github.io/delta-rs/

Apache License 2.0

1.98k stars 365 forks source link

fix(python): reuse state in `to_pyarrow_dataset` #2485

Closed ion-elgreco closed 1 month ago

ion-elgreco commented 1 month ago

Description

Reuse the state so we save time instead of reconstructing and verifying the state.

adriangb commented 1 month ago

Yup this fixes the issue 🥳

adriangb commented 1 month ago

@ion-elgreco looks like it needs a rebase. Also just curious if you'll be making a release with this fix since it's such a huge performance win for an essential feature?

ion-elgreco commented 1 month ago

@ion-elgreco looks like it needs a rebase. Also just curious if you'll be making a release with this fix since it's such a huge performance win for an essential feature?

@wjones127 could you take a look at the PR?