search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
2.34k stars 414 forks source link

feat: provide direct TableProvider integration in datafusion-python #3012

Closed timsaucer closed 1 day ago

timsaucer commented 2 days ago

Description

This change adds in a single method on the Delta Table object to expose a PyCapsule that provides a DataFusion FFI Table Provider. With this change, you can register a delta table in datafusion-python without the need to export a pyarrow_dataset. This enables full push down filter support, greatly improving performance in some cases.

Related Issue(s)

Closes #2536

Documentation

This is a follow on from https://github.com/apache/datafusion/pull/12920 and https://github.com/apache/datafusion-python/issues/823

github-actions[bot] commented 2 days ago

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

timsaucer commented 2 days ago

FYI @ion-elgreco

ion-elgreco commented 2 days ago

@timsaucer super excited with this change!

ion-elgreco commented 2 days ago

Can you add the attribute in here as well:

https://github.com/delta-io/delta-rs/blob/main/python/deltalake/_internal.pyi

codecov[bot] commented 2 days ago

Codecov Report

Attention: Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.

Project coverage is 72.43%. Comparing base (c091a82) to head (2d7fdb8). Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
python/src/lib.rs 0.00% 10 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3012 +/- ## ========================================== - Coverage 72.47% 72.43% -0.04% ========================================== Files 128 128 Lines 40831 40841 +10 Branches 40831 40841 +10 ========================================== - Hits 29592 29584 -8 - Misses 9356 9366 +10 - Partials 1883 1891 +8 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features:

timsaucer commented 2 days ago

I'll try to address that ci failure tomorrow morning

adriangb commented 1 day ago

Amazing!