delta-incubator / delta-kernel-rs

A native Delta implementation for integration with any query engine
Apache License 2.0
121 stars 32 forks source link

Ensure we return a sorted listing when using a local client #344

Open nicklan opened 1 week ago

nicklan commented 1 week ago

This ensures that calls to list_files are returned sorted if a LocalFileSystem ObjectStore is used.

It's not clean for a few reasons:

  1. We have to materialize the entire iter in order to sort it
  2. We can't easily detect if we're local just by using type_of or similar. We have a foreign trait object that doesn't have an as_any on it, so we can force the reference into an Any which would allow us to use is, and we can't add an implementation of something like as_any because dyn ObjectStore isn't Sized. So this resorts to formating the object at creation (since ObjectStore requires Display) and checking if it starts with LocalFileSystem....

Adds a test that when using local client things come back sorted now. Without these changes the test failed.

I think we should merge (or something like it) and then also see about adding a list_sorted and list_with_offset_sorted to ObjectStore

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 93.54839% with 4 lines in your changes missing coverage. Please review.

Project coverage is 74.12%. Comparing base (b3a3b7d) to head (1e03452).

Files with missing lines Patch % Lines
kernel/src/engine/default/filesystem.rs 91.83% 0 Missing and 4 partials :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #344 +/- ## ========================================== + Coverage 74.03% 74.12% +0.09% ========================================== Files 43 44 +1 Lines 8137 8183 +46 Branches 8137 8183 +46 ========================================== + Hits 6024 6066 +42 Misses 1733 1733 - Partials 380 384 +4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.