search

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
https://lancedb.github.io/lance/
Apache License 2.0
3.97k stars 224 forks source link

perf: optimize reading transactions in commit loop #3117

Closed wjones127 closed 12 hours ago

wjones127 commented 1 week ago

Closes #3057

codecov-commenter commented 1 week ago

Codecov Report

Attention: Patch coverage is 78.35294% with 92 lines in your changes missing coverage. Please review.

Project coverage is 77.87%. Comparing base (566082f) to head (4203b71). Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
rust/lance/src/io/commit.rs 80.46% 12 Missing and 13 partials :warning:
rust/lance/src/utils/test.rs 25.00% 21 Missing :warning:
rust/lance/src/dataset.rs 63.26% 14 Missing and 4 partials :warning:
rust/lance/src/dataset/transaction.rs 20.00% 6 Missing and 2 partials :warning:
rust/lance-table/src/format/fragment.rs 0.00% 2 Missing and 4 partials :warning:
rust/lance/src/dataset/write/commit.rs 96.55% 0 Missing and 5 partials :warning:
rust/lance/src/index.rs 57.14% 2 Missing and 1 partial :warning:
rust/lance-table/src/io/commit.rs 71.42% 1 Missing and 1 partial :warning:
rust/lance-io/src/object_store.rs 94.11% 0 Missing and 1 partial :warning:
...ust/lance-table/src/io/commit/external_manifest.rs 91.66% 0 Missing and 1 partial :warning:
... and 2 more
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3117 +/- ## ========================================== - Coverage 77.92% 77.87% -0.06% ========================================== Files 242 242 Lines 81738 82015 +277 Branches 81738 82015 +277 ========================================== + Hits 63695 63869 +174 - Misses 14861 14940 +79 - Partials 3182 3206 +24 ``` | [Flag](https://app.codecov.io/gh/lancedb/lance/pull/3117/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/lancedb/lance/pull/3117/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | `77.87% <78.35%> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features: