lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
https://lancedb.github.io/lance/
Apache License 2.0
3.97k stars 224 forks source link

chore: lance-encoding test utility now properly aligns buffers #3116

Closed westonpace closed 1 week ago

westonpace commented 1 week ago

We align buffers in the file writer but we were not doing the same thing in the test utility. This forced encodings to do extra copies. We remove one such copy in this PR.

Closes #3115

codecov-commenter commented 1 week ago

Codecov Report

Attention: Patch coverage is 66.66667% with 5 lines in your changes missing coverage. Please review.

Project coverage is 77.13%. Comparing base (c237bcb) to head (b5117b9).

Files with missing lines Patch % Lines
rust/lance-encoding/src/encoder.rs 16.66% 5 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3116 +/- ## ========================================== + Coverage 77.11% 77.13% +0.01% ========================================== Files 240 240 Lines 80643 80652 +9 Branches 80643 80652 +9 ========================================== + Hits 62190 62213 +23 + Misses 15275 15274 -1 + Partials 3178 3165 -13 ``` | [Flag](https://app.codecov.io/gh/lancedb/lance/pull/3116/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/lancedb/lance/pull/3116/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | `77.13% <66.66%> (+0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

broccoliSpicy commented 1 week ago

some benchmark results related to this PR: before eliminating copy:

Screenshot 2024-11-11 at 2 55 14 PM

after eliminating copy:

Screenshot 2024-11-11 at 2 53 20 PM