search

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
https://lancedb.github.io/lance/
Apache License 2.0
3.97k stars 224 forks source link

perf: transpose the PQ codes to improve search performance #3120

Closed BubbleCal closed 1 week ago

BubbleCal commented 1 week ago

30% improved:

5242880,L2,PQ=96,DIM=1536
                        time:   [161.24 ms 162.71 ms 164.08 ms]
                        change: [-30.390% -29.789% -29.192%] (p = 0.00 < 0.10)
                        Performance has improved.

Benchmarking 5242880,Cosine,PQ=96,DIM=1536: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 9.2s or enable flat sampling.
5242880,Cosine,PQ=96,DIM=1536
                        time:   [165.53 ms 167.12 ms 168.49 ms]
                        change: [-30.725% -30.233% -29.772%] (p = 0.00 < 0.10)
                        Performance has improved.
codecov-commenter commented 1 week ago

Codecov Report

Attention: Patch coverage is 97.08738% with 6 lines in your changes missing coverage. Please review.

Project coverage is 77.13%. Comparing base (4ab270b) to head (b5302f7).

Files with missing lines Patch % Lines
rust/lance-index/src/vector/pq/storage.rs 94.82% 1 Missing and 2 partials :warning:
rust/lance/src/index/vector/builder.rs 85.71% 0 Missing and 1 partial :warning:
rust/lance/src/index/vector/ivf.rs 83.33% 0 Missing and 1 partial :warning:
rust/lance/src/index/vector/pq.rs 98.03% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3120 +/- ## ========================================== + Coverage 77.12% 77.13% +0.01% ========================================== Files 240 240 Lines 80652 80759 +107 Branches 80652 80759 +107 ========================================== + Hits 62203 62296 +93 Misses 15279 15279 - Partials 3170 3184 +14 ``` | [Flag](https://app.codecov.io/gh/lancedb/lance/pull/3120/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/lancedb/lance/pull/3120/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | `77.13% <97.08%> (+0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.