lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
https://lancedb.github.io/lance/
Apache License 2.0
3.45k stars 180 forks source link

chore: use rayon in pq transform and remove async calls #2343

Closed eddyxu closed 4 weeks ago

eddyxu commented 4 weeks ago

one nit + would you happen to have benchmark numbers? did not find a handy benchmark for PQ::transform yet, could be next time.

codecov-commenter commented 4 weeks ago

Codecov Report

Attention: Patch coverage is 94.33962% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 80.79%. Comparing base (4cb0a2b) to head (0e374af).

Files Patch % Lines
rust/lance-linalg/src/kmeans.rs 81.81% 4 Missing :warning:
rust/lance-index/src/vector/pq.rs 98.76% 0 Missing and 1 partial :warning:
rust/lance-index/src/vector/pq/transform.rs 0.00% 0 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2343 +/- ## ========================================== + Coverage 80.78% 80.79% +0.01% ========================================== Files 192 192 Lines 56189 56213 +24 Branches 56189 56213 +24 ========================================== + Hits 45392 45420 +28 - Misses 8128 8129 +1 + Partials 2669 2664 -5 ``` | [Flag](https://app.codecov.io/gh/lancedb/lance/pull/2343/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/lancedb/lance/pull/2343/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | `80.79% <94.33%> (+0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.