lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
https://lancedb.github.io/lance/
Apache License 2.0
3.97k stars 224 forks source link

chore: cleanup decoder benches #3118

Closed westonpace closed 3 days ago

westonpace commented 1 week ago

This ensures that the setup routines happen inside of the benchmark function. Otherwise datagen will be run for all benchmarks during the collection phase and the system will run out of RAM.

This also changes the routines to use FileMetadataCache::no_cache as it turns out moka doesn't immediately free its cache space on drop and, when used in a tight benchmark loop, this leads to running out of memory.

codecov-commenter commented 1 week ago

Codecov Report

Attention: Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.

Project coverage is 77.88%. Comparing base (f257489) to head (7cdf3b8). Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
rust/lance-encoding-datafusion/src/zone.rs 0.00% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3118 +/- ## ========================================== - Coverage 77.91% 77.88% -0.03% ========================================== Files 240 240 Lines 81564 81465 -99 Branches 81564 81465 -99 ========================================== - Hits 63550 63452 -98 - Misses 14806 14819 +13 + Partials 3208 3194 -14 ``` | [Flag](https://app.codecov.io/gh/lancedb/lance/pull/3118/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/lancedb/lance/pull/3118/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb) | `77.88% <81.81%> (-0.03%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lancedb#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

broccoliSpicy commented 1 week ago

related to #2725