search

benbrandt / text-splitter

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.
MIT License
212 stars 14 forks source link

Performance Optimization: Expanding binary search window #231

Closed benbrandt closed 3 weeks ago

benbrandt commented 4 weeks ago

Optimizes the binary search to not be too greedy when checking the size of sections.

Previously, I relied on the fact that we'd find a semantic level that didn't fit in the chunk size. However, it is quite likely there won't be one, and then the reliance on having this to scope the sections we are binary searching over means that in worst case, we are binary searching over the entire rest of the document, which might be large search area, and requires tokenizing huge chunks of text, which is the real bottleneck.

Now, we increase the search area by a factor of 2 each successful attempt, and also keep track of updating our starting low value for the binary search if we already know it is too low.

codecov[bot] commented 4 weeks ago

Codecov Report

Attention: Patch coverage is 98.26087% with 4 lines in your changes missing coverage. Please review.

Project coverage is 99.50%. Comparing base (b1b39d1) to head (26998f1).

:exclamation: Current head 26998f1 differs from pull request most recent head a38ded3

Please upload reports for the commit a38ded3 to get more accurate results.

Files Patch % Lines
src/splitter.rs 96.33% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #231 +/- ## ========================================== - Coverage 99.65% 99.50% -0.15% ========================================== Files 11 11 Lines 2047 2033 -14 ========================================== - Hits 2040 2023 -17 - Misses 7 10 +3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.