search

benbrandt / text-splitter

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.
MIT License
235 stars 15 forks source link

Initial CodeSplitter #176

Closed benbrandt closed 3 months ago

benbrandt commented 3 months ago

Still in a test module for now to figure out which internal APIs need to change to support this.

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 99.57687% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 99.74%. Comparing base (a416fa9) to head (7191700).

Files Patch % Lines
src/splitter.rs 99.38% 2 Missing :warning:
src/splitter/text.rs 98.61% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #176 +/- ## ========================================== - Coverage 99.78% 99.74% -0.04% ========================================== Files 9 10 +1 Lines 1858 1988 +130 ========================================== + Hits 1854 1983 +129 - Misses 4 5 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.