benbrandt / text-splitter

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.
MIT License
245 stars 15 forks source link

Semantic level cleanup #170

Closed benbrandt closed 4 months ago

benbrandt commented 4 months ago

Extracts common logic into more defined areas:

codecov[bot] commented 4 months ago

Codecov Report

Attention: Patch coverage is 99.46237% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 99.73%. Comparing base (5b66e38) to head (50c4f14). Report is 1 commits behind head on main.

Files Patch % Lines
src/text.rs 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #170 +/- ## ======================================= Coverage 99.72% 99.73% ======================================= Files 8 9 +1 Lines 1812 1863 +51 ======================================= + Hits 1807 1858 +51 Misses 5 5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.