search

benbrandt / text-splitter

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.
MIT License
235 stars 15 forks source link

Make experimental CodeSplitter public #179

Closed benbrandt closed 3 months ago

benbrandt commented 3 months ago

This exposes CodeSplitter as Experimental. This allows me to test that it works from the outside, but also not promise too much in case I need to make a release in the near future, as this likely needs a lot of testing, and also there is still a lot to figure out with regards to the Python bindings.

Adds new snapshot tests, and also slims down the amount of snapshots used in order to have manageable testing times.

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 97.97980% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 99.65%. Comparing base (2abaf31) to head (14b21d3).

Files Patch % Lines
src/splitter/code.rs 97.61% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #179 +/- ## ========================================== - Coverage 99.74% 99.65% -0.10% ========================================== Files 10 10 Lines 1988 2036 +48 ========================================== + Hits 1983 2029 +46 - Misses 5 7 +2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.