[#30] text splitter enhancements

OoriData / OgbujiPT

Client-side toolkit for using large language models, including where self-hosted

Apache License 2.0

101 stars 8 forks source link

Closed uogbuji closed 3 months ago

uogbuji commented 3 months ago

Create text_helper.text_split_fuzzy() as basically a generator version of text_helper.text_splitter(), and deprecate the latter
Create text_helper.text_split() which brooks no overlap