search

OoriData / OgbujiPT

Client-side toolkit for using large language models, including where self-hosted
Apache License 2.0
103 stars 8 forks source link

Token splitter functions #84

Open uogbuji opened 4 months ago

uogbuji commented 4 months ago

Following on from muddled notes in #30, Create a token-aware text_helper.token_splitter() class which works at the tokenizer level.