add proper tokenizer - current estimate is causing problems with non-ascii characters

josephrocca / OpenCharacters

Simple little web interface for creating characters and chatting with them. It's basically a single HTML file - no server. Share characters using a link (character data is stored within the URL itself). All chat data is stored in your browser using IndexedDB. Currently supports OpenAI APIs and ~any Hugging Face model.

https://josephrocca.github.io/OpenCharacters

MIT License

362 stars 59 forks source link

add proper tokenizer - current estimate is causing problems with non-ascii characters #18

Open josephrocca opened 1 year ago

josephrocca commented 1 year ago

maybe port/compile tiktoken to js (via tiktoken-rs to wasm, or tiktoken-py with pyodide)

josephrocca commented 1 year ago

Kinda fixed this by just using a GPT-3 tokenizer for all models - but not sure how close this'll be for GPT-4 - need to look into it. A close estimate is all we really need for now