search

egonSchiele / chisel

Open source writing app
https://chiseleditor.com
Other
38 stars 3 forks source link

Tokenizer #63

Open belladoreai opened 1 year ago

belladoreai commented 1 year ago

Hey, I happened to notice you started using my tokenizer: https://github.com/belladoreai/llama-tokenizer-js

I was happy to see you use it, but then I realized you are creating this on top of OpenAI's API. OpenAI uses different tokenization than LLaMA, so the token counts will not be the same. I think you need to use a different library, such as https://github.com/niieani/gpt-tokenizer

egonSchiele commented 1 year ago

Thank you! I was using yours to estimate, but an exact count would be better. I'll check it out.