openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.17k stars 751 forks source link

0) Add the jtokkit test suite examples to validate the cl100k_base, p50k_base & r50k_base encodings #237

Open paplorinc opened 6 months ago

paplorinc commented 6 months ago

See:

To make it simpler and smarter to input the parameters, instead of the CSV inputs I've pushed them as code.

I've split out the tests from the optimization work I've started here, see: