search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.61k stars 785 forks source link

feat: Add option TIKTOKEN_FORCE_CACHE #170

Open Kontrolix opened 1 year ago

Kontrolix commented 1 year ago

Add taking into account the env var TIKTOKEN_FORCE_CACHE. When TIKTOKEN_FORCE_CACHE is set to "1", tiktoken will read BPE files only from the local cache.

It allows us to have control over when BPE files are downloaded from the internet.