search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.03k stars 748 forks source link

Using offline: `.tiktoken` file gets deleted automatically on Linux #279

Open nkilm opened 2 months ago

nkilm commented 2 months ago

Description

I'm trying to setup tiktoken locally with the help of this guide - https://stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer. But the problem I'm facing is that the .tiktoken file - cl100k_base.tiktoken is getting deleted automatically after code execution on Linux machine.

OS: Linux tiktoken version - 0.6.0

Yuxiang1995 commented 1 month ago

I meet the same question!

Yuxiang1995 commented 1 month ago

cl100k_base.tiktoken was updated, download the latest version can fix it

nkilm commented 1 month ago

@Yuxiang1995 Thanks! I'll checkout and let you know.