search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.52k stars 856 forks source link

Using offline: `.tiktoken` file gets deleted automatically on Linux #279

Closed nkilm closed 3 months ago

nkilm commented 7 months ago

Description

I'm trying to setup tiktoken locally with the help of this guide - https://stackoverflow.com/questions/76106366/how-to-use-tiktoken-in-offline-mode-computer. But the problem I'm facing is that the .tiktoken file - cl100k_base.tiktoken is getting deleted automatically after code execution on Linux machine.

OS: Linux tiktoken version - 0.6.0

Yuxiang1995 commented 6 months ago

I meet the same question!

Yuxiang1995 commented 6 months ago

cl100k_base.tiktoken was updated, download the latest version can fix it

nkilm commented 6 months ago

@Yuxiang1995 Thanks! I'll checkout and let you know.

nkilm commented 3 months ago

Update: Latest version of .tiktoken fixes this issue.