search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12k stars 818 forks source link

ValueError: not enough values to unpack (expected 2, got 1) when tiktoken.get_encoding("cl100k_base") #330

Closed hzh12345678 closed 3 hours ago

hzh12345678 commented 2 months ago

As the title, ValueError: not enough values to unpack (expected 2, got 1) when tiktoken.get_encoding("cl100k_base")

Screenshot 2024-08-02 231714
hauntsaninja commented 3 hours ago

Hm, it looks like the vocabulary file is corrupted or something. Recent versions of tiktoken will run hash checks against downloaded vocabularies. If you can reproduce with latest tiktoken (0.8) please re-open the issue