unknown encoding

[yangwendy]

pip install tiktoken in python 3.10

import tiktoken
enc = tiktoken.encoding_for_model("text-davinci-003")

Report error: ValueError: Unknown encoding p50k_base

assert ENCODING_CONSTRUCTORS is not None 59 if encoding_name not in ENCODING_CONSTRUCTORS: ---> 60 raise ValueError(f"Unknown encoding {encoding_name}")

[hauntsaninja]

This is not enough information to reproduce the problem. Could you run these commands and paste the full output:

python --version
python -c 'import platform; print(platform.platform())'
python -m venv env
source env/bin/activate
env/bin/python -m pip install wheel
env/bin/python -m pip install tiktoken
env/bin/python -c 'import tiktoken; print(tiktoken.get_encoding("gpt2"))'
env/bin/python -c 'import site; import os; print(os.listdir(site.getsitepackages()[0]))'

[eddir]

Happens to me too when I try to pack my application into exe through pyinstaller. Seems like module tiktoken_ext is unavailable after packaging. I tried to add this module but I got another error: "FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\eddir\PycharmProjects\Deus\output\main\blobfile\VERSION'" .

PyInstaller args: pyinstaller --noconfirm --onedir --windowed --collect-all "tiktoken_ext" "C:/Users/eddir/PycharmProjects/Deus/main.py"

Looks like the importing system for encoding is a little bit confuse.

[hauntsaninja]

Thanks, there's another issue in which people are talking about pyinstaller: https://github.com/openai/tiktoken/issues/43

If anyone has an issue that does not involve pyinstaller, please run the commands in https://github.com/openai/tiktoken/issues/51#issuecomment-1455490628 and paste the full log

[hauntsaninja]

OP hasn't responded, so closing. #43 is the right issue to talk about pyinstaller in

[rishabhgupta93]

I am getting similar issue while loading the encoding:

The code snippet is as follows:

_import tiktoken from llama_index.callbacks import CallbackManager, TokenCountingHandler enc = tiktoken.get_encoding("WhereIsAI/UAE-Large-V1") tokencounter = TokenCountingHandler(tokenizer= enc.encode)

and the error i am getting is as follows:

_--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[20], line 3 1 import tiktoken 2 from llama_index.callbacks import CallbackManager, TokenCountingHandler ----> 3 enc = tiktoken.get_encoding("WhereIsAI/UAE-Large-V1") 4 token_counter = TokenCountingHandler(tokenizer= enc.encode)

File f:\pycharmprojects\llamaindex\venv\lib\site-packages\tiktoken\registry.py:68, in get_encoding(encoding_name) 65 assert ENCODING_CONSTRUCTORS is not None 67 if encoding_name not in ENCODING_CONSTRUCTORS: ---> 68 raise ValueError( 69 f"Unknown encoding {encoding_name}. Plugins found: {_available_plugin_modules()}" 70 ) 72 constructor = ENCODING_CONSTRUCTORS[encoding_name] 73 enc = Encoding(**constructor())

ValueError: Unknown encoding WhereIsAI/UAE-Large-V1. Plugins found: ['tiktoken_ext.openaipublic']