search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.17k stars 751 forks source link

tiktoken not found even though its installed. #235

Closed markcam1 closed 6 months ago

markcam1 commented 6 months ago

I'm using tiktoken in a streamlit app that connects to openai.

Unfortunately, I get this error in the frontend.

Could not import tiktoken python package. This is needed in order to for OpenAIEmbeddings. Please install it with pip install tiktoken.

However, tiktoken is installed in the python env. See below.

$ pip list Package Version

.... streamlit 1.29.0 tenacity 8.2.3 tiktoken 0.5.2 toml 0.10.2 toolz 0.12.0 tornado 6.4 tqdm 4.66.1 typing_extensions 4.9.0 typing-inspect 0.9.0 tzdata 2023.4 tzlocal 5.2 urllib3 2.1.0 validators 0.22.0 watchdog 3.0.0 wikipedia 1.4.0 yarl 1.9.4 youtube-transcript-api 0.6.2 zipp 3.17.0

I also cannot import the module into a file, because I get a runtime error that the module couldn't be found.

This same functionality works fine when not using the streamlit app.

Does anyone know how to solve this issue?

hauntsaninja commented 6 months ago

Are you sure pip corresponds to the right environment? In general, if you run python script.py use python -m pip list to specifically check the environment corresponding to that Python.

Closing, since this is exceedingly unlikely to be an issue in tiktoken