ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken'

[onlychenchen]

I pulled the code and tried to run the use case: """ import tiktoken enc = tiktoken.get_encoding("cl100k_base") assert enc.decode(enc.encode("hello world")) == "hello world" enc = tiktoken.encoding_for_model("gpt-4") """ but error: File "D:\tiktoken-main\tiktoken\core.py", line 9, in from tiktoken import _tiktoken ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken'(most likely due to a circular import) how to fix this error? i need help, thanks!

[hauntsaninja]

Do you have some other file name tiktoken.py, e.g. like https://github.com/openai/tiktoken/issues/106? There isn't enough information to reproduce this error.

[akshat-g]

@hauntsaninja - Actually, I am facing the same issue. I am using marvin - https://pypi.org/project/marvin/ It uses tiktoken as a dependency. So it installes tiktoken as well. I am trying to use marvin on AWS lambda. When I invoke my lambda, error occurs: cannot import name '_tiktoken' from partially initialized module 'tiktoken' (most likely due to a circular import) (/opt/python/lib/python3.9/site-packages/tiktoken/__init__.py)

It's here in tiktoken/core.py

from __future__ import annotations

import functools
from concurrent.futures import ThreadPoolExecutor
from typing import AbstractSet, Collection, Literal, NoReturn, Optional, Union

import regex

from tiktoken import _tiktoken

I have no file names as tiktoken.py so probably that is not causing the issue. Also I am using openai library as well, but not sure if that is causing the issue.