openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.06k stars 749 forks source link

ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken' #273

Open sazirod opened 3 months ago

sazirod commented 3 months ago

I am using a freshly installed version of tiktoken==0.6.0

I am using the latest version of Anaconda with python 3.12.2 and trying to:

import tiktoken

and getting:

ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken'



Cell In[4], line 1
----> 1 import tiktoken
      3 enc = tiktoken.get_encoding("cl100k_base")
      4 assert enc.decode(enc.encode("hello world")) == "hello world"

File c:\users/zzz/appdata/local/programs/python/python312/lib/site-packages\tiktoken\__init__.py:2
      1 # This is the public API of tiktoken
----> 2 from .core import Encoding as Encoding
      3 from .model import encoding_for_model as encoding_for_model
      4 from .model import encoding_name_for_model as encoding_name_for_model

File c:\users/zzz/appdata/local/programs/python/python312/lib/site-packages\tiktoken\core.py:9
      5 from typing import AbstractSet, Collection, Literal, NoReturn, Optional, Union
      7 import regex
----> 9 from tiktoken import _tiktoken
     12 class Encoding:
     13     def __init__(
     14         self,
     15         name: str,
   (...)
     20         explicit_n_vocab: Optional[int] = None,
     21     ):

ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken' (most likely due to a circular import) (c:\users/zzz/appdata/local/programs/python/python312/lib/site-packages\tiktoken\__init__.py)```