openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.48k stars 856 forks source link

AttributeError: partially initialized module 'tiktoken' has no attribute 'get_encoding #257

Closed TimotejZavski closed 9 months ago

TimotejZavski commented 9 months ago

I just copied code from readme:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4")

and it gave me this error:

(base) xxxxx@xxx-MacBook-Pro desktop % python ger.py
Traceback (most recent call last):
  File "/Users/xxxxx/Desktop/ger.py", line 1, in <module>
    import tiktoken
  File "/Users/xxxxx/miniconda3/lib/python3.11/site-packages/tiktoken/__init__.py", line 2, in <module>
    from .core import Encoding as Encoding
  File "/Users/xxxxx/miniconda3/lib/python3.11/site-packages/tiktoken/core.py", line 4, in <module>
    from concurrent.futures import ThreadPoolExecutor
  File "/Users/xxxxx/miniconda3/lib/python3.11/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "/Users/xxxxx/miniconda3/lib/python3.11/concurrent/futures/_base.py", line 7, in <module>
    import logging
  File "/Users/xxxxx/miniconda3/lib/python3.11/logging/__init__.py", line 26, in <module>
    import sys, os, time, io, re, traceback, warnings, weakref, collections.abc
  File "/Users/xxxxx/miniconda3/lib/python3.11/traceback.py", line 5, in <module>
    import linecache
  File "/Users/xxxxx/miniconda3/lib/python3.11/linecache.py", line 11, in <module>
    import tokenize
  File "/Users/xxxxx/miniconda3/lib/python3.11/tokenize.py", line 35, in <module>
    from token import *
  File "/Users/xxxxx/Desktop/token.py", line 639, in <module>
    print(num_tokens_from_string(ger, "cl100k_base"))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxxxx/Desktop/token.py", line 4, in num_tokens_from_string
    encoding = tiktoken.get_encoding(encoding_name)
               ^^^^^^^^^^^^^^^^^^^^^
AttributeError: partially initialized module 'tiktoken' has no attribute 'get_encoding' (most likely due to a circular import)

I installed tiktoken before with pip..

hauntsaninja commented 9 months ago

Your file named token.py is clobbering the standard library module named token. You can see this by reading the stacktrace.

Coincidentally, I have a PR open against CPython that will make the Python interpreter itself provide a better error message in such cases: https://github.com/python/cpython/pull/113769