search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.76k stars 801 forks source link

Cannot import tiktoken into flask app - PyO3 modules may only be initialized once per interpreter process #138

Closed simondpalmer closed 1 year ago

simondpalmer commented 1 year ago

I can load tiktoken locally with flask but once I try to use it on PythonAnywhere in production it errors with. Both are using Python 3.10

Traceback (most recent call last):
  File "/home/simonpalmer/ideally/flask_api.py", line 11, in <module>
    import tiktoken
  File "/home/simonpalmer/.local/lib/python3.10/site-packages/tiktoken/__init__.py", line 1, in <module>
    from .core import Encoding as Encoding
  File "/home/simonpalmer/.local/lib/python3.10/site-packages/tiktoken/core.py", line 9, in <module>
    from tiktoken import _tiktoken
ImportError: PyO3 modules may only be initialized once per interpreter process

Which version of tiktoken has migrated to Py03?

hauntsaninja commented 1 year ago

This is the same issue you opened here: https://github.com/openai/tiktoken/issues/126 Please discuss there. In particular, create a reproducer.