Cannot import tiktoken need Pyo3 migration

[simondpalmer]

Which version of tiktoken is migrated to the newer version of PyO3? migrating to new pyO3(I am running tiktoken fine locally but when I use it on PythonAnywhere I get the following error)

Traceback (most recent call last):
  File "/home/simonpalmer/ideally/flask_api.py", line 11, in <module>
    import tiktoken
  File "/home/simonpalmer/.local/lib/python3.10/site-packages/tiktoken/__init__.py", line 1, in <module>
    from .core import Encoding as Encoding
  File "/home/simonpalmer/.local/lib/python3.10/site-packages/tiktoken/core.py", line 9, in <module>
    from tiktoken import _tiktoken
ImportError: PyO3 modules may only be initialized once per interpreter process

[hauntsaninja]

All open source releases of tiktoken have used PyO3 0.17. If you can find a local repro, this will be easier to resolve

[simondpalmer]

@hauntsaninja I'm not sure I understand a local "repro"?

[hauntsaninja]

A way for me to reproduce the problem that isn't specific to pythonanywhere.com

[simondpalmer]

Oh okay thanks I'll see what I can do

[hauntsaninja]

Feel free to re-open if you have a reproducer

[phipsi71]

Got the same error when importing flask_jwt_extended, but only when running in PROD with uWSGI: (In Dev on vscode, I got no errors).

....
Traceback (most recent call last):
  File "main.py", line 11, in <module>
    from flask_jwt_extended import JWTManager
  File "/usr/local/lib/python3.11/site-packages/flask_jwt_extended/__init__.py", line 1, in <module>
    from .jwt_manager import JWTManager
  File "/usr/local/lib/python3.11/site-packages/flask_jwt_extended/jwt_manager.py", line 6, in <module>
    import jwt
  File "/usr/local/lib/python3.11/site-packages/jwt/__init__.py", line 1, in <module>
    from .api_jwk import PyJWK, PyJWKSet
  File "/usr/local/lib/python3.11/site-packages/jwt/api_jwk.py", line 7, in <module>
    from .algorithms import get_default_algorithms, has_crypto, requires_cryptography
  File "/usr/local/lib/python3.11/site-packages/jwt/algorithms.py", line 12, in <module>
    from .utils import (
  File "/usr/local/lib/python3.11/site-packages/jwt/utils.py", line 7, in <module>
    from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
  File "/usr/local/lib64/python3.11/site-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in <module>
    from cryptography.hazmat._oid import ObjectIdentifier
  File "/usr/local/lib64/python3.11/site-packages/cryptography/hazmat/_oid.py", line 9, in <module>
    from cryptography.hazmat.bindings._rust import (
**ImportError: PyO3 modules may only be initialized once per interpreter process**

[simondpalmer]

I resolved this by going to a different remote server provider (PythonAnywhere to Railway)