Closed pipiche38 closed 1 year ago
alex
commented
1 year ago Looks like the proximate cause is https://github.com/PyO3/pyo3/commit/78ba70d2b4cdae1228561700bab62da793801d18, which came with fact that cryptography 41 upgrades the version of pyo3 we use.
However, that doesn't tell us why zigpy is attempting to initialize the rust module multiple times. Unfortunately, I know nothing about zigpy, so hard to say.
github-actions[bot]
commented
1 year ago This issue has been waiting for a reporter response for 3 days. It will be auto-closed if no activity occurs in the next 5 days.
prazumovsky
commented
1 year ago We found the same issue in ceph mgr repository. When mgr process starts their modules (which are on python), it fails with the same error during pyOpenSsl init:
File "/usr/share/ceph/mgr/restful/__init__.py", line 1, in <module>
from .module import Module
File "/usr/share/ceph/mgr/restful/module.py", line 21, in <module>
from OpenSSL import crypto
File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import SSL, crypto
File "/usr/local/lib/python3.9/site-packages/OpenSSL/SSL.py", line 9, in <module>
from OpenSSL._util import (
File "/usr/local/lib/python3.9/site-packages/OpenSSL/_util.py", line 6, in <module>
from cryptography.hazmat.bindings.openssl.binding import Binding
File "/usr/local/lib64/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module>
from cryptography.exceptions import InternalError
File "/usr/local/lib64/python3.9/site-packages/cryptography/exceptions.py", line 9, in <module>
from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
ImportError: PyO3 modules may only be initialized once per interpreter processcryptography version is 41.0.1, pyOpenSsl version is 23.2.0.
pipiche38
commented
1 year ago Looks like the proximate cause is PyO3/pyo3@78ba70d, which came with fact that cryptography 41 upgrades the version of pyo3 we use.
However, that doesn't tell us why zigpy is attempting to initialize the rust module multiple times. Unfortunately, I know nothing about zigpy, so hard to say.
I don't think that is an issue on zigpy itself, it is most-likely the embedded python which is simply restarting the python application, might related to https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/guide/src/migration.md#each-pymodule-can-now-only-be-initialized-once-per-process
fjmnav-nudge
commented
1 year ago We upgraded today to 41.0.1 and had to revert it because we found the same problem. In this case the problem was with flask-jwt-extended library:
[INFO] INTERNALERROR> from flask_jwt_extended import get_current_user
[INFO] INTERNALERROR> File "/opt/atlassian/pipelines/agent/build/target/venv/build/cpython-3.10.5.final.0/lib/python3.10/site-packages/flask_jwt_extended/__init__.py", line 1, in <module>
[INFO] INTERNALERROR> from .jwt_manager import JWTManager as JWTManager
[INFO] INTERNALERROR> File "/opt/atlassian/pipelines/agent/build/target/venv/build/cpython-3.10.5.final.0/lib/python3.10/site-packages/flask_jwt_extended/jwt_manager.py", line 6, in <module>
[INFO] INTERNALERROR> import jwt
[INFO] INTERNALERROR> File "/opt/atlassian/pipelines/agent/build/target/venv/build/cpython-3.10.5.final.0/lib/python3.10/site-packages/jwt/__init__.py", line 1, in <module>
[INFO] INTERNALERROR> from .api_jwk import PyJWK, PyJWKSet
[INFO] INTERNALERROR> File "/opt/atlassian/pipelines/agent/build/target/venv/build/cpython-3.10.5.final.0/lib/python3.10/site-packages/jwt/api_jwk.py", line 7, in <module>
[INFO] INTERNALERROR> from .algorithms import get_default_algorithms, has_crypto, requires_cryptography
[INFO] INTERNALERROR> File "/opt/atlassian/pipelines/agent/build/target/venv/build/cpython-3.10.5.final.0/lib/python3.10/site-packages/jwt/algorithms.py", line 12, in <module>
[INFO] INTERNALERROR> from .utils import (
[INFO] INTERNALERROR> File "/opt/atlassian/pipelines/agent/build/target/venv/build/cpython-3.10.5.final.0/lib/python3.10/site-packages/jwt/utils.py", line 7, in <module>
[INFO] INTERNALERROR> from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
[INFO] INTERNALERROR> File "/opt/atlassian/pipelines/agent/build/target/venv/build/cpython-3.10.5.final.0/lib/python3.10/site-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in <module>
[INFO] INTERNALERROR> from cryptography.hazmat._oid import ObjectIdentifier
[INFO] INTERNALERROR> File "/opt/atlassian/pipelines/agent/build/target/venv/build/cpython-3.10.5.final.0/lib/python3.10/site-packages/cryptography/hazmat/_oid.py", line 9, in <module>
[INFO] INTERNALERROR> from cryptography.hazmat.bindings._rust import (
[INFO] INTERNALERROR> ImportError: PyO3 modules may only be initialized once per interpreter processHow are you deploying python such that you're initializing modules multiple times?
prazumovsky
commented
1 year ago from ceph side it's a bit complicated question :) we are running ceph-mgr process where an initialization of submodules happens. These submodules are written on python, probably ceph-mgr initializes cryptography and one of the submodule initializes pyopenssl which re-initializes cryptography for some reason.
Current WA is to decrease pyopenssl to March release (23.1.1) and cryptography to 40.0.2.
github-actions[bot]
commented
1 year ago This issue has been waiting for a reporter response for 3 days. It will be auto-closed if no activity occurs in the next 5 days.
reaperhulk
commented
1 year ago Is there a simple reproducer we can use to understand more about how this is occurring? Can we make this exception apepar with a single CPython invocation or does it require a more complex runtime with, e.g. subinterpreters.
Baggerone
commented
1 year ago FYI, I have a django app that uses djangosaml2. It now is broken and shows this error.
My apache conf file includes WSGIScriptAlias / /var/django/amp/wsgi.py
alex
commented
1 year ago Is any other particular mod_wsgi configuration required or present?
Baggerone
commented
1 year ago Is any other particular mod_wsgi configuration required or present?
Not that I can see.
In case it helps, this is my wsgi.py file.
import os, sys
sys.path.append('/var/django')
sys.path.append('/var/django/amp')
os.environ['DJANGO_SETTINGS_MODULE'] = 'amp.settings'
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()Ok, it looks like mod_wsgi uses sub-interpreters by default, https://modwsgi.readthedocs.io/en/latest/user-guides/processes-and-threading.html#python-sub-interpreters it makes sense to me that sub-interpreters would trigger this.
Unfortunately I don't think there's anything we can do ourselves here, I think this needs to become an issue on pyo3 to discuss what would be required to support sub-interpreters.
amswiatkowski
commented
1 year ago Same issue here, AWS Lambda with Python 3.8 runtime where cryptography 41.0.2 is imported fails to bootstrap.
2023-07-24T11:38:04.149+02:00 ModuleNotFoundError: No module named '_cffi_backend'
2023-07-24T11:38:04.149+02:00 thread '<unnamed>' panicked at 'Python API call failed', /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/err/mod.rs:790:5
2023-07-24T11:38:04.194+02:00 from jose import jwt
2023-07-24T11:38:04.194+02:00 File "/opt/python/jose/jwt.py", line 6, in <module>
2023-07-24T11:38:04.194+02:00 from jose import jws
2023-07-24T11:38:04.194+02:00 File "/opt/python/jose/jws.py", line 5, in <module>
2023-07-24T11:38:04.194+02:00 from jose import jwk
2023-07-24T11:38:04.194+02:00 File "/opt/python/jose/jwk.py", line 1, in <module>
2023-07-24T11:38:04.194+02:00 from jose.backends.base import Key
2023-07-24T11:38:04.194+02:00 File "/opt/python/jose/backends/__init__.py", line 2, in <module>
2023-07-24T11:38:04.194+02:00 from jose.backends.cryptography_backend import get_random_bytes # noqa: F401
2023-07-24T11:38:04.194+02:00 File "/opt/python/jose/backends/cryptography_backend.py", line 4, in <module>
2023-07-24T11:38:04.194+02:00 from cryptography.exceptions import InvalidSignature, InvalidTag
2023-07-24T11:38:04.194+02:00 File "/opt/python/cryptography/exceptions.py", line 9, in <module>
2023-07-24T11:38:04.194+02:00 from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
2023-07-24T11:38:04.194+02:00 pyo3_runtime.PanicException: Python API call failedReverting to 40.0.2 was the solution.
alex
commented
1 year ago This does not appear to be the same error at all, it has a totally different error message. Please read the first line:
ModuleNotFoundError: No module named '_cffi_backend'
This indicates that your lambda does not have the cffi module installed.
amswiatkowski
commented
1 year ago cffi is listed in my poetry.lock and is in Lambda layer, but still I guess if it won't be listed there, version revert won't be solution for me.
yogeshmahajan-1903
commented
1 year ago This can be fixed with https://github.com/pyca/cryptography/issues/9016
cwkfs
commented
1 year ago This can be fixed with #9016
The above comment re. solution seems to link back to this issue itself?
alex
commented
1 year ago This should be fixed for most use cases with our next release (pyo3 0.20), for actual sub-interpreter use cases, work is needed on pyo3 itself.
Different behaviour between 41.0.0, 41.0.1 and 40.0.2.
We are using cryptography via zigpy libraries. All of that is embedded into a C++ application which used Embedded Python Libraries in order to run some python codes.
We have detected a behaviour change since 40.0.2, which made our code not working anymore ( see the here after stack trace)
any ideas would be more than welcome
with cryptography 41.0.0 and cryptography 41.0.1