search

pyca / cryptography

cryptography is a package designed to expose cryptographic primitives and recipes to Python developers.
https://cryptography.io
Other
6.7k stars 1.54k forks source link

ImportError: PyO3 modules may only be initialized once per interpreter process #12080

Closed linas closed 5 hours ago

linas commented 6 hours ago

This is in reference to issue #9016 which is marked resolved, closed and locked. Its currently December 2024, and this remains unfixed, i.e. an open issue, in both Debian Stable (bullseye) and Debian Testing. There's no additional information I can add to this issue report that is not already noted in issue #9016. There's no obvious work-around or patch available that can be deployed. So ???

alex commented 6 hours ago

As discussed in that issue thread, supporting sub-interpreters requires https://github.com/PyO3/pyo3/issues/3451 to be resolved in pyo3.

linas commented 6 hours ago

OK, closing, and redirecting my attention there.

linas commented 6 hours ago

Hmm. Is there a way to revert whatever it is that is breaking, until a fix becomes available for https://github.com/PyO3/pyo3/issues/3451 ? The maintainer there is posting remarks about "issues in my personal life" and "now that college classes are in session" and "this is more complicated than I thought", coupled to it being 14 months since opened with apparently no actual progress, makes me think that a patch is years away, or might never arrive.

In case that issue is never resolved, or only gets a fix years from now, is there perhaps some intermediate steps that can be taken in this project to work around that bug? Maybe just not use PyO3 at all, or not use sub-interpreters, something along those lines?

alex commented 6 hours ago

No, there is no straightforward way to revert the relevant change.

On Sun, Dec 1, 2024 at 2:23 PM Linas Vepštas @.***> wrote:

Hmm. Is there a way to revert whatever it is that is breaking, until a fix becomes available for PyO3/pyo3#3451 https://github.com/PyO3/pyo3/issues/3451 ? The maintainer there is posting remarks about "issues in my personal life" and "now that college classes are in session" and "this is more complicated than I thought", coupled to it being 14 months since opened with apparently no actual progress, makes me think that a patch is years away, or might never arrive.

In case that issue is never resolved, or only gets a fix years from now, is there perhaps some intermediate steps that can be taken in this project to work around that bug? Maybe just not use PyO3 at all, or not use sub-interpreters, something along those lines?

— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/12080#issuecomment-2510216658, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBBQJGO5IP3CTBUPFCT2DNO2DAVCNFSM6AAAAABSZ7CRA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJQGIYTMNRVHA . You are receiving this because you commented.Message ID: @.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

linas commented 6 hours ago

So the current plan is to stay broken, possibly forever? I don't get it.

alex commented 6 hours ago

I wouldn't call it a plan, but yes, the reality is that unless someone steps up to do the work, sub-interpreters will remain unsupported.

linas commented 6 hours ago

Who is using the sub-interpreters? Some blob of code somewhere in the stack decides "OK, start subinterpreter here". If I can locate that, then that would be the place to disable or bypass or mutex-lock or whatever.

I'm coming at this from the Ceph side, as a Ceph user (not developer). In the past, this message seemed to be an anoying warning that merely clogged up the error logs, but could be safely ignored. Today, I'm actually trying to do something, and this is appears to be a blocker. I think. I'll double check to see if it really is blocking, or just an annoying warning.

alex commented 6 hours ago

Ceph is making use of sub-interpreters. I do not know why.

linas commented 5 hours ago

For the record, the stack trace I'm looking at is this:

# ceph dashboard create-self-signed-cert
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
    return self.func(mgr, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/dashboard/module.py", line 476, in set_mgr_created_self_signed_cert
    cert, pkey = create_self_signed_cert('IT', 'ceph-dashboard')
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/mgr_util.py", line 527, in create_self_signed_cert
    from OpenSSL import crypto
  File "/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import SSL, crypto
  File "/lib/python3/dist-packages/OpenSSL/SSL.py", line 19, in <module>
    from OpenSSL.crypto import (
  File "/lib/python3/dist-packages/OpenSSL/crypto.py", line 21, in <module>
    from cryptography import utils, x509
  File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 6, in <module>
    from cryptography.x509 import certificate_transparency
  File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py", line 10, in <module>
    from cryptography.hazmat.bindings._rust import x509 as rust_x509
ImportError: PyO3 modules may only be initialized once per interpreter process

This looks like a straightforward naive API call to create a cert, with nothing fancy going on. The only "obvious" work-around would be some if (not already imported) import OpenSSL.crypto. Not clear if that will work, but this kind of protection from multi-imports is common in python code. I've seen it in ROS (robot operating system), and in Blender (3D animation system) which have complex multi-module cascading everybody is calling everything all the time in unpredictable order kinds of ways. Given the complexity of Ceph, its probably equally uuuhh, "stochastic" (its got a zillion daemons running on a zillion hosts, so like ROS)

Search engines report that the Ceph variant of this is being discussed on a variety of forums. The generic work-around is "revert to an earlier version of the python crypto module". I'll see if I can get the Debian maintainers to roll back to an early 2023 version of crypto, but I imagine there will be pushback. Maybe the Ceph people can do something. Sigh.

alex commented 5 hours ago

The relevant portion of the ceph code is at https://github.com/ceph/ceph/blob/69c919838b4774559e407d3ce6a08d668c9b6f51/src/mgr/PyModule.cc#L282

On Sun, Dec 1, 2024 at 2:58 PM Linas Vepštas @.***> wrote:

For the record, the stack trace I'm looking at is this:

ceph dashboard create-self-signed-cert

Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/dashboard/module.py", line 476, in set_mgr_created_self_signed_cert cert, pkey = create_self_signed_cert('IT', 'ceph-dashboard') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_util.py", line 527, in create_self_signed_cert from OpenSSL import crypto File "/lib/python3/dist-packages/OpenSSL/init.py", line 8, in from OpenSSL import SSL, crypto File "/lib/python3/dist-packages/OpenSSL/SSL.py", line 19, in from OpenSSL.crypto import ( File "/lib/python3/dist-packages/OpenSSL/crypto.py", line 21, in from cryptography import utils, x509 File "/lib/python3/dist-packages/cryptography/x509/init.py", line 6, in from cryptography.x509 import certificate_transparency File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py", line 10, in from cryptography.hazmat.bindings._rust import x509 as rust_x509 ImportError: PyO3 modules may only be initialized once per interpreter process

This looks like a straightforward naive API call to create a cert, with nothing fancy going on. The only "obvious" work-around would be some if (not already imported) import OpenSSL.crypto. Not clear if that will work, but this kind of protection from multi-imports is common in python code. I've seen it in ROS (robot operating system), and in Blender (3D animation system) which have complex multi-module cascading everybody is calling everything all the time in unpredictable order kinds of ways. Given the complexity of Ceph, its probably equally uuuhh, "stochastic" (its got a zillion daemons running on a zillion hosts, so like ROS)

Search engines report that the Ceph variant of this is being discussed on a variety of forums. The generic work-around is "revert to an earlier version of the python crypto module". I'll see if I can get the Debian maintainers to roll back to an early 2023 version of crypto, but I imagine there will be pushback. Maybe the Ceph people can do something. Sigh.

— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/12080#issuecomment-2510228599, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBDTJYKJKFPP4NPGSWT2DNTAFAVCNFSM6AAAAABSZ7CRA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJQGIZDQNJZHE . You are receiving this because you commented.Message ID: @.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

linas commented 5 hours ago

Sigh. OK. Closing again. Here's a relevant redhat bug report: https://bugzilla.redhat.com/show_bug.cgi?id=2255688

I'll do some noisy posting here as I find other relevant threads, so that the next person will, auhhh be saved some time tracking this down.

linas commented 5 hours ago

Also-ran:

reaperhulk commented 4 hours ago

If the dep isn't needed then yes, that's a good solution. For future folks who may find this thread: subinterpreters were never explicitly supported by this project and when they "worked" they were untested and potentially unsound with our module. Subinterpreters have a long and checkered history in Python and projects that consume them ultimately need to either minimize their deps to only those who explicitly claim support or work with upstreams to create such support.

The only way forward here is by assisting the pyo3 project in explicitly supporting subinterpreters. That may take the form of direct contribution, a variety of possible sponsorships, or simply reaching out to the pyo3 maintainers on their issue tracker to reopen discussion since it previously stalled.