search

pyca / cryptography

cryptography is a package designed to expose cryptographic primitives and recipes to Python developers.
https://cryptography.io
Other
6.52k stars 1.5k forks source link

double free (fasttop) crashes on python 3.11 #10368

Closed rahulgharpinde closed 1 month ago

rahulgharpinde commented 6 months ago

Observe multiple crash on python 3.11 environment with cryptography module. We have a cluster environment on which multiple process and monitoring scripts are keep running in a interval. From traceback it is not clear which python code causing this issue intermittently. Need help to debug this issue in detail.

backtrace 1: 

Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50        return ret;
[Current thread is 1 (Thread 0x7f251e3ec700 (LWP 2803598))]
Missing separate debuginfos, use: yum debuginfo-install libgcc-8.5.0-18.2.el8_8.x86_64 openssl-libs-1.1.1k-12.el8_8.x86_64
(gdb)
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f2534d41ea5 in __GI_abort () at abort.c:79
#2  0x00007f2534dafcd7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f2534ea929b "%s\n")
    at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007f2534db6fdc in malloc_printerr (str=str@entry=0x7f2534eaafc0 "double free or corruption (fasttop)")
    at malloc.c:5449
#4  0x00007f2534db8cbd in _int_free (av=0x7f250c000020, p=0x7f250c005db0, have_lock=<optimized out>) at malloc.c:4343
#5  0x00007f2522236740 in init_thread_stop.part ()
   from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#6  0x00007f2522236763 in init_thread_destructor ()
   from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#7  0x00007f2535876451 in __nptl_deallocate_tsd () at pthread_create.c:301
#8  0x00007f25358771eb in __nptl_deallocate_tsd () at pthread_create.c:256
#9  start_thread (arg=<optimized out>) at pthread_create.c:490
#10 0x00007f2534d59e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

backtrace 2:

Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50        return ret;
[Current thread is 1 (Thread 0x7f0438a4b740 (LWP 962187))]
Missing separate debuginfos, use: yum debuginfo-install libgcc-8.5.0-18.2.el8_8.x86_64 libstdc++-8.5.0-18.2.el8_8.x86_64 openssl-libs-1.1.1k-12.el8_8.x86_64
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f04372f5ea5 in __GI_abort () at abort.c:79
#2  0x00007f0437363cd7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f043745d29b "%s\n")
    at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007f043736afdc in malloc_printerr (str=str@entry=0x7f043745efc0 "double free or corruption (fasttop)")
    at malloc.c:5449
#4  0x00007f043736ccbd in _int_free (av=0x7f03f0000020, p=0x7f03f0005e40, have_lock=<optimized out>) at malloc.c:4343
#5  0x00007f0424d0f850 in ossl_cleanup_thread ()
   from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#6  0x00007f0424d0ee48 in OPENSSL_cleanup ()
   from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#7  0x00007f043732526c in __run_exit_handlers (status=0, listp=0x7f0437693758 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#8  0x00007f04373253a0 in __GI_exit (status=<optimized out>) at exit.c:139
#9  0x00007f043730ed8c in __libc_start_main (main=0x560d0079d710 <main>, argc=5, argv=0x7ffc1f8d2bf8, init=
    0x560d0079d810 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc1f8d2be8)
    at ../csu/libc-start.c:336
#10 0x0000560d0079d74e in _start ()
(gdb) quit

backtrace 3:

Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50        return ret;
[Current thread is 1 (Thread 0x7f05f2ffd700 (LWP 3045101))]
Missing separate debuginfos, use: yum debuginfo-install libgcc-8.5.0-18.2.el8_8.x86_64 libstdc++-8.5.0-18.2.el8_8.x86_64 openssl-libs-1.1.1k-12.el8_8.x86_64
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f06226e1ea5 in __GI_abort () at abort.c:79
#2  0x00007f062274fcd7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f062284929b "%s\n")
    at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007f0622756fdc in malloc_printerr (str=str@entry=0x7f062284afc0 "double free or corruption (fasttop)")
    at malloc.c:5449
#4  0x00007f0622758cbd in _int_free (av=0x7f05e4000020, p=0x7f05e4008db0, have_lock=<optimized out>) at malloc.c:4343
#5  0x00007f0623216451 in __nptl_deallocate_tsd () at pthread_create.c:301
#6  0x00007f06232171eb in __nptl_deallocate_tsd () at pthread_create.c:256
#7  start_thread (arg=<optimized out>) at pthread_create.c:490
#8  0x00007f06226f9e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) quit

Versions of Python, cryptography, pip, and setuptools /opt/VRTSnas/python/venv/bin/pip -V pip 23.2.1 from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/pip (python 3.11)

/opt/VRTSnas/python/venv/bin/pip show cryptography Name: cryptography Version: 41.0.7 Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers. Home-page: Author: Author-email: The Python Cryptographic Authority and individual contributors cryptography-dev@python.org License: Apache-2.0 OR BSD-3-Clause Location: /opt/VRTSnas/python/venv/lib64/python3.11/site-packages Requires: cffi Required-by: paramiko, pyOpenSSL

/opt/VRTSnas/python/venv/bin/pip show setuptools Name: setuptools Version: 65.5.1 Summary: Easily download, build, install, upgrade, and uninstall Python packages Home-page: https://github.com/pypa/setuptools Author: Python Packaging Authority Author-email: distutils-sig@python.org License: Location: /opt/VRTSnas/python/venv/lib64/python3.11/site-packages Requires: Required-by:

OpenSSL 1.1.1k FIPS 25 Mar 2021 Red Hat Enterprise Linux release 8.8 (Ootpa)

How you installed cryptography : pip install -r requirements.txt

Steps for reproducing your bug : There are no exact steps to reproduce this issue. Facing this issue intermittently and also not getting exact trace from where this is getting dump.

alex commented 6 months ago

Is this a multi-threaded program?

This, unfortunately (and frustratingly) looks like it may be a product of: https://github.com/openssl/openssl/issues/17490

Where effectively the main thread runs its exit handles concurrently with OpenSSL usage by another thread.

alex commented 6 months ago

https://github.com/openssl/openssl/issues/17469 is a more involved description of the atexit issue

alex commented 6 months ago

And https://github.com/python/cpython/issues/114653 is a cpython thread with a simiilar crash/cause.

rahulgharpinde commented 6 months ago

@alex Is this a multi-threaded program? --> Yes, we use multi-threading to execute command on multiple node using paramiko. From gdb info threads output, seems like it could be https://github.com/openssl/openssl/issues/17469 issue. Is there any workaround for this? 

(gdb) info threads
  Id   Target Id                                     Frame
* 1    Thread 0x7f251e3ec700 (LWP 2803598) (Exiting) __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
  2    Thread 0x7f2536497740 (LWP 2803421)           0x00007f2522238bcd in OPENSSL_strcasecmp () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
  3    Thread 0x7f24fffff700 (LWP 2803603)           0x00007f2534e44251 in __GI___poll (fds=0x7f24ffffe9c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7f251cbe9700 (LWP 2803602)           0x00007f2534e44251 in __GI___poll (fds=0x7f251cbe89c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
(gdb)
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f251e3ec700 (LWP 2803598))]
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50        return ret;
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f2534d41ea5 in __GI_abort () at abort.c:79
#2  0x00007f2534dafcd7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f2534ea929b "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007f2534db6fdc in malloc_printerr (str=str@entry=0x7f2534eaafc0 "double free or corruption (fasttop)") at malloc.c:5449
#4  0x00007f2534db8cbd in _int_free (av=0x7f250c000020, p=0x7f250c005db0, have_lock=<optimized out>) at malloc.c:4343
#5  0x00007f2522236740 in init_thread_stop.part () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#6  0x00007f2522236763 in init_thread_destructor () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#7  0x00007f2535876451 in __nptl_deallocate_tsd () at pthread_create.c:301
#8  0x00007f25358771eb in __nptl_deallocate_tsd () at pthread_create.c:256
#9  start_thread (arg=<optimized out>) at pthread_create.c:490
#10 0x00007f2534d59e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f2536497740 (LWP 2803421))]
#0  0x00007f2522238bcd in OPENSSL_strcasecmp () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
(gdb) bt
#0  0x00007f2522238bcd in OPENSSL_strcasecmp () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#1  0x00007f25222323af in OPENSSL_LH_delete () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#2  0x00007f2522250ddc in OBJ_NAME_remove () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#3  0x00007f25222325c4 in OPENSSL_LH_doall () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#4  0x00007f2522250fb7 in OBJ_NAME_cleanup () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#5  0x00007f252221f6e8 in evp_cleanup_int () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#6  0x00007f2522235e52 in OPENSSL_cleanup () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#7  0x00007f2534d7126c in __run_exit_handlers (status=0, listp=0x7f25350df758 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#8  0x00007f2534d713a0 in __GI_exit (status=<optimized out>) at exit.c:139
#9  0x00007f2534d5ad8c in __libc_start_main (main=0x55aabf0b0710 <main>, argc=3, argv=0x7ffeb2381368, init=0x55aabf0b0810 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7ffeb2381358) at ../csu/libc-start.c:336
#10 0x000055aabf0b074e in _start ()
(gdb) 
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f24fffff700 (LWP 2803603))]
#0  0x00007f2534e44251 in __GI___poll (fds=0x7f24ffffe9c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
29        return SYSCALL_CANCEL (poll, fds, nfds, timeout);
(gdb) bt
#0  0x00007f2534e44251 in __GI___poll (fds=0x7f24ffffe9c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f2526bb5d20 in poll (__timeout=100, __nfds=1, __fds=0x7f24ffffe9c0) at /usr/include/bits/poll2.h:38
#2  internal_select (writing=writing@entry=0, interval=<optimized out>, connect=connect@entry=0, s=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:779
#3  0x00007f2526bb5df7 in sock_call_ex (s=s@entry=0x7f2520455780, writing=writing@entry=0, sock_func=sock_func@entry=0x7f2526bb41b7 <sock_recv_impl>, data=data@entry=0x7f24ffffea70,
    connect=connect@entry=0, err=err@entry=0x0, timeout=100000000) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:874
#4  0x00007f2526bb638b in sock_call (data=0x7f24ffffea70, func=0x7f2526bb41b7 <sock_recv_impl>, writing=0, s=0x7f2520455780) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:3533
#5  sock_recv_guts (s=s@entry=0x7f2520455780, cbuf=<optimized out>, len=<optimized out>, flags=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:3533
#6  0x00007f2526bb655d in sock_recv (s=0x7f2520455780, args=args@entry=(16,)) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/cpython/bytesobject.h:37
#7  0x00007f2535ce3d8d in method_vectorcall_VARARGS (func=<optimized out>, args=0x7f2526eec3c8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/descrobject.c:330
#8  0x00007f2535cc48c3 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<method_descriptor at remote 0x7f2526abca90>, tstate=0x7f2514005030)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_call.h:77
#9  PyObject_Vectorcall (callable=<method_descriptor at remote 0x7f2526abca90>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:299
#10 0x00007f2535c82442 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/ceval.c:4772
#11 0x00007f2535c8ed2c in _PyEval_EvalFrame (throwflag=0, frame=0x7f2526eec020, tstate=0x7f2514005030) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_ceval.h:70
#12 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, func=<optimized out>, tstate=<optimized out>)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/ceval.c:6435
#13 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=<optimized out>, func=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:393
#14 _PyObject_VectorcallTstate (tstate=0x7f2514005030, callable=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>, tstate=0x7f2514005030, callable=<optimized out>,
    args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_call.h:92
#15 0x00007f2535d74a58 in method_vectorcall (method=<optimized out>, args=0x7f253621dc10 <_PyRuntime+58928>, nargsf=<optimized out>, kwnames=0x0)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/classobject.c:67
#16 0x00007f2535cb17bb in _PyVectorcall_Call (kwargs=<optimized out>, tuple=(), callable=<method at remote 0x7f2520479780>, func=0x7f2535d74940 <method_vectorcall>, tstate=0x7f2514005030)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:245
#17 _PyObject_Call (tstate=0x7f2514005030, callable=<method at remote 0x7f2520479780>, args=(), kwargs=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:328
#18 0x00007f2535d9e40c in thread_run (boot_raw=0x7f2520465290) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/_threadmodule.c:1092
#19 0x00007f2535d87924 in pythread_wrapper (arg=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/thread_pthread.h:241
#20 0x00007f25358771ca in start_thread (arg=<optimized out>) at pthread_create.c:479
#21 0x00007f2534d59e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) 
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f251cbe9700 (LWP 2803602))]
#0  0x00007f2534e44251 in __GI___poll (fds=0x7f251cbe89c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
29        return SYSCALL_CANCEL (poll, fds, nfds, timeout);
(gdb) bt
#0  0x00007f2534e44251 in __GI___poll (fds=0x7f251cbe89c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f2526bb5d20 in poll (__timeout=100, __nfds=1, __fds=0x7f251cbe89c0) at /usr/include/bits/poll2.h:38
#2  internal_select (writing=writing@entry=0, interval=<optimized out>, connect=connect@entry=0, s=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:779
#3  0x00007f2526bb5df7 in sock_call_ex (s=s@entry=0x7f25204556c0, writing=writing@entry=0, sock_func=sock_func@entry=0x7f2526bb41b7 <sock_recv_impl>, data=data@entry=0x7f251cbe8a70,
    connect=connect@entry=0, err=err@entry=0x0, timeout=100000000) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:874
#4  0x00007f2526bb638b in sock_call (data=0x7f251cbe8a70, func=0x7f2526bb41b7 <sock_recv_impl>, writing=0, s=0x7f25204556c0) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:3533
#5  sock_recv_guts (s=s@entry=0x7f25204556c0, cbuf=<optimized out>, len=<optimized out>, flags=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:3533
#6  0x00007f2526bb655d in sock_recv (s=0x7f25204556c0, args=args@entry=(16,)) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/cpython/bytesobject.h:37
#7  0x00007f2535ce3d8d in method_vectorcall_VARARGS (func=<optimized out>, args=0x7f2526ef43c8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/descrobject.c:330
#8  0x00007f2535cc48c3 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<method_descriptor at remote 0x7f2526abca90>, tstate=0x7f2510005000)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_call.h:77
#9  PyObject_Vectorcall (callable=<method_descriptor at remote 0x7f2526abca90>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:299
#10 0x00007f2535c82442 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/ceval.c:4772
#11 0x00007f2535c8ed2c in _PyEval_EvalFrame (throwflag=0, frame=0x7f2526ef4020, tstate=0x7f2510005000) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_ceval.h:70
#12 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, func=<optimized out>, tstate=<optimized out>)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/ceval.c:6435
#13 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=<optimized out>, func=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:393
#14 _PyObject_VectorcallTstate (tstate=0x7f2510005000, callable=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>, tstate=0x7f2510005000, callable=<optimized out>,
    args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_call.h:92
#15 0x00007f2535d74a58 in method_vectorcall (method=<optimized out>, args=0x7f253621dc10 <_PyRuntime+58928>, nargsf=<optimized out>, kwnames=0x0)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/classobject.c:67
#16 0x00007f2535cb17bb in _PyVectorcall_Call (kwargs=<optimized out>, tuple=(), callable=<method at remote 0x7f2520479600>, func=0x7f2535d74940 <method_vectorcall>, tstate=0x7f2510005000)
    at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:245
#17 _PyObject_Call (tstate=0x7f2510005000, callable=<method at remote 0x7f2520479600>, args=(), kwargs=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:328
#18 0x00007f2535d9e40c in thread_run (boot_raw=0x7f2520465320) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/_threadmodule.c:1092
#19 0x00007f2535d87924 in pythread_wrapper (arg=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/thread_pthread.h:241
#20 0x00007f25358771ca in start_thread (arg=<optimized out>) at pthread_create.c:479
#21 0x00007f2534d59e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
alex commented 6 months ago

If you're able to wait for other threads to complete before exiting main, that's likely your best bet.

Once https://github.com/openssl/openssl/commit/99fb31c167e322186c6f576cfaa8f433f4fed117 is in an OpenSSL release, I think we'll build with it which will solve this issue.

Unfortunately I don't have a great suggestion beyond that.

reaperhulk commented 6 months ago

That feature looks like it’ll be in 3.3, which is scheduled for April. We could conceivably ship it in a hypothetical May release.

botovq commented 5 months ago

A call to OPENSSL_init_ssl() (which is exposed in rust-openssl) with OPENSSL_INIT_NO_ATEXIT could solve this. Not sure if it would be acceptable for cryptography to expose this function or to add it to its init handlers.

alex commented 5 months ago

We could call the init function, but it's not clear what happens if OpenSSL has already been initialized when it's called -- this will happen in basically any case where we're using a system OpenSSL.

On Sun, Mar 10, 2024 at 5:14 PM Theo Buehler @.***> wrote:

A call to OPENSSL_init_ssl() https://www.openssl.org/docs/man3.2/man3/OPENSSL_init_crypto.html (which is exposed in rust-openssl) with OPENSSL_INIT_NO_ATEXIT could solve this. Not sure if it would be acceptable for cryptography to expose this function or to add it to its init handlers.

— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/10368#issuecomment-1987364804, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBAVMKD3PDBK5OBDGDLYXTEMPAVCNFSM6AAAAABDADHKR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGM3DIOBQGQ . You are receiving this because you were mentioned.Message ID: @.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

botovq commented 5 months ago

I don't follow. Automatic library initialization is supposed to happen during the first API call that needs it. Why is that different for a system OpenSSL?

Once the atexit handler is registered, it won't be unregistered by subsequent init calls.

reaperhulk commented 5 months ago

If we're linked against system OpenSSL then Python or other things in process may init before we do.

rahulgharpinde commented 5 months ago

@alex https://github.com/openssl/openssl/commit/99fb31c167e322186c6f576cfaa8f433f4fed117 to consume this fix, what all dependent packages need to update. And is there any documentation, which will help us to manually build package to test fix.

alex commented 5 months ago

For us to take advantage of this, it needs to be in an openssl release, not merely in git.

Our installation documentation describes how to build against an openssl

On Tue, Mar 19, 2024, 9:13 AM rahulgharpinde @.***> wrote:

@alex https://github.com/alex @.*** https://github.com/openssl/openssl/commit/99fb31c167e322186c6f576cfaa8f433f4fed117 to consume this fix, what all dependent packages need to update. And is there any documentation, which will help us to manually build package to test fix.

— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/10368#issuecomment-2007143764, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBGTLOKSOYA3ITV6LNTYZA2XPAVCNFSM6AAAAABDADHKR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXGE2DGNZWGQ . You are receiving this because you were mentioned.Message ID: @.***>

rahulgharpinde commented 3 months ago

@alex , by when https://github.com/openssl/openssl/issues/17469 issue will get fix and in which release it will be available? Any idea?

alex commented 3 months ago

We'll upgrade to to OpenSSL 3.3 in our official wheels in our next release. We do not currently have a schedule for it.

On Tue, May 14, 2024 at 6:51 AM rahulgharpinde @.***> wrote:

@alex https://github.com/alex , by when openssl/openssl#17469 https://github.com/openssl/openssl/issues/17469 issue will get fix and in which release it will be available? Any idea?

— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/10368#issuecomment-2109889817, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBBZIRKJBNZNO7QZTCTZCHUE5AVCNFSM6AAAAABDADHKR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZHA4DSOBRG4 . You are receiving this because you were mentioned.Message ID: @.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

alex commented 1 month ago

We have bumped to 3.3.1 in infra, we'll be doing a release shortly.