Closed rahulgharpinde closed 1 month ago
Is this a multi-threaded program?
This, unfortunately (and frustratingly) looks like it may be a product of: https://github.com/openssl/openssl/issues/17490
Where effectively the main thread runs its exit handles concurrently with OpenSSL usage by another thread.
https://github.com/openssl/openssl/issues/17469 is a more involved description of the atexit issue
And https://github.com/python/cpython/issues/114653 is a cpython thread with a simiilar crash/cause.
@alex Is this a multi-threaded program? --> Yes, we use multi-threading to execute command on multiple node using paramiko. From gdb info threads output, seems like it could be https://github.com/openssl/openssl/issues/17469 issue. Is there any workaround for this?
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f251e3ec700 (LWP 2803598) (Exiting) __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
2 Thread 0x7f2536497740 (LWP 2803421) 0x00007f2522238bcd in OPENSSL_strcasecmp () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
3 Thread 0x7f24fffff700 (LWP 2803603) 0x00007f2534e44251 in __GI___poll (fds=0x7f24ffffe9c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
4 Thread 0x7f251cbe9700 (LWP 2803602) 0x00007f2534e44251 in __GI___poll (fds=0x7f251cbe89c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
(gdb)
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f251e3ec700 (LWP 2803598))]
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 return ret;
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f2534d41ea5 in __GI_abort () at abort.c:79
#2 0x00007f2534dafcd7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f2534ea929b "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3 0x00007f2534db6fdc in malloc_printerr (str=str@entry=0x7f2534eaafc0 "double free or corruption (fasttop)") at malloc.c:5449
#4 0x00007f2534db8cbd in _int_free (av=0x7f250c000020, p=0x7f250c005db0, have_lock=<optimized out>) at malloc.c:4343
#5 0x00007f2522236740 in init_thread_stop.part () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#6 0x00007f2522236763 in init_thread_destructor () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#7 0x00007f2535876451 in __nptl_deallocate_tsd () at pthread_create.c:301
#8 0x00007f25358771eb in __nptl_deallocate_tsd () at pthread_create.c:256
#9 start_thread (arg=<optimized out>) at pthread_create.c:490
#10 0x00007f2534d59e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f2536497740 (LWP 2803421))]
#0 0x00007f2522238bcd in OPENSSL_strcasecmp () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
(gdb) bt
#0 0x00007f2522238bcd in OPENSSL_strcasecmp () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#1 0x00007f25222323af in OPENSSL_LH_delete () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#2 0x00007f2522250ddc in OBJ_NAME_remove () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#3 0x00007f25222325c4 in OPENSSL_LH_doall () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#4 0x00007f2522250fb7 in OBJ_NAME_cleanup () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#5 0x00007f252221f6e8 in evp_cleanup_int () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#6 0x00007f2522235e52 in OPENSSL_cleanup () from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
#7 0x00007f2534d7126c in __run_exit_handlers (status=0, listp=0x7f25350df758 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#8 0x00007f2534d713a0 in __GI_exit (status=<optimized out>) at exit.c:139
#9 0x00007f2534d5ad8c in __libc_start_main (main=0x55aabf0b0710 <main>, argc=3, argv=0x7ffeb2381368, init=0x55aabf0b0810 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7ffeb2381358) at ../csu/libc-start.c:336
#10 0x000055aabf0b074e in _start ()
(gdb)
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f24fffff700 (LWP 2803603))]
#0 0x00007f2534e44251 in __GI___poll (fds=0x7f24ffffe9c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
29 return SYSCALL_CANCEL (poll, fds, nfds, timeout);
(gdb) bt
#0 0x00007f2534e44251 in __GI___poll (fds=0x7f24ffffe9c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f2526bb5d20 in poll (__timeout=100, __nfds=1, __fds=0x7f24ffffe9c0) at /usr/include/bits/poll2.h:38
#2 internal_select (writing=writing@entry=0, interval=<optimized out>, connect=connect@entry=0, s=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:779
#3 0x00007f2526bb5df7 in sock_call_ex (s=s@entry=0x7f2520455780, writing=writing@entry=0, sock_func=sock_func@entry=0x7f2526bb41b7 <sock_recv_impl>, data=data@entry=0x7f24ffffea70,
connect=connect@entry=0, err=err@entry=0x0, timeout=100000000) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:874
#4 0x00007f2526bb638b in sock_call (data=0x7f24ffffea70, func=0x7f2526bb41b7 <sock_recv_impl>, writing=0, s=0x7f2520455780) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:3533
#5 sock_recv_guts (s=s@entry=0x7f2520455780, cbuf=<optimized out>, len=<optimized out>, flags=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:3533
#6 0x00007f2526bb655d in sock_recv (s=0x7f2520455780, args=args@entry=(16,)) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/cpython/bytesobject.h:37
#7 0x00007f2535ce3d8d in method_vectorcall_VARARGS (func=<optimized out>, args=0x7f2526eec3c8, nargsf=<optimized out>, kwnames=<optimized out>)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/descrobject.c:330
#8 0x00007f2535cc48c3 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<method_descriptor at remote 0x7f2526abca90>, tstate=0x7f2514005030)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_call.h:77
#9 PyObject_Vectorcall (callable=<method_descriptor at remote 0x7f2526abca90>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:299
#10 0x00007f2535c82442 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/ceval.c:4772
#11 0x00007f2535c8ed2c in _PyEval_EvalFrame (throwflag=0, frame=0x7f2526eec020, tstate=0x7f2514005030) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_ceval.h:70
#12 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, func=<optimized out>, tstate=<optimized out>)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/ceval.c:6435
#13 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=<optimized out>, func=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:393
#14 _PyObject_VectorcallTstate (tstate=0x7f2514005030, callable=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>, tstate=0x7f2514005030, callable=<optimized out>,
args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_call.h:92
#15 0x00007f2535d74a58 in method_vectorcall (method=<optimized out>, args=0x7f253621dc10 <_PyRuntime+58928>, nargsf=<optimized out>, kwnames=0x0)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/classobject.c:67
#16 0x00007f2535cb17bb in _PyVectorcall_Call (kwargs=<optimized out>, tuple=(), callable=<method at remote 0x7f2520479780>, func=0x7f2535d74940 <method_vectorcall>, tstate=0x7f2514005030)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:245
#17 _PyObject_Call (tstate=0x7f2514005030, callable=<method at remote 0x7f2520479780>, args=(), kwargs=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:328
#18 0x00007f2535d9e40c in thread_run (boot_raw=0x7f2520465290) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/_threadmodule.c:1092
#19 0x00007f2535d87924 in pythread_wrapper (arg=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/thread_pthread.h:241
#20 0x00007f25358771ca in start_thread (arg=<optimized out>) at pthread_create.c:479
#21 0x00007f2534d59e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f251cbe9700 (LWP 2803602))]
#0 0x00007f2534e44251 in __GI___poll (fds=0x7f251cbe89c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
29 return SYSCALL_CANCEL (poll, fds, nfds, timeout);
(gdb) bt
#0 0x00007f2534e44251 in __GI___poll (fds=0x7f251cbe89c0, nfds=1, timeout=100) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f2526bb5d20 in poll (__timeout=100, __nfds=1, __fds=0x7f251cbe89c0) at /usr/include/bits/poll2.h:38
#2 internal_select (writing=writing@entry=0, interval=<optimized out>, connect=connect@entry=0, s=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:779
#3 0x00007f2526bb5df7 in sock_call_ex (s=s@entry=0x7f25204556c0, writing=writing@entry=0, sock_func=sock_func@entry=0x7f2526bb41b7 <sock_recv_impl>, data=data@entry=0x7f251cbe8a70,
connect=connect@entry=0, err=err@entry=0x0, timeout=100000000) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:874
#4 0x00007f2526bb638b in sock_call (data=0x7f251cbe8a70, func=0x7f2526bb41b7 <sock_recv_impl>, writing=0, s=0x7f25204556c0) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:3533
#5 sock_recv_guts (s=s@entry=0x7f25204556c0, cbuf=<optimized out>, len=<optimized out>, flags=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/socketmodule.c:3533
#6 0x00007f2526bb655d in sock_recv (s=0x7f25204556c0, args=args@entry=(16,)) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/cpython/bytesobject.h:37
#7 0x00007f2535ce3d8d in method_vectorcall_VARARGS (func=<optimized out>, args=0x7f2526ef43c8, nargsf=<optimized out>, kwnames=<optimized out>)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/descrobject.c:330
#8 0x00007f2535cc48c3 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<method_descriptor at remote 0x7f2526abca90>, tstate=0x7f2510005000)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_call.h:77
#9 PyObject_Vectorcall (callable=<method_descriptor at remote 0x7f2526abca90>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:299
#10 0x00007f2535c82442 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/ceval.c:4772
#11 0x00007f2535c8ed2c in _PyEval_EvalFrame (throwflag=0, frame=0x7f2526ef4020, tstate=0x7f2510005000) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_ceval.h:70
#12 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, func=<optimized out>, tstate=<optimized out>)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/ceval.c:6435
#13 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=<optimized out>, func=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:393
#14 _PyObject_VectorcallTstate (tstate=0x7f2510005000, callable=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>, tstate=0x7f2510005000, callable=<optimized out>,
args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Include/internal/pycore_call.h:92
#15 0x00007f2535d74a58 in method_vectorcall (method=<optimized out>, args=0x7f253621dc10 <_PyRuntime+58928>, nargsf=<optimized out>, kwnames=0x0)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/classobject.c:67
#16 0x00007f2535cb17bb in _PyVectorcall_Call (kwargs=<optimized out>, tuple=(), callable=<method at remote 0x7f2520479600>, func=0x7f2535d74940 <method_vectorcall>, tstate=0x7f2510005000)
at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:245
#17 _PyObject_Call (tstate=0x7f2510005000, callable=<method at remote 0x7f2520479600>, args=(), kwargs=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Objects/call.c:328
#18 0x00007f2535d9e40c in thread_run (boot_raw=0x7f2520465320) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Modules/_threadmodule.c:1092
#19 0x00007f2535d87924 in pythread_wrapper (arg=<optimized out>) at /usr/src/debug/python3.11-3.11.2-2.el8_8.2.x86_64/Python/thread_pthread.h:241
#20 0x00007f25358771ca in start_thread (arg=<optimized out>) at pthread_create.c:479
#21 0x00007f2534d59e73 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
If you're able to wait for other threads to complete before exiting main, that's likely your best bet.
Once https://github.com/openssl/openssl/commit/99fb31c167e322186c6f576cfaa8f433f4fed117 is in an OpenSSL release, I think we'll build with it which will solve this issue.
Unfortunately I don't have a great suggestion beyond that.
That feature looks like it’ll be in 3.3, which is scheduled for April. We could conceivably ship it in a hypothetical May release.
A call to OPENSSL_init_ssl()
(which is exposed in rust-openssl) with OPENSSL_INIT_NO_ATEXIT
could solve this. Not sure if it would be acceptable for cryptography to expose this function or to add it to its init handlers.
We could call the init function, but it's not clear what happens if OpenSSL has already been initialized when it's called -- this will happen in basically any case where we're using a system OpenSSL.
On Sun, Mar 10, 2024 at 5:14 PM Theo Buehler @.***> wrote:
A call to OPENSSL_init_ssl() https://www.openssl.org/docs/man3.2/man3/OPENSSL_init_crypto.html (which is exposed in rust-openssl) with OPENSSL_INIT_NO_ATEXIT could solve this. Not sure if it would be acceptable for cryptography to expose this function or to add it to its init handlers.
— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/10368#issuecomment-1987364804, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBAVMKD3PDBK5OBDGDLYXTEMPAVCNFSM6AAAAABDADHKR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGM3DIOBQGQ . You are receiving this because you were mentioned.Message ID: @.***>
-- All that is necessary for evil to succeed is for good people to do nothing.
I don't follow. Automatic library initialization is supposed to happen during the first API call that needs it. Why is that different for a system OpenSSL?
Once the atexit handler is registered, it won't be unregistered by subsequent init calls.
If we're linked against system OpenSSL then Python or other things in process may init before we do.
@alex https://github.com/openssl/openssl/commit/99fb31c167e322186c6f576cfaa8f433f4fed117 to consume this fix, what all dependent packages need to update. And is there any documentation, which will help us to manually build package to test fix.
For us to take advantage of this, it needs to be in an openssl release, not merely in git.
Our installation documentation describes how to build against an openssl
On Tue, Mar 19, 2024, 9:13 AM rahulgharpinde @.***> wrote:
@alex https://github.com/alex @.*** https://github.com/openssl/openssl/commit/99fb31c167e322186c6f576cfaa8f433f4fed117 to consume this fix, what all dependent packages need to update. And is there any documentation, which will help us to manually build package to test fix.
— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/10368#issuecomment-2007143764, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBGTLOKSOYA3ITV6LNTYZA2XPAVCNFSM6AAAAABDADHKR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXGE2DGNZWGQ . You are receiving this because you were mentioned.Message ID: @.***>
@alex , by when https://github.com/openssl/openssl/issues/17469 issue will get fix and in which release it will be available? Any idea?
We'll upgrade to to OpenSSL 3.3 in our official wheels in our next release. We do not currently have a schedule for it.
On Tue, May 14, 2024 at 6:51 AM rahulgharpinde @.***> wrote:
@alex https://github.com/alex , by when openssl/openssl#17469 https://github.com/openssl/openssl/issues/17469 issue will get fix and in which release it will be available? Any idea?
— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/10368#issuecomment-2109889817, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBBZIRKJBNZNO7QZTCTZCHUE5AVCNFSM6AAAAABDADHKR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZHA4DSOBRG4 . You are receiving this because you were mentioned.Message ID: @.***>
-- All that is necessary for evil to succeed is for good people to do nothing.
We have bumped to 3.3.1 in infra, we'll be doing a release shortly.
Observe multiple crash on python 3.11 environment with cryptography module. We have a cluster environment on which multiple process and monitoring scripts are keep running in a interval. From traceback it is not clear which python code causing this issue intermittently. Need help to debug this issue in detail.
backtrace 1:
backtrace 2:
backtrace 3:
Versions of Python,
cryptography
,pip
, andsetuptools
/opt/VRTSnas/python/venv/bin/pip -V pip 23.2.1 from /opt/VRTSnas/python/venv/lib64/python3.11/site-packages/pip (python 3.11)/opt/VRTSnas/python/venv/bin/pip show cryptography Name: cryptography Version: 41.0.7 Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers. Home-page: Author: Author-email: The Python Cryptographic Authority and individual contributors cryptography-dev@python.org License: Apache-2.0 OR BSD-3-Clause Location: /opt/VRTSnas/python/venv/lib64/python3.11/site-packages Requires: cffi Required-by: paramiko, pyOpenSSL
/opt/VRTSnas/python/venv/bin/pip show setuptools Name: setuptools Version: 65.5.1 Summary: Easily download, build, install, upgrade, and uninstall Python packages Home-page: https://github.com/pypa/setuptools Author: Python Packaging Authority Author-email: distutils-sig@python.org License: Location: /opt/VRTSnas/python/venv/lib64/python3.11/site-packages Requires: Required-by:
OpenSSL 1.1.1k FIPS 25 Mar 2021 Red Hat Enterprise Linux release 8.8 (Ootpa)
How you installed
cryptography
: pip install -r requirements.txtSteps for reproducing your bug : There are no exact steps to reproduce this issue. Facing this issue intermittently and also not getting exact trace from where this is getting dump.