search

python / cpython

The Python programming language
https://www.python.org
Other
62.59k stars 30.03k forks source link

Segmentation fault when loading shared libraries in shiboken2 with Python 3.12 #118777

Closed petronny closed 4 months ago

petronny commented 4 months ago

Crash report

What happened?

TL;DR:

$ pip install shiboken2 --user --break-system-packages
$ mkdir shiboken2
$ cd shiboken2
$ cp ~/.local/lib/python3.12/site-packages/shiboken2/{libshiboken2.abi3.so.5.13,shiboken2.abi3.so} .
$ yes | pip uninstall shiboken2 --break-system-packages
$ python -c 'import shiboken2'
Segmentation fault

$ gdb python                                                                                                                                                                                                                                            :(
GNU gdb (GDB) 14.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
                                                                                                                                                                                                                                                                                        This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Reading symbols from /home/petron/.cache/debuginfod_client/60c1de655b83f2e714badbc272f245e2108e48c3/debuginfo...
(gdb) run -c 'import shiboken2'
Starting program: /usr/bin/python -c 'import shiboken2'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
                                                                                                                                                                                                                                                                                        Program received signal SIGSEGV, Segmentation fault.
0x00007ffff79830de in PyType_HasFeature (feature=<optimized out>, type=<optimized out>) at ./Include/object.h:966
966         flags = type->tp_flags;
(gdb) bt
#0  0x00007ffff79830de in PyType_HasFeature (feature=<optimized out>, type=<optimized out>) at ./Include/object.h:966
#1  PyDict_GetItem (op=0x0, key=0x7ffff6d037b0) at Objects/dictobject.c:1667
#2  0x00007ffff79372d5 in PyDict_GetItemString (v=0x0, key=<optimized out>) at Objects/dictobject.c:3895
#3  0x00007ffff6223888 in add_more_getsets () from /home/petron/shiboken2/libshiboken2.abi3.so.5.13
#4  0x00007ffff6225b06 in FinishSignatureInitialization () from /home/petron/shiboken2/libshiboken2.abi3.so.5.13
#5  0x00007ffff6802365 in PyInit_shiboken2 () from /home/petron/shiboken2/shiboken2.abi3.so
#6  0x00007ffff7a70f0d in _PyImport_LoadDynamicModuleWithSpec (fp=<optimized out>, spec=0x7ffff6d01970) at ./Python/importdl.c:169
#7  _imp_create_dynamic_impl (module=<optimized out>, file=<optimized out>, spec=0x7ffff6d01970) at Python/import.c:3775
#8  _imp_create_dynamic (module=<optimized out>, args=<optimized out>, nargs=<optimized out>) at Python/clinic/import.c.h:506
#9  0x00007ffff79a677b in cfunction_vectorcall_FASTCALL (func=<optimized out>, args=0x7ffff6d003e8, nargsf=1, kwnames=<optimized out>) at Objects/methodobject.c:422
#10 0x00007ffff788cb8e in PyCFunction_Call (kwargs=0x7ffff7619c80, args=0x7ffff6d003d0, callable=0x7ffff75b72e0) at Objects/call.c:387
#11 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fbe368, throwflag=<optimized out>) at Python/bytecodes.c:3254
#12 0x00007ffff79a4da6 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffffd8d0, callable=0x7ffff75c4040, tstate=0x7ffff7e22ae8 <_PyRuntime+459656>) at ./Include/internal/pycore_call.h:92
#13 object_vacall (tstate=tstate@entry=0x7ffff7e22ae8 <_PyRuntime+459656>, base=<optimized out>, callable=0x7ffff75c4040, vargs=0x7fffffffd960) at Objects/call.c:850
#14 0x00007ffff79d43c5 in PyObject_CallMethodObjArgs (obj=<optimized out>, name=<optimized out>) at Objects/call.c:911
#15 0x00007ffff79d3996 in import_find_and_load (abs_name=0x7ffff75cd370, tstate=0x7ffff7e22ae8 <_PyRuntime+459656>) at Python/import.c:2779
#16 PyImport_ImportModuleLevelObject (name=0x7ffff75cd370, globals=<optimized out>, locals=<optimized out>, fromlist=0x7ffff7da9de0 <_Py_NoneStruct>, level=0) at Python/import.c:2862
#17 0x00007ffff7897265 in import_name (level=0x7ffff7db3428 <_PyRuntime+3272>, fromlist=0x7ffff7da9de0 <_Py_NoneStruct>, name=0x7ffff75cd370, frame=0x7ffff7fbe020, tstate=<optimized out>) at Python/ceval.c:2482
#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fbe020, throwflag=<optimized out>) at Python/bytecodes.c:2135
#19 0x00007ffff7a3d767 in PyEval_EvalCode (co=0x7ffff6d84ed0, globals=<optimized out>, locals=0x7ffff7619bc0) at Python/ceval.c:578
#20 0x00007ffff7a608b7 in run_eval_code_obj (tstate=tstate@entry=0x7ffff7e22ae8 <_PyRuntime+459656>, co=co@entry=0x7ffff6d84ed0, globals=globals@entry=0x7ffff7619bc0, locals=locals@entry=0x7ffff7619bc0) at Python/pythonrun.c:1722
#21 0x00007ffff7a5b9dc in run_mod (mod=mod@entry=0x555555614538, filename=filename@entry=0x7ffff7db85c0 <_PyRuntime+24160>, globals=globals@entry=0x7ffff7619bc0, locals=locals@entry=0x7ffff7619bc0, flags=flags@entry=0x7fffffffde70, arena=arena@entry=0x7ffff753be50)
    at Python/pythonrun.c:1743
#22 0x00007ffff7a4dedd in PyRun_StringFlags (str=<optimized out>, start=<optimized out>, globals=0x7ffff7619bc0, locals=0x7ffff7619bc0, flags=0x7fffffffde70) at Python/pythonrun.c:1618
#23 0x00007ffff7a4db60 in PyRun_SimpleStringFlags (command=0x7ffff7619d90 "import shiboken2\n", flags=0x7fffffffde70) at Python/pythonrun.c:480
#24 0x00007ffff7a6cdce in pymain_run_command (command=<optimized out>) at Modules/main.c:255
#25 pymain_run_python (exitcode=0x7fffffffde44) at Modules/main.c:620
#26 Py_RunMain () at Modules/main.c:709
#27 0x00007ffff7a28fab in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:763
#28 0x00007ffff7643cd0 in ?? () from /usr/lib/libc.so.6
#29 0x00007ffff7643d8a in __libc_start_main () from /usr/lib/libc.so.6
#30 0x0000555555555045 in _start ()

I'm trying to build shiboken2 (part of pyside2) with Python 3.12 on Arch Linux.

I'm awared that pyside2 doesn't officially support Python 3.12 even Python 3.11. But it actually used to work well with Python 3.11.

However, since Python 3.12 the build of pyside2 throws segmentation fault during import PySide2. We then noticed that directly installing pyside2 from pypi and running import PySide2 will also raise a similar segmentation fault. We believe these two segmentation faults are same so there is no need to actually run the build process.

Then we found the segmentation fault is raised during import shiboken2 when running import PySide2 with the pypi installation. And the minimal environment to reproduce the error is just grabbing the two libraries and try to load them with Python 3.12.

See also: https://github.com/arch4edu/arch4edu/issues/268

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.12.3 (main, Apr 23 2024, 09:16:07) [GCC 13.2.1 20240417]

gaogaotiantian commented 4 months ago

The fact that is crashed in PyInit_shiboken2 implies that this might be a shiboken problem instead of a CPython one. With C extensions, you can easily crash CPython with incorrect code and there's nothing CPython can do.

From the backtrace it seems like PyDict_GetItemString() gets a NULL pointer from add_more_getsets which is shiboken code. If I had to guess, that's where the issue is.

In any case, if this is indeed a CPython issue, you should be able to reproduce it without any 3rd party library. For this specific matter, I don't think this is a CPython bug.

petronny commented 4 months ago

The fact that is crashed in PyInit_shiboken2 implies that this might be a shiboken problem instead of a CPython one. With C extensions, you can easily crash CPython with incorrect code and there's nothing CPython can do.

From the backtrace it seems like PyDict_GetItemString() gets a NULL pointer from add_more_getsets which is shiboken code. If I had to guess, that's where the issue is.

In any case, if this is indeed a CPython issue, you should be able to reproduce it without any 3rd party library. For this specific matter, I don't think this is a CPython bug.

Thank you for your reply. I believe some changes in Python 3.12 must be related to this issue since these two libaries can be loaded without any problem in Python 3.11. I will check if there is any change about PyDict_GetItemString().

chgnrdv commented 4 months ago

FinishSignatureInitialization calls PySide_PatchTypes, which calls add_more_getsets on different types here: https://code.qt.io/cgit/pyside/pyside-setup.git/tree/sources/shiboken2/libshiboken/signature.cpp?h=5.13#n873 which calls PyType_Ready on type and then tries to access its tp_dict field: https://code.qt.io/cgit/pyside/pyside-setup.git/tree/sources/shiboken2/libshiboken/signature.cpp?h=5.13#n628 But is it correct in 3.12 for static built-in types as PyCFunction_Type/PyType_Type/wrapper-descriptor? Docs say that tp_dict is always equal to NULL for these types and that PyType_GetDict should be used. https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_dict

petronny commented 4 months ago

tp_dict is always equal to NULL for these types and that PyType_GetDict should be used. https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_dict

That must be the reason. I'm trying to make a patch to add_more_getsets() like directly returning 0 if dict pointer is NULL or reassign it to PyType_GetDict().

petronny commented 4 months ago

I add a line if (dict == NULL) dict = PyType_GetDict(type); under PyObject *dict = type->tp_dict; to ensure that dict pointer is not NULL. After rebuilding shiboken2 from source the segmentation fault is gone.

It looks like the problem is solved but the build of pyside2 still has an error. It can be bypassed by just removing this line but I appreciate if anybody can help to resolve it properly.

Traceback (most recent call last):
  File "/build/pyside2/src/pyside-setup-opensource-src-5.15.13/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py", line 317, in <module>
    generate_all_pyi(outpath, options=options)
  File "/build/pyside2/src/pyside-setup-opensource-src-5.15.13/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py", line 294, in generate_all_pyi
    generate_pyi(import_name, outpath, options)
  File "/build/pyside2/src/pyside-setup-opensource-src-5.15.13/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py", line 220, in generate_pyi
    fmt = Formatter(outfile)
          ^^^^^^^^^^^^^^^^^^
  File "/build/pyside2/src/pyside-setup-opensource-src-5.15.13/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py", line 119, in __init__
    typing.TypeVar.__repr__ = _typevar__repr__
    ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot set '__repr__' attribute of immutable type 'typing.TypeVar'
make[2]: *** [sources/pyside2/PySide2/QtCore/CMakeFiles/QtCore_pyi.dir/build.make:70: sources/pyside2/PySide2/QtCore/CMakeFiles/QtCore_pyi] Error 1
make[1]: *** [CMakeFiles/Makefile2:1573: sources/pyside2/PySide2/QtCore/CMakeFiles/QtCore_pyi.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....