Python v3.13 related things to look into

rwgk commented 1 year ago

Using this to track things to look into when we start testing with Python 3.13:

https://github.com/python/cpython/pull/108763#issuecomment-1745262279 — Can/should we use PyObject_VisitManagedDict(), PyObject_ClearManagedDict() in pybind11_traverse, pybind11_clear?

rwgk commented 1 year ago

@wjakob @henryiii FYI

tacaswell commented 11 months ago

Currently (v3.13.0a1-422-gd9fd33a869) defining an enum appears to cause segfault on import of the module. This shows up in Matplotlib an in the pybind11 tests.

tacaswell commented 11 months ago

Running the pybind11 tests with faulthandler turned on gives:

Fatal Python error: Segmentation fault

Current thread 0x00007fdfed898740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1304 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 915 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1325 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1354 in _find_and_load
  File "/home/tcaswell/source/p/pybind/pybind11/tests/conftest.py", line 20 in <module>
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/assertion/rewrite.py", line 175 in exec_module
  File "<frozen importlib._bootstrap>", line 929 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1325 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1354 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1381 in _gcd_import
  File "/home/tcaswell/.pybuild/cp_main/lib/python3.13/importlib/__init__.py", line 88 in import_module
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/pathlib.py", line 567 in import_path
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 649 in _importconftest
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 616 in _loadconftestmodules
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 588 in _try_load_conftest
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 571 in _set_initial_conftests
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1160 in pytest_load_initial_conftests
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1334 in _preparse
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1434 in parse
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1082 in pytest_cmdline_parse
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 337 in _prepareconfig
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 155 in main
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 197 in console_main
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pytest/__main__.py", line 5 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: numpy._core._multiarray_umath, numpy._core._multiarray_tests, numpy.linalg._umath_linalg (total: 3)
make[3]: *** [tests/CMakeFiles/pytest.dir/build.make:74: tests/CMakeFiles/pytest] Segmentation fault (core dumped)

which implicates

    import pybind11_tests

rwgk commented 11 months ago

Tiny brain dump:

Either:

Assume it's a bug in v3.13.0a1 that others might find and fix: wait for v3.13.0a2 and try again.

Or locally build v3.13.0a1 from sources with debugging enabled, then pin-point the exact location of the segfault in the C code. Inspect the backtrace to figure out the root cause.

tacaswell commented 11 months ago

Apparently the last time I had to bisect CPython was also a pybind11 + enum related bug so I had a bisect run script + example project for this all set up 🤣

666b68e8f252e3c6238d6eed1fc82937a774316f is the first bad commit
commit 666b68e8f252e3c6238d6eed1fc82937a774316f
Author: Mark Shannon <mark@hotpy.org>
Date:   Fri Aug 11 20:05:56 2023 +0100

    GH-106485: Create object's dict-values instead of creating __dict__, when we can. (GH-107843)

 Objects/dictobject.c | 4 +---
 Objects/object.c     | 8 ++++++++
 2 files changed, 9 insertions(+), 3 deletions(-)
bisect found first bad commit

repo example : https://github.com/tacaswell/pybindbug

tacaswell commented 11 months ago

xref https://github.com/python/cpython/pull/107843

wjakob commented 11 months ago

Those functions were specifically introduced for tools like pybind11, so I think that using them is a good idea. One downside that currently keeps me from using them in nanobind is that they aren't part of the limited API (yet).

tacaswell commented 7 months ago

Not sure why, but my enum segfault is fixed. I don't have time to bisect if this was a change to pybind11 or a change upstream that fixed it.

susilehtola commented 4 months ago

SOME pybind11 ~fails to build~ TESTS FAIL in the Fedora development branch (rawhide) which has Python 3.13, making this bug somewhat time sensitive

https://bugzilla.redhat.com/show_bug.cgi?id=2293099

rwgk commented 4 months ago

I edited the comment to make it more precise

rwgk commented 4 months ago

Could you please try to insert

pytest.gc_collect()

between the del ... and the failing assert ... (looks like it's in 3 places)

That's just a quick guess.

And a quick thing to try.

I expect that it will not help, but good to confirm.

rwgk commented 4 months ago

Note that the test passes in the pybind11 GHA, which currently uses Python 3.13.0b3:

https://github.com/pybind/pybind11/actions/runs/9764673326/job/26953564324 (🐍 3.13 • ubuntu-20.04 • x64)
https://github.com/pybind/pybind11/actions/runs/9764673326/job/26953557039 (Manylinux on 🐍 3.13t • GIL, aka free-threading)

susilehtola commented 4 months ago

Thanks, the issue was solved by updating pybind11 to version 2.13.1.

pybind / pybind11

Python v3.13 related things to look into #4870