Python v3.13 related things to look into

rwgk commented 9 months ago

Using this to track things to look into when we start testing with Python 3.13:

https://github.com/python/cpython/pull/108763#issuecomment-1745262279 — Can/should we use PyObject_VisitManagedDict(), PyObject_ClearManagedDict() in pybind11_traverse, pybind11_clear?

rwgk commented 9 months ago

@wjakob @henryiii FYI

tacaswell commented 7 months ago

Currently (v3.13.0a1-422-gd9fd33a869) defining an enum appears to cause segfault on import of the module. This shows up in Matplotlib an in the pybind11 tests.

tacaswell commented 7 months ago

Running the pybind11 tests with faulthandler turned on gives:

Fatal Python error: Segmentation fault

Current thread 0x00007fdfed898740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1304 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 915 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1325 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1354 in _find_and_load
  File "/home/tcaswell/source/p/pybind/pybind11/tests/conftest.py", line 20 in <module>
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/assertion/rewrite.py", line 175 in exec_module
  File "<frozen importlib._bootstrap>", line 929 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1325 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1354 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1381 in _gcd_import
  File "/home/tcaswell/.pybuild/cp_main/lib/python3.13/importlib/__init__.py", line 88 in import_module
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/pathlib.py", line 567 in import_path
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 649 in _importconftest
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 616 in _loadconftestmodules
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 588 in _try_load_conftest
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 571 in _set_initial_conftests
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1160 in pytest_load_initial_conftests
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1334 in _preparse
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1434 in parse
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 1082 in pytest_cmdline_parse
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 337 in _prepareconfig
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 155 in main
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/_pytest/config/__init__.py", line 197 in console_main
  File "/home/tcaswell/.virtualenvs/cp_main/lib/python3.13/site-packages/pytest/__main__.py", line 5 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: numpy._core._multiarray_umath, numpy._core._multiarray_tests, numpy.linalg._umath_linalg (total: 3)
make[3]: *** [tests/CMakeFiles/pytest.dir/build.make:74: tests/CMakeFiles/pytest] Segmentation fault (core dumped)

which implicates

    import pybind11_tests

rwgk commented 7 months ago

Tiny brain dump:

Either:

Assume it's a bug in v3.13.0a1 that others might find and fix: wait for v3.13.0a2 and try again.

Or locally build v3.13.0a1 from sources with debugging enabled, then pin-point the exact location of the segfault in the C code. Inspect the backtrace to figure out the root cause.

tacaswell commented 7 months ago

Apparently the last time I had to bisect CPython was also a pybind11 + enum related bug so I had a bisect run script + example project for this all set up 🤣

666b68e8f252e3c6238d6eed1fc82937a774316f is the first bad commit
commit 666b68e8f252e3c6238d6eed1fc82937a774316f
Author: Mark Shannon <mark@hotpy.org>
Date:   Fri Aug 11 20:05:56 2023 +0100

    GH-106485: Create object's dict-values instead of creating __dict__, when we can. (GH-107843)

 Objects/dictobject.c | 4 +---
 Objects/object.c     | 8 ++++++++
 2 files changed, 9 insertions(+), 3 deletions(-)
bisect found first bad commit

repo example : https://github.com/tacaswell/pybindbug

tacaswell commented 7 months ago

xref https://github.com/python/cpython/pull/107843

wjakob commented 7 months ago

Those functions were specifically introduced for tools like pybind11, so I think that using them is a good idea. One downside that currently keeps me from using them in nanobind is that they aren't part of the limited API (yet).

tacaswell commented 3 months ago

Not sure why, but my enum segfault is fixed. I don't have time to bisect if this was a change to pybind11 or a change upstream that fixed it.

susilehtola commented 6 days ago

SOME pybind11 ~fails to build~ TESTS FAIL in the Fedora development branch (rawhide) which has Python 3.13, making this bug somewhat time sensitive

https://bugzilla.redhat.com/show_bug.cgi?id=2293099

rwgk commented 6 days ago

I edited the comment to make it more precise

rwgk commented 6 days ago

Could you please try to insert

pytest.gc_collect()

between the del ... and the failing assert ... (looks like it's in 3 places)

That's just a quick guess.

And a quick thing to try.

I expect that it will not help, but good to confirm.

rwgk commented 6 days ago

Note that the test passes in the pybind11 GHA, which currently uses Python 3.13.0b3:

https://github.com/pybind/pybind11/actions/runs/9764673326/job/26953564324 (🐍 3.13 • ubuntu-20.04 • x64)
https://github.com/pybind/pybind11/actions/runs/9764673326/job/26953557039 (Manylinux on 🐍 3.13t • GIL, aka free-threading)

susilehtola commented 3 days ago

Thanks, the issue was solved by updating pybind11 to version 2.13.1.

pybind / pybind11

Python v3.13 related things to look into #4870