Initial attempt at support for Python 3.13.

[indygreg]

Thank you for this PR!

I glanced at this and it seems to have all the components for 3.13 support. Let's see what CI says...

[kpfleming]

Once the CI is happy I'll rebase this and squash it down to one commit, and them mark it ready for review.

[kpfleming]

The builds are failing because the patches for configure.ac from 3.12 won't apply, which was expected. Those patches will need to be regenerated for 3.13.

[kpfleming]

A bit more progress this morning. A couple of new patches, disabling one (for apple-cross support), but a local Linux build still fails, just much much farther into the process.

[kpfleming]

Well, I think I've reached the end of what I can reasonably accomplish at this point. The current build failure (in Modules/_curses_panel.c) points to some macro not being defined, or a header not being included, but I've looked over all the files involved and I can't see what is causing this to happen. The changes in this file between 3.12 and 3.13 are not drastic, so I can only assume something has changed in the CPython build system itself.

[zanieb]

~Suspicious: https://github.com/python/cpython/pull/111567~

Edit: Perhaps not actually, while it uses curses, curses_panel has been around for a long time and hasn't changed much due to the REPL changes.

[zanieb]

Here's the build changes in the release notes

[zanieb]

I presume the newly bundled mimalloc is going to require some sort of handling. Still not quite sure what's causing this particular failure though.

[indygreg]

./Modules/clinic/_curses_panel.c.h:169:36: error: implicit declaration of function '_Py_SINGLETON' [-Werror=implicit-function-declaration]

_Py_SINGLETON is defined by pycore_global_objects.h. This is probably a missing #include.

python-build-standalone is likely seeing this error because we statically compile extension modules. This configuration isn't well tested by upstream CPython. So it's likely that nobody has noticed yet.

Since this source file was generated by clinic, it is possibly a bug in the clinic code generator, which may want to emit the missing #include. We should file an upstream issue and get @vstinner's eyeballs on it.

[ofek]

I'm very excited to start testing free-threading support after this is fixed and merged!

[kpfleming]

Issue opened: https://github.com/python/cpython/issues/119661

[kpfleming]

In the interests of saving the GitHub Actions runners from running jobs that will fail anyway, I'm running local builds now :-)

The patch from Victor for the clinic modules was successful, those build properly now. The next problem is:

cpython-3.13> clang -pthread  -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -fdebug-default-version=4 -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw   -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include -fdebug-default-version=4 -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw  -fPIC -DPy_BUILD_CORE_BUILTIN -c ./Modules/syslogmodule.c -o Modules/syslogmodule.o
cpython-3.13> ./Modules/readline.c:1305:21: error: incompatible function pointer types assigning to 'Function *' (aka 'int (*)(const char *, int)') from 'int (void)' [-Wincompatible-function-pointer-types]
cpython-3.13>  1305 |     rl_start
cpython-3.13> up_hook = on_startup_hook;
cpython-3.13>       |                     ^ ~~~~~~~~~~~~~~~
cpython-3.13> 1 error generated.
cpython-3.13> Makefile:3392: recipe for target 'Modules/readline.o' failed

I suspect that support for building against libedit may have regressed in 3.13.

[kpfleming]

Turns out that one was easy to fix... I'm continuing onwards :-)

[kpfleming]

OK, new status report: the actual CPython build succeeds, but then the process stops because Tools/scripts/run_tests.py does not exist in 3.13. I'll clean up and push what I've got so far, but this one will be harder to resolve.

[zanieb]

run_tests.py was removed in https://github.com/python/cpython/pull/110156 as part of https://github.com/python/cpython/issues/110152. In brief, the suggestion is:

Just run ./python -m test --slow-ci, make buildbottest or make test instead. Python test runner (regrtest) now handles cross-compilation and HOSTRUNNER.

[zanieb]

Here's a hacky patch that gets to the next failure :) https://github.com/zanieb/python-build-standalone/pull/5/commits/805c76fa7f7f162b63df83d7dda046417e88eb4c

[kpfleming]

I've just updated to 3.13.0b2, and removed the patch that was only needed until that release.

[zanieb]

The next problem I'm encountering is during validation:

validating dist/cpython-3.13.0b1-armv7-unknown-linux-gnueabihf-lto-20240605T2025.tar.zst
Error: errors found
  error: extra/unknown extension module: _interpchannels
  error: extra/unknown extension module: _interpqueues
  error: extra/unknown extension module: _interpreters
  error: extra/unknown extension module: _suggestions
  error: extra/unknown extension module: _sysconfig
  error: extra/unknown extension module: _testexternalinspection
  error: missing extension module: _crypt
  error: missing extension module: _xxinterpchannels
  error: missing extension module: _xxsubinterpreters
  error: missing extension module: audioop
  error: missing extension module: ossaudiodev
  error: missing extension module: spwd
  error: PYTHON.json referenced object file not in tar archive: python/build/Modules/_testinternalcapi/pytime.o
  error: PYTHON.json referenced object file not in tar archive: python/build/Modules/_testinternalcapi/set.o
  error: PYTHON.json referenced object file not in tar archive: python/build/Modules/_testinternalcapi/test_critical_sections.o
  error: PYTHON.json referenced object file not in tar archive: python/build/Modules/_testinternalcapi/test_lock.o

which is confusing because at least some of these look correct in extension-modules.yml

Edit: Resolved now.

[kpfleming]

I left a FIXME in validation.rs because it will need to be updated to have the proper list. This is why :)

[zanieb]

Great I'll take a swing at updating that.

[zanieb]

Now I have a spicy failure on just the PGO builds

2024-06-06T14:41:13.8509515Z cpython-3.13> LD_LIBRARY_PATH=/build/Python-3.13.0b1 ./python -m test --pgo --timeout=
2024-06-06T14:41:14.4721772Z cpython-3.13> Using random seed: 3630533237
2024-06-06T14:41:14.4722931Z cpython-3.13> 0:00:00 load avg: 2.53 Run 44 tests sequentially
2024-06-06T14:41:14.4723987Z cpython-3.13> 0:00:00 load avg: 2.53 [ 1/44] test_array
2024-06-06T14:41:15.9231051Z cpython-3.13> 0:00:01 load avg: 2.41 [ 2/44] test_base64
2024-06-06T14:41:18.6740282Z cpython-3.13> 0:00:04 load avg: 2.41 [ 3/44] test_binascii
2024-06-06T14:41:18.7698062Z cpython-3.13> 0:00:04 load avg: 2.41 [ 4/44] test_binop
2024-06-06T14:41:18.8463920Z cpython-3.13> 0:00:04 load avg: 2.41 [ 5/44] test_bisect
2024-06-06T14:41:18.9465060Z cpython-3.13> 0:00:04 load avg: 2.41 [ 6/44] test_bytes
2024-06-06T14:41:19.0936489Z cpython-3.13> 0:00:04 load avg: 2.41 [ 7/44] test_bz2 -- test_bytes skipped
2024-06-06T14:41:22.1080773Z cpython-3.13> 0:00:07 load avg: 2.30 [ 8/44] test_cmath
2024-06-06T14:41:22.2968922Z cpython-3.13> 0:00:07 load avg: 2.30 [ 9/44] test_codecs
2024-06-06T14:41:23.5730666Z cpython-3.13> 0:00:09 load avg: 2.30 [10/44] test_collections
2024-06-06T14:41:24.3534783Z cpython-3.13> 0:00:09 load avg: 2.30 [11/44] test_complex
2024-06-06T14:41:24.6535516Z cpython-3.13> 0:00:10 load avg: 2.30 [12/44] test_dataclasses
2024-06-06T14:41:25.2374900Z cpython-3.13> 0:00:10 load avg: 2.30 [13/44] test_datetime
2024-06-06T14:41:29.2557137Z cpython-3.13> 0:00:14 load avg: 2.19 [14/44] test_decimal
2024-06-06T14:41:32.8490327Z cpython-3.13> 0:00:18 load avg: 2.10 [15/44] test_difflib
2024-06-06T14:41:33.7806220Z cpython-3.13> 0:00:19 load avg: 2.10 [16/44] test_embed
2024-06-06T14:41:45.9800425Z cpython-3.13> Fatal Python error: Segmentation fault
2024-06-06T14:41:45.9801940Z cpython-3.13> 
2024-06-06T14:41:45.9803753Z cpython-3.13> Current thread 0x00007f1ac0fbd740 (most recent call first):
2024-06-06T14:41:45.9805390Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/test_embed.py", line 1620 in test_global_pathconfig
2024-06-06T14:41:45.9806989Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/unittest/case.py", line 606 in _callTestMethod
2024-06-06T14:41:45.9808342Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/unittest/case.py", line 651 in run
2024-06-06T14:41:45.9811098Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/unittest/case.py", line 707 in __call__
2024-06-06T14:41:45.9812381Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/unittest/suite.py", line 122 in run
2024-06-06T14:41:45.9813849Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/unittest/suite.py", line 84 in __call__
2024-06-06T14:41:45.9815235Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/unittest/suite.py", line 122 in run
2024-06-06T14:41:45.9816564Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/unittest/suite.py", line 84 in __call__
2024-06-06T14:41:45.9818271Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/testresult.py", line 146 in run
2024-06-06T14:41:45.9819125Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/single.py", line 57 in
2024-06-06T14:41:45.9819693Z cpython-3.13> _run_suite
2024-06-06T14:41:45.9820311Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/single.py", line 37 in run_unittest
2024-06-06T14:41:45.9820929Z cpython-3.13>   File
2024-06-06T14:41:45.9821502Z cpython-3.13> "/build/Python-3.13.0b1/Lib/test/libregrtest/single.py", line 132 in test_func
2024-06-06T14:41:45.9822423Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/single.py", line 88 in regrtest_runner
2024-06-06T14:41:45.9823610Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/single.py", line 135 in _load_run_test
2024-06-06T14:41:45.9824601Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/single.py", line 178 in _runtest_env_changed_exc
2024-06-06T14:41:45.9825561Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/single.py", line 278 in _runtest
2024-06-06T14:41:45.9826480Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/single.py", line 309 in run_single_test
2024-06-06T14:41:45.9827378Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/main.py", line 355 in run_test
2024-06-06T14:41:45.9828320Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/main.py", line 389 in run_tests_sequentially
2024-06-06T14:41:45.9829253Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/main.py", line 533 in _run_tests
2024-06-06T14:41:45.9830257Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/main.py", line 568 in run_tests
2024-06-06T14:41:45.9831155Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/main.py", line 731 in main
2024-06-06T14:41:45.9831996Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/libregrtest/main.py", line 739 in main
2024-06-06T14:41:45.9832875Z cpython-3.13>   File "/build/Python-3.13.0b1/Lib/test/__main__.py", line 2 in <module>
2024-06-06T14:41:45.9833528Z cpython-3.13>   File "<frozen runpy>", line 88 in _run_code
2024-06-06T14:41:45.9834110Z cpython-3.13>   File "<frozen runpy>", line 198 in _run_module_as_main
2024-06-06T14:41:45.9834656Z cpython-3.13> Extension modules: _testinternalcapi
2024-06-06T14:41:45.9835059Z cpython-3.13> (total: 1)
2024-06-06T14:41:46.1367157Z cpython-3.13> Segmentation fault (core dumped)
2024-06-06T14:41:46.1368488Z cpython-3.13> Makefile:868: recipe for target 'profile-bolt-stamp' failed
2024-06-06T14:41:46.1369458Z cpython-3.13> make[1]: Leaving directory '/build/Python-3.13.0b1'
2024-06-06T14:41:46.1370196Z cpython-3.13> make[1]: *** [profile-bolt-stamp] Error 139
2024-06-06T14:41:46.1380350Z cpython-3.13> make: *** [bolt-opt] Error 2
2024-06-06T14:41:46.1381212Z cpython-3.13> Makefile:899: recipe for target 'bolt-opt' failed
2024-06-06T14:41:48.0010999Z Traceback (most recent call last):
2024-06-06T14:41:48.0021028Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1219, in <module>
2024-06-06T14:41:48.0021814Z     sys.exit(main())
2024-06-06T14:41:48.0022074Z              ^^^^^^
2024-06-06T14:41:48.0023383Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1201, in main
2024-06-06T14:41:48.0024527Z     build_cpython(
2024-06-06T14:41:48.0025830Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 803, in build_cpython
2024-06-06T14:41:48.0027150Z     build_env.run("build-cpython.sh", environment=env)
2024-06-06T14:41:48.0028525Z   File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/buildenv.py", line 95, in run
2024-06-06T14:41:48.0029426Z     container_exec(self.container, program, user=user, environment=environment)
2024-06-06T14:41:48.0030512Z   File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/docker.py", line 140, in container_exec
2024-06-06T14:41:48.0031434Z     raise Exception("exit code %d from %s" % (inspect_res["ExitCode"], command))
2024-06-06T14:41:48.0032032Z Exception: exit code 2 from /build/build-cpython.sh
2024-06-06T14:41:48.0307350Z make: *** [Makefile:325: /home/runner/work/python-build-standalone/python-build-standalone/build/cpython-3.13.0b1-x86_64-unknown-linux-gnu-pgo.tar] Error 1
2024-06-06T14:41:48.0444158Z ##[error]Process completed with exit code 2.

https://github.com/python/cpython/blob/2268289a47c6e3c9a220b53697f9480ec390466f/Lib/test/test_embed.py#L1620

[zanieb]

And this error on the MUSL builds

2024-06-06T15:22:59.7323527Z cpython-3.13> checking for the platform triplet based on compiler characteristics...
2024-06-06T15:22:59.7534322Z cpython-3.13> x86_64-linux-musl
2024-06-06T15:22:59.7544473Z cpython-3.13> checking for multiarch...
2024-06-06T15:22:59.7721531Z cpython-3.13> configure: error: internal configure error for the platform triplet, please file a bug report
2024-06-06T15:23:03.4816270Z Traceback (most recent call last):
2024-06-06T15:23:03.4828543Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1219, in <module>
2024-06-06T15:23:03.4829764Z     sys.exit(main())
2024-06-06T15:23:03.4830118Z              ^^^^^^
2024-06-06T15:23:03.4831310Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1201, in main
2024-06-06T15:23:03.4832339Z     build_cpython(
2024-06-06T15:23:03.4833609Z   File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 803, in build_cpython
2024-06-06T15:23:03.4834795Z     build_env.run("build-cpython.sh", environment=env)
2024-06-06T15:23:03.4835741Z   File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/buildenv.py", line 95, in run
2024-06-06T15:23:03.4836594Z     container_exec(self.container, program, user=user, environment=environment)
2024-06-06T15:23:03.4837611Z   File "/home/runner/work/python-build-standalone/python-build-standalone/pythonbuild/docker.py", line 140, in container_exec
2024-06-06T15:23:03.4838520Z     raise Exception("exit code %d from %s" % (inspect_res["ExitCode"], command))
2024-06-06T15:23:03.4839101Z Exception: exit code 1 from /build/build-cpython.sh
2024-06-06T15:23:03.5124987Z make: *** [Makefile:325: /home/runner/work/python-build-standalone/python-build-standalone/build/cpython-3.13.0b1-x86_64_v3-unknown-linux-musl-debug.tar] Error 1
2024-06-06T15:23:03.5261193Z ##[error]Process completed with exit code 2.

https://github.com/python/cpython/blob/78634cfa3dd4b542897835d5f097604dbeb0f3fd/configure.ac#L1103