Experimental support for Python 3.10 in Sage 9.5

slel commented 4 years ago

This is to support Python 3.10, which was released on 2021-10-04. https://docs.python.org/3.10/whatsnew/changelog.html#changelog

Thanks to the following tickets, Sage is mostly ready for Python 3.10 in addition to 3.9, 3.8, 3.7.

needs Cython 0.29.23 (#31445)
31855 Patch/upgrade gmpy2 to add python 3.10 support
31856 Upgrade pyzmq, babel - to fix error: implicit declaration of function 'PyObject_AsWriteBuffer' is invalid in C99 with pyzmq-19.0.2
32815 numpy
32837 cffi: Update to 1.15.0
32852 Update traitlets to 5.1.1 (for python 3.9.8, 3.10)
32671 Update pip to 21.3.1, distlib to 0.3.3
31295 Meta-ticket: Replace imports from deprecated distutils
32930 IPython/Jupyter upgrade for python 3.10 support
33013 Fix tests with Python 3.10

In this ticket, we enable use of system Python 3.10 but issue a warning regarding its experimental status for Sage.

The ticket does not include the upgrade of the python3 SPKG to Python 3.10. Many systems already ship Python 3.10. To test, thanks to https://wiki.sagemath.org/ReleaseTours/sage-9.5#Separate_virtual_environment_for_Python_packages, a full rebuild of the Sage distribution is not necessary - only the Python packages will be built from scratch in a separate venv.

PEP 619 -- Python 3.10 Release Schedule
What's new in Python 3.10
30767: Upgrade to Python 3.10
30184: Support Python 3.9
30384: Adopt the “time window-based” policy for support of Python versions from NEP 29

Depends on #33013 Depends on #32930 Depends on #33040

CC: @mkoeppe @slel @kiwifb @tornaria @antonio-rojas

Component: packages: standard

Keywords: upgrade, python

Author: Matthias Koeppe, Gonzalo Tornaría

Branch: df9f1d4

Reviewer: Antonio Rojas, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/30766

mkoeppe commented 3 years ago

comment:40

Tests (via #30767) run at https://github.com/mkoeppe/sage/actions/runs/1239984523 -- all -minimal builds will use python 3.10.0rc1

mkoeppe commented 3 years ago

Changed dependencies from #32488, #30768, #31855, #32492, #32518, #32519 to #32488, #30768, #31855, #32492, #32518, #32519, #32520

mkoeppe commented 3 years ago

comment:42

distutils deprecation -> #32565:

sage -t --long --random-seed=0 src/sage/plot/plot.py
**********************************************************************
File "src/sage/plot/plot.py", line 513, in sage.plot.plot
Failed example:
    os.system("sage -c \"if 'matplotlib' in sys.modules: sys.exit(1)\"") # long time
Expected:
    0
Got:
    /Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10.0rc1/lib/python3.10/site-packages/sage/features/__init__.py:55: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
      from distutils.errors import CCompilerError
    0

mkoeppe commented 3 years ago

comment:43

Many failures from improved error messages -> #32520

sage -t --long --random-seed=0 src/sage/geometry/polyhedron/base.py
**********************************************************************
File "src/sage/geometry/polyhedron/base.py", line 214, in sage.geometry.polyhedron.base.Polyhedron_base.__init__
Failed example:
    p = Polyhedron_field(parent, Vrep, 'nonsense',  # py3
                         Vrep_minimal=True, Hrep_minimal=True, pref_rep='Vrep')
Expected:
    Traceback (most recent call last):
    ...
    TypeError: _init_Hrepresentation() takes 3 positional arguments but 9 were given
Got:
    <BLANKLINE>
    Traceback (most recent call last):
...
    TypeError: Polyhedron_field._init_Hrepresentation() takes 3 positional arguments but 9 were given
**********************************************************************

mkoeppe commented 3 years ago

comment:44

Stricter parser:

sage -t --long --random-seed=0 src/sage/schemes/riemann_surfaces/riemann_surface.py
**********************************************************************
File "src/sage/schemes/riemann_surfaces/riemann_surface.py", line 1393, in sage.schemes.riemann_surfaces.riemann_surface.RiemannSurface.make_zw_interpolator
Failed example:
    all(f(*g(i*0.1)).abs() < 1e-13for i in range(10))
Expected:
    True
Got:
    DeprecationWarning: invalid decimal literal
    True

mkoeppe commented 3 years ago

comment:45

Sage numbers no longer accepted:

sage -t --long --random-seed=0 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 104, in sage.tests.cmdline.test_executable
Failed example:
    (out, err, ret) = test_executable(["sleep", "1"], timeout=0.1)
Expected:
    Traceback (most recent call last):
    ...
    RuntimeError: timeout in test_executable()
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10.0rc1/lib/python3.10/site-packages/sage/doctest/forker.py", line 704, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10.0rc1/lib/python3.10/site-packages/sage/doctest/forker.py", line 1098, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.tests.cmdline.test_executable[5]>", line 1, in <module>
        (out, err, ret) = test_executable(["sleep", "1"], timeout=RealNumber('0.1'))
      File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10.0rc1/lib/python3.10/site-packages/sage/tests/cmdline.py", line 769, in test_executable
        rlist = select.select(rfd, [], [], timeout)[0]
    TypeError: timeout must be a float or None

sage -t --long --random-seed=0 src/sage_docbuild/utils.py
**********************************************************************
File "src/sage_docbuild/utils.py", line 91, in sage_docbuild.utils
Failed example:
    build_many(target, range(8), processes=8)
Expected:
    Traceback (most recent call last):
    ...
    ZeroDivisionError: rational division by zero
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10.0rc1/lib/python3.10/site-packages/sage/doctest/forker.py", line 704, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10.0rc1/lib/python3.10/site-packages/sage/doctest/forker.py", line 1098, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_docbuild.utils[6]>", line 1, in <module>
        build_many(target, range(Integer(8)), processes=Integer(8))
      File "/Users/mkoeppe/s/sage/sage-rebasing/local/var/lib/sage/venv-python3.10.0rc1/lib/python3.10/site-packages/sage_docbuild/utils.py", line 289, in build_many
        raise worker_exc.original_exception
    TypeError: 'sage.rings.real_mpfr.RealLiteral' object cannot be interpreted as an integer

mkoeppe commented 3 years ago

comment:46

inspect:

sage -t --long --random-seed=0 src/sage/misc/sageinspect.py
**********************************************************************
File "src/sage/misc/sageinspect.py", line 1561, in sage.misc.sageinspect.sage_getargspec
Failed example:
    shell.run_cell('f??')
Expected:
    ...the source code string...
Got:
    ---------------------------------------------------------------------------
    OSError                                   Traceback (most recent call last)

mkoeppe commented 3 years ago

comment:47

sage -t --long --random-seed=0 src/sage/categories/loop_crystals.py
**********************************************************************
File "src/sage/categories/loop_crystals.py", line 115, in sage.categories.loop_crystals.LoopCrystals.ParentMethods.digraph
Failed example:
    G.latex_options()  # optional - dot2tex
Expected:
    LaTeX options for Digraph on 29 vertices:
    {...'edge_options': <function ... at ...>...}
Got:
    LaTeX options for Digraph on 29 vertices: {}

mkoeppe commented 3 years ago

comment:48

sage -t --long --random-seed=0 src/sage/cpython/atexit.pyx
**********************************************************************
File "src/sage/cpython/atexit.pyx", line 61, in sage.cpython.atexit
Failed example:
    with restore_atexit(clear=True):
        atexit._run_exitfuncs()  # Should be none registered
        atexit.register(handler, 1, 2, c=3)
        with restore_atexit():
            atexit._run_exitfuncs()  # Run just registered handler
        atexit._run_exitfuncs()  # Handler should be run again
Expected:
    <function handler at 0x...>
    ((1, 2), {'c': 3})
    ((1, 2), {'c': 3})
Got:
    <function handler at 0x357023a30>
    ((1, 2), {'c': 3})

mkoeppe commented 3 years ago

Changed dependencies from #32488, #30768, #31855, #32492, #32518, #32519, #32520 to #32488, #30768, #31855, #32492, #32518, #32519, #32520, #32565

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from 2749e67 to ab26562

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

ab26562 Merge tag '9.5.beta2' into t/30766/support_python_3_10

mkoeppe commented 3 years ago

Changed dependencies from #32488, #30768, #31855, #32492, #32518, #32519, #32520, #32565 to #31855, #32519, #32520, #32565

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from ab26562 to bc2b01d

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`964bf64`	`git grep -l 'TypeError: .() takes' \| xargs sed -i.bak 's/TypeError: \(.\)() takes/TypeError: ...\1() takes/'`
`247a077`	`git grep -l 'TypeError: .() got an unexpected' \| xargs sed -i.bak 's/TypeError: \(.\)() got an unexpected/TypeError: ...\1() got an unexpected/'`
`82d9329`	`git grep -l 'TypeError: .() missing' \| xargs sed -i.bak 's/TypeError: \(.\)() missing/TypeError: ...\1() missing/'`
`e80b828`	`git grep -l 'TypeError: float() argument must be.' \| xargs sed -i.bak 's/TypeError: float() argument must be./TypeError: float() argument must be a string or a... number.../'`
`98a1cee`	`git grep -l 'AttributeError: can.* set attribute.' \| xargs sed -i.bak 's/AttributeError: can. set attribute.*/AttributeError: can...t set attribute.../'`
`8511141`	`git grep -l 'SyntaxError: invalid.' \| xargs sed -i.bak 's/SyntaxError: invalid./SyntaxError: invalid .../'`
`65c586b`	`Fix 2 more doctests to accept exception messages reworded in python 3.10`
`288bc84`	`Merge branch 'public/32521' of git://trac.sagemath.org/sage into t/32520/fix_doctests_to_accept_exception_messages_reworded_in_python_3_10`
`68d07f8`	`Merge tag '9.5.beta2' into t/32520/fix_doctests_to_accept_exception_messages_reworded_in_python_3_10`
`bc2b01d`	`Merge #32520`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Changed commit from bc2b01d to c73bc97

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 3 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`20e60f2`	`sage.features.Executable: Remove use of distutils`
`8c763f4`	`sage.features.CCompilerError: Move global import from distutils.errors into method`
`30ea064`	`Merge #32565`
`380b63a`	`build/pkgs/cvxopt: Update to 1.2.7 for Python 3.10 compatibility`
`c73bc97`	`Merge #32519`

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,5 @@
-This is to support Python 3.10.
-- Python 3.10.0rc1 released 2021-08-11.
-- Python 3.10.0 expected release: 2021-10-04.
+This is to support Python 3.10, which was released on 2021-10-04.
+https://docs.python.org/3.10/whatsnew/changelog.html#changelog

 This ticket is for the patches necessary to support
 Python 3.10 in addition to 3.9, 3.8, 3.7.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -7,7 +7,8 @@
 - needs Cython 0.29.23 (#31445)
 - #31855 Patch/upgrade `gmpy2` to add python 3.10 support
 - #31856 Upgrade `pyzmq`, `babel` - to fix `error: implicit declaration of function 'PyObject_AsWriteBuffer' is invalid in C99` with pyzmq-19.0.2 
-- `numpy`
+- #32815 `numpy`
+- #32837 `cffi`: Update to 1.15.0

 It does not include the actual upgrade to Python 3.10.
 To test, see #30767.

tornaria commented 3 years ago

comment:58

I'm testing c73bc97 using system python 3.10 (void linux).

Compilation succeeds. Then running ./sage gives:

/opt/sage/sage-git/local/lib/python3.10/site-packages/prompt_toolkit/application/application.py:882: DeprecationWarning: There is no current event loop
  loop = get_event_loop()
sage: ...

Running ./sage -tp 8 --all gives:

lots of crashes apparently related to cysignals; stack trace starts with

/opt/sage/sage-git/local/lib/python3.10/site-packages/cysignals/signals.cpython-310-x86_64-linux-gnu.so(+0x7e30)[0x7f2c06dbae30]
/opt/sage/sage-git/local/lib/python3.10/site-packages/cysignals/signals.cpython-310-x86_64-linux-gnu.so(+0x7ee9)[0x7f2c06dbaee9]
/opt/sage/sage-git/local/lib/python3.10/site-packages/cysignals/signals.cpython-310-x86_64-linux-gnu.so(+0xafd5)[0x7f2c06dbdfd5]
/usr/lib/libc.so.6(+0x3d000)[0x7f2c07fcb000]
/opt/sage/sage-git/local/lib/python3.10/site-packages/sage/cpython/atexit.cpython-310-x86_64-linux-gnu.so(+0x5131)[0x7f2c073e1131]
/opt/sage/sage-git/local/lib/python3.10/site-packages/sage/cpython/atexit.cpython-310-x86_64-linux-gnu.so(+0x5932)[0x7f2c073e1932]
/usr/lib/libpython3.10.so.1.0(+0xec50e)[0x7f2c0824050e]
/usr/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x47d8)[0x7f2c081bdd68]
...

more deprecation warnings

/opt/sage/sage-git/local/bin/cysignals-CSI:44: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

All I'm doing is, starting with a clean copy of c73bc97

$ ./bootstrap
$ ./configure
$ make build -j12
$ ./sage -tp 8 --all

I have logs, let me know if you need more info.

My goal is packaging sage for void linux, but it's already migrated to python 3.10 so this is a blocker.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -9,6 +9,7 @@
 - #31856 Upgrade `pyzmq`, `babel` - to fix `error: implicit declaration of function 'PyObject_AsWriteBuffer' is invalid in C99` with pyzmq-19.0.2 
 - #32815 `numpy`
 - #32837 `cffi`: Update to 1.15.0
+- #32852 Update `traitlets` to 5.1.1 (for python 3.9.8, 3.10)

 It does not include the actual upgrade to Python 3.10.
 To test, see #30767.

mkoeppe commented 3 years ago

Description changed:

--- 
+++ 
@@ -10,6 +10,7 @@
 - #32815 `numpy`
 - #32837 `cffi`: Update to 1.15.0
 - #32852 Update `traitlets` to 5.1.1 (for python 3.9.8, 3.10)
+- #32671 Update `pip` to 21.3.1, `distlib` to 0.3.3

 It does not include the actual upgrade to Python 3.10.
 To test, see #30767.

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -11,6 +11,7 @@
 - #32837 `cffi`: Update to 1.15.0
 - #32852 Update `traitlets` to 5.1.1 (for python 3.9.8, 3.10)
 - #32671 Update `pip` to 21.3.1, `distlib` to 0.3.3
+- #31295 Meta-ticket: Replace imports from deprecated `distutils`

 It does not include the actual upgrade to Python 3.10.
 To test, see #30767.

enriqueartal commented 2 years ago

comment:62

After updating the last cysignals (unreleased 1.11.0, needed for Fedora 35) I put below the failed tests. I get some errors as in previous posts (I needed to restart several times due to many cysignal crashes); by the way make ptestlong produced many such crashes while sage -t -a -p8 worked better.

The prompt_toolkit warning disappears with the 3.0.22 version.

sage -t --random-seed=186690325287508787735378550154836726084 src/sage/functions/log.py  # 3 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/functions/other.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/interfaces/fricas.py  # 110 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/misc/sageinspect.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/schemes/riemann_surfaces/riemann_surface.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/symbolic/integration/external.py  # 13 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/symbolic/integration/integral.py  # 3 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/tests/cmdline.py  # Timed out
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/calculus/calculus.py  # 6 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/calculus/desolvers.py  # 13 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/combinat/posets/posets.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/rings/real_mpfr.pyx  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/functions/gamma.py  # 4 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/matrix/matrix1.pyx  # 4 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/functions/error.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/plot/colors.py  # 5 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/geometry/hyperbolic_space/hyperbolic_model.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/symbolic/expression_conversions.py  # 16 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/functions/generalized.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/schemes/elliptic_curves/lseries_ell.py  # 2 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/structure/unique_representation.py  # 1 doctest failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage_docbuild/utils.py  # 2 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/libs/giac/__init__.py  # 7 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/lfunctions/lcalc.py  # 2 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/symbolic/callable.py  # 5 doctests failed
sage -t --random-seed=186690325287508787735378550154836726084 src/sage/cpython/atexit.pyx  # 6 doctests failed

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -12,6 +12,7 @@
 - #32852 Update `traitlets` to 5.1.1 (for python 3.9.8, 3.10)
 - #32671 Update `pip` to 21.3.1, `distlib` to 0.3.3
 - #31295 Meta-ticket: Replace imports from deprecated `distutils`
+- #32930 IPython/Jupyter upgrade for python 3.10 support

 It does not include the actual upgrade to Python 3.10.
 To test, see #30767.

antonio-rojas commented 2 years ago

comment:64

Replying to @tornaria:

Running ./sage -tp 8 --all gives:

lots of crashes apparently related to cysignals; stack trace starts with

/opt/sage/sage-git/local/lib/python3.10/site-packages/cysignals/signals.cpython-310-x86_64-linux-gnu.so(+0x7e30)[0x7f2c06dbae30]
/opt/sage/sage-git/local/lib/python3.10/site-packages/cysignals/signals.cpython-310-x86_64-linux-gnu.so(+0x7ee9)[0x7f2c06dbaee9]
/opt/sage/sage-git/local/lib/python3.10/site-packages/cysignals/signals.cpython-310-x86_64-linux-gnu.so(+0xafd5)[0x7f2c06dbdfd5]
/usr/lib/libc.so.6(+0x3d000)[0x7f2c07fcb000]
/opt/sage/sage-git/local/lib/python3.10/site-packages/sage/cpython/atexit.cpython-310-x86_64-linux-gnu.so(+0x5131)[0x7f2c073e1131]
/opt/sage/sage-git/local/lib/python3.10/site-packages/sage/cpython/atexit.cpython-310-x86_64-linux-gnu.so(+0x5932)[0x7f2c073e1932]
/usr/lib/libpython3.10.so.1.0(+0xec50e)[0x7f2c0824050e]
/usr/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x47d8)[0x7f2c081bdd68]
...

Just updated to 3.10 and am also seeing this when running doctests, but not 100% of the times. Any idea where this comes from?

antonio-rojas commented 2 years ago

comment:65

The crash is happening in the __Pyx_INCREF call here

    /* "sage/cpython/atexit.pyx":188
 *         else:
 *             kwargs = {}
 *         exithandlers.append((<object>callback.func,             # <<<<<<<<<<<<<<
 *                                 <object>callback.args,
 *                                 kwargs))
 */
    __pyx_t_1 = PyTuple_New(3); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 188, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_1);
    __Pyx_INCREF(((PyObject *)__pyx_v_callback.func));
    __Pyx_GIVEREF(((PyObject *)__pyx_v_callback.func));
    PyTuple_SET_ITEM(__pyx_t_1, 0, ((PyObject *)__pyx_v_callback.func));

antonio-rojas commented 2 years ago

comment:66

Update: the segfault seems to happen when running tests iff there is another sage-runtests process running. After interrupting the doctests, the sage-runtests process takes a long time to get cleaned up, and if you try to run tests again in the meantime you will get the segfault.

antonio-rojas commented 2 years ago

comment:67

After #33013, the only remaining issues in sagelib are

sage -t --long /usr/lib/python3.10/site-packages/sage/tests/cmdline.py  # 3 doctests failed
sage -t --long /usr/lib/python3.10/site-packages/sage/doctest/test.py  # 16 doctests failed

which are caused by the spawned sage-runtests child processes segfaulting as mentioned above, and

sage -t --long /usr/lib/python3.10/site-packages/sage/cpython/atexit.pyx  # 6 doctests failed

which are likely related.

antonio-rojas commented 2 years ago

Changed dependencies from #31855, #32519, #32520, #32565 to #31855, #32519, #32520, #32565, #33013

tornaria commented 2 years ago

comment:68

Indeed, running tests in parallel breaks badly but running them not in parallel goes very far.

This is what I got with ./sage -t --all:

----------------------------------------------------------------------
sage -t --random-seed=231622868948624456778064911551695162592 src/sage/tests/cmdline.py  # 5 doctests failed
sage -t --random-seed=231622868948624456778064911551695162592 src/sage/quivers/algebra_elements.pxi  # 1 doctest failed
sage -t --random-seed=231622868948624456778064911551695162592 src/sage/cpython/atexit.pyx  # 6 doctests failed
sage -t --random-seed=231622868948624456778064911551695162592 src/sage/interfaces/sage0.py  # 1 doctest failed
sage -t --random-seed=231622868948624456778064911551695162592 src/sage_docbuild/utils.py  # 2 doctests failed
----------------------------------------------------------------------

Maybe I missed some patch, e.g a few failures are distutils deprecation warnings in .../venv-python3.10/lib/python3.10/site-packages/matplotlib/__init__.py:88.

What I used is beta8 + #30766 + #33013. What did I miss?

Maybe someone can rebase on top of beta8 so we all work on top of a common point?

As for the failure: it seems the atexit module internals have changed in python3.10, a quick look seems to suggest that per-module state was changed to per-thread state; sage.cpython.atexit is probably doing something wrong which manifests when there is more than one thread.

On the positive side, now the structs and some functions are declared in (internal) headers so it might be possible to implement this in a more robust way (formerly everything was in a C file and it seems sage.python.atexit copied a bunch of stuff which now may be incorrect).

What are the chances that sagemath 9.5 can support system python 3.10? I've been working to get sagemath into voidlinux but they won't accept a vendored python.

mkoeppe commented 2 years ago

comment:69

I would suggest that we defer full support for Python 3.10 to Sage 9.6 but already accept system Python 3.10 with a warning

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`e65ae6c`	`build/pkgs/python3/spkg-configure.m4: Accept system python 3.10.x`
`59173d0`	`pkgs/sage-setup/setup.cfg: Allow Python 3.10`
`c902b7b`	`src/setup.cfg.m4: Allow Python 3.10`
`cb30552`	`build/pkgs/python3/spkg-configure.m4: Warn on system python 3.10.x`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from c73bc97 to cb30552

mkoeppe commented 2 years ago

Author: Matthias Koeppe

mkoeppe commented 2 years ago

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1239984523 to none

mkoeppe commented 2 years ago

Changed dependencies from #31855, #32519, #32520, #32565, #33013 to #33013, #32930

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`5e144e9`	`build/pkgs/jedi: Update to 0.18.1`
`6347f98`	`build/pkgs/prompt_toolkit: Update to 3.0.22`
`28e535d`	`build/pkgs/widgetsnbextension: Update to 3.5.2`
`63078ac`	`build/pkgs/terminado: Update to 0.12.1`
`22de5dc`	`build/pkgs/notebook: Update to 6.4.6`
`a0fe18b`	`build/pkgs/nbclient: Update to 0.5.4`
`ef3fc7f`	`build/pkgs/jupyter_client: Update to 7.1.0`
`a593bc9`	`build/pkgs/jupyter_core: Update to 4.9.1`
`66c16b1`	`build/pkgs/importlib_metadata: Update to 4.8.2`
`02aa94c`	`Merge #32930`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from cb30552 to 02aa94c

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,7 +1,7 @@
 This is to support Python 3.10, which was released on 2021-10-04.
 https://docs.python.org/3.10/whatsnew/changelog.html#changelog

-This ticket is for the patches necessary to support
+Thanks to the following tickets, Sage is mostly ready for 
 Python 3.10 in addition to 3.9, 3.8, 3.7.

 - needs Cython 0.29.23 (#31445)
@@ -13,15 +13,17 @@
 - #32671 Update `pip` to 21.3.1, `distlib` to 0.3.3
 - #31295 Meta-ticket: Replace imports from deprecated `distutils`
 - #32930 IPython/Jupyter upgrade for python 3.10 support
+- #33013 Fix tests with Python 3.10

-It does not include the actual upgrade to Python 3.10.
-To test, see #30767.
+In this ticket, we enable use of system Python 3.10 but issue a warning regarding its experimental status for Sage. 
+
+The ticket does not include the upgrade of the `python3` SPKG to Python 3.10.  Many systems already ship Python 3.10.  To test, thanks to https://wiki.sagemath.org/ReleaseTours/sage-9.5#Separate_virtual_environment_for_Python_packages, a full rebuild of the Sage distribution is not necessary - only the Python packages will be  built from scratch in a separate venv.

 Related:

 - [PEP 619 -- Python 3.10 Release Schedule](https://www.python.org/dev/peps/pep-0619/)
 - [What's new in Python 3.10](https://docs.python.org/3.10/whatsnew/3.10.html)
-- #30767: Test ticket: Python 3.10 development releases
+- #30767: Upgrade to Python 3.10
 - #30184: Support Python 3.9
 - #30384: Adopt the “time window-based” policy for support of Python versions from NEP 29

mkoeppe commented 2 years ago

comment:75

Replying to @tornaria:

Maybe someone can rebase [...] so we all work on top of a common point?

Done

mkoeppe commented 2 years ago

comment:76

Replying to @tornaria:

As for the failure: it seems the atexit module internals have changed in python3.10, a quick look seems to suggest that per-module state was changed to per-thread state; sage.cpython.atexit is probably doing something wrong which manifests when there is more than one thread.

There is an unreleased change related to atexit mentioned in https://docs.python.org/3.10/whatsnew/changelog.html#core-and-builtins

mkoeppe commented 2 years ago

comment:77

I've put this change as a patch on #30767. It looks like it fixes the crashes in make ptest

mkoeppe commented 2 years ago

comment:78

However, I get these failures:

sage -t --random-seed=335595092046712014150830110238612394725 src/sage/cpython/atexit.pyx
**********************************************************************
File "src/sage/cpython/atexit.pyx", line 61, in sage.cpython.atexit
Failed example:
    with restore_atexit(clear=True):
        atexit._run_exitfuncs()  # Should be none registered
        atexit.register(handler, 1, 2, c=3)
        with restore_atexit():
            atexit._run_exitfuncs()  # Run just registered handler
        atexit._run_exitfuncs()  # Handler should be run again
Expected:
    <function handler at 0x...>
    ((1, 2), {'c': 3})
    ((1, 2), {'c': 3})
Got:
    <function handler at 0x1727cc940>
    ((1, 2), {'c': 3})
**********************************************************************
File "src/sage/cpython/atexit.pyx", line 77, in sage.cpython.atexit
Failed example:
    with restore_atexit(clear=False, run=True):
        # original handlers are run when exiting the context
        pass
Expected:
    ((4, 5), {'d': 6})
    ((1, 2), {'c': 3})
Got:
    <BLANKLINE>
**********************************************************************
File "src/sage/cpython/atexit.pyx", line 85, in sage.cpython.atexit
Failed example:
    atexit._run_exitfuncs()
Expected:
    ((4, 5), {'d': 6})
    ((1, 2), {'c': 3})
Got:
    <BLANKLINE>
**********************************************************************
File "src/sage/cpython/atexit.pyx", line 97, in sage.cpython.atexit
Failed example:
    print("Initial exit handlers:\n{}".format(_get_exithandlers()))
Expected:
    Initial exit handlers:
    [(<function handler at 0x...>, (1, 2), {'c': 3}),
     (<function handler at 0x...>, (4, 5), {'d': 6})]
Got:
    Initial exit handlers:
    []
**********************************************************************
File "src/sage/cpython/atexit.pyx", line 104, in sage.cpython.atexit
Failed example:
    print("After restore_atexit:\n{}".format(_get_exithandlers()))
Expected:
    After restore_atexit:
    [(<function handler at 0x...>, (1, 2), {'c': 3}),
     (<function handler at 0x...>, (4, 5), {'d': 6})]
Got:
    After restore_atexit:
    []
**********************************************************************
File "src/sage/cpython/atexit.pyx", line 114, in sage.cpython.atexit
Failed example:
    print("After restore_atexit with clear=True:\n{}".format(
          _get_exithandlers()))
Expected:
    After restore_atexit with clear=True:
    [(<function handler at 0x...>, (1, 2), {'c': 3}),
     (<function handler at 0x...>, (4, 5), {'d': 6})]
Got:
    After restore_atexit with clear=True:
    []
**********************************************************************
1 item had failures:
   6 of  20 in sage.cpython.atexit
    [19 tests, 6 failures, 0.03 s]

antonio-rojas commented 2 years ago

comment:80

The python patch also fixes the crashes for me. Besides those six failures in sage.cpython.atexit, I'm also getting

**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/doctest/test.py", line 509, in sage.doctest.test
Failed example:
    os.path.isfile(F)  # long time
Expected:
    False
Got:
    True
**********************************************************************

Looks like atexit integration is totally broken, but it shouldn't affect normal Sage usage.

mkoeppe commented 2 years ago

comment:81

I think we should just disable all of sage.cpython.atexit on Python 3.10 for Sage 9.5

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

b5c7399 sage.cpython.atexit: Disable for Python >= 3.10.0

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 02aa94c to b5c7399

mkoeppe commented 2 years ago

comment:83

This fixes the segfaults for me with (unpatched) Python 3.10 from homebrew.

It looks like doctest timeouts are not correctly reported with this change on Python 3.10

sage -t --random-seed=15134756989416810321066706961638592844 src/sage/manifolds/differentiable/integrated_curve.py
    Error in doctesting framework (no result returned)

But I think this is good enough for Sage 9.5.

sheerluck commented 2 years ago

comment:84

atexit.pyx commit fixes the segfaults for tox-docker-gentoo-python3.10-standard

sagemath / sage

Experimental support for Python 3.10 in Sage 9.5 #30766

31855 Patch/upgrade `gmpy2` to add python 3.10 support

31856 Upgrade `pyzmq`, `babel` - to fix `error: implicit declaration of function 'PyObject_AsWriteBuffer' is invalid in C99` with pyzmq-19.0.2

32815 `numpy`

32837 `cffi`: Update to 1.15.0

32852 Update `traitlets` to 5.1.1 (for python 3.9.8, 3.10)

32671 Update `pip` to 21.3.1, `distlib` to 0.3.3

31295 Meta-ticket: Replace imports from deprecated `distutils`

32930 IPython/Jupyter upgrade for python 3.10 support

33013 Fix tests with Python 3.10

30767: Upgrade to Python 3.10

30184: Support Python 3.9

30384: Adopt the “time window-based” policy for support of Python versions from NEP 29

sagemath / sage

Experimental support for Python 3.10 in Sage 9.5 #30766

31855 Patch/upgrade gmpy2 to add python 3.10 support

31856 Upgrade pyzmq, babel - to fix error: implicit declaration of function 'PyObject_AsWriteBuffer' is invalid in C99 with pyzmq-19.0.2

32815 numpy

32837 cffi: Update to 1.15.0

32852 Update traitlets to 5.1.1 (for python 3.9.8, 3.10)

32671 Update pip to 21.3.1, distlib to 0.3.3

31295 Meta-ticket: Replace imports from deprecated distutils

32930 IPython/Jupyter upgrade for python 3.10 support

33013 Fix tests with Python 3.10

30767: Upgrade to Python 3.10

30184: Support Python 3.9

30384: Adopt the “time window-based” policy for support of Python versions from NEP 29

31855 Patch/upgrade `gmpy2` to add python 3.10 support

31856 Upgrade `pyzmq`, `babel` - to fix `error: implicit declaration of function 'PyObject_AsWriteBuffer' is invalid in C99` with pyzmq-19.0.2

32815 `numpy`

32837 `cffi`: Update to 1.15.0

32852 Update `traitlets` to 5.1.1 (for python 3.9.8, 3.10)

32671 Update `pip` to 21.3.1, `distlib` to 0.3.3

31295 Meta-ticket: Replace imports from deprecated `distutils`