search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.44k stars 480 forks source link

Upgrade python to 3.11 #33842

Closed mkoeppe closed 1 year ago

mkoeppe commented 2 years ago

Issues

Depends on #32423 Depends on #33530 Depends on #33878 Depends on #34795

CC: @kiwifb @antonio-rojas @nbruin @dimpase @jhpalmieri

Component: packages: standard

Author: Gonzalo Tornaría, Matthias Koeppe, Andrey Belgorodski

Branch/Commit: ac0105e

Reviewer: Matthias Koeppe, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/33842

mkoeppe commented 2 years ago

Branch: u/mkoeppe/test_ticket__python_3_11

mkoeppe commented 2 years ago
comment:2

It builds. First failure: building cython - we need to wait for a version with 3.11 support - https://github.com/cython/cython/commits/0.29.x

New commits:

3d3a5b7build/pkgs/python3: Update to 3.11.0b1
a2b955ebuild/pkgs/python3/patches/cygwin-socket-tcpnodelay-21649.patch: Remove, upstreamed
mkoeppe commented 2 years ago

Commit: a2b955e

mkoeppe commented 2 years ago
comment:3

Cython update in #33864

mkoeppe commented 2 years ago

Dependencies: #33864

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a2b955e to 694cab5

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

92e9cffbuild/pkgs/cython: Update to 0.29.30
694cab5Merge #33864
mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1 +1,3 @@
+Issues
+- https://github.com/sagemath/memory_allocator/issues/6
mkoeppe commented 2 years ago
comment:6
  [pyzmq-22.3.0]       #include "longintrepr.h"
  [pyzmq-22.3.0]                ^~~~~~~~~~~~~~~
  [pyzmq-22.3.0]     1 error generated.

same error as in memory_allocator

mkoeppe commented 2 years ago
comment:7

Fix is in 23.0.0 (still beta) https://pyzmq.readthedocs.io/en/latest/changelog.html https://pypi.org/project/pyzmq/#history

mkoeppe commented 2 years ago
comment:8

cypari2:

  gcc -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -I./cypari2 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/site-packages/cysignals -I/usr/local/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/include/python3.11 -c cypari2/convert.c -o build/temp.macosx-12.4-x86_64-3.11/cypari2/convert.o
  cypari2/convert.c:3347:21: error: cannot take the address of an rvalue of type 'Py_ssize_t' (aka 'long')
    __pyx_v_sizeptr = &Py_SIZE(((PyObject *)__pyx_v_x));
                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
mkoeppe commented 2 years ago

Changed dependencies from #33864 to #33864, #32423

mkoeppe commented 2 years ago
comment:10
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/var/tmp/sage/build/jupyter_jsmol-0.2.4/src/setupbase.py", line 630, in _compile_pattern
      return re.compile(res, flags=flags).match
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/__init__.py", line 225, in compile
      return _compile(pattern, flags)
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/__init__.py", line 273, in _compile
      p = _compiler.compile(pattern, flags)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_compiler.py", line 759, in compile
      p = _parser.parse(p, flags)
          ^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 979, in parse
      p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 454, in _parse_sub
      itemsappend(_parse(source, state, verbose, nested + 1,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 840, in _parse
      raise source.error('global flags not at the start '
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  re.error: global flags not at the start of the expression at position 52
  error: subprocess-exited-with-error
mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,3 +1,4 @@
 Issues
 - https://github.com/sagemath/memory_allocator/issues/6
+- https://github.com/sagemath/cypari2/issues/114
mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,5 @@
 Issues
 - https://github.com/sagemath/memory_allocator/issues/6
 - https://github.com/sagemath/cypari2/issues/114
+- https://github.com/fplll/fpylll/issues/230
mkoeppe commented 2 years ago

Changed dependencies from #33864, #32423 to #33864, #32423, #33866

mkoeppe commented 2 years ago

Changed dependencies from #33864, #32423, #33866 to #33864, #32423, #33866, #33566

malb commented 2 years ago
comment:15

This should be fixed in FPyLLL upstream. I can cut new packages?

mkoeppe commented 2 years ago
comment:16

Yes please!

kliem commented 2 years ago

Changed dependencies from #33864, #32423, #33866, #33566 to #33864, #32423, #33866, #33566, #33872

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

7619755Merge #32937
253a8f2Merge tag '9.6.rc4' into t/32423/update_numpy_to_1_22_x__scipy_1_8_x___requires_dropping_python_3_7
d188705Merge #33782
be053cbMerge #32423
ce9a905build/pkgs/jupyter_jsmol: Update to 2022.1.0
cea777cbuild/pkgs/jupyter_packaging: Make it a normal standard package
731424bMerge #33866
b75fd6etest memory_allocator 0.1.3a1
542326cMerge #33872
e61481ebuild/pkgs/fpylll: Update to 0.5.7
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 694cab5 to e61481e

malb commented 2 years ago
comment:19

See https://github.com/sagemath/sage-prod/issues/33913 for FP(y)LLL

mkoeppe commented 2 years ago

Changed dependencies from #33864, #32423, #33866, #33566, #33872 to #33864, #32423, #33866, #33566, #33872, #33878

mkoeppe commented 2 years ago

Changed dependencies from #33864, #32423, #33866, #33566, #33872, #33878 to #33864, #32423, #33866, #33566, #33872, #33878, #33913

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from e61481e to 39a71bc

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f3f1b5dtest new changes to pari
2f2584eMerge #33878
4f46253FPLLL 5.4.2 and FPyLLL 0.5.7
39a71bcMerge #33913
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

da1b912change more version numbers for FP(y)LLL
a623db7Merge #33913
8091a7bupdate memory allocator to 0.1.3
49cf595Merge #33872
481d782test cypari update
11a0e57build/bin/write-dockerfile.sh: Add bootstrap-conda
b494326Merge branch 'u/mkoeppe/fix_tox_docker_builds' of git://trac.sagemath.org/sage into public/33878
b826c17Merge #33878
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 39a71bc to b826c17

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

49a7ca5Merge #33866
eb586f1build/pkgs/soupsieve: Update to 2.3.2.post1
b0258f3build/pkgs/beautifulsoup4: Update to 4.11.1
7580e32build/pkgs/nbformat: Update to 5.4.0
58b4329build/pkgs/matplotlib/dependencies: Remove pytz
f216a8ebuild/pkgs/soupsieve/dependencies: Add hatchling
843d4b3build/pkgs/fastjsonschema: New, dep of nbformat
254694cMerge #33866
2a7342fsrc/sage/repl/display/jsmol_iframe.py: Update extension name following version 2022.1.0
0b71f8bMerge #33530
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from b826c17 to 0b71f8b

mkoeppe commented 2 years ago

Changed dependencies from #33864, #32423, #33866, #33566, #33872, #33878, #33913 to #32423, #33530, #33878

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 0b71f8b to b5ceaf0

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

f3f1b5dtest new changes to pari
481d782test cypari update
b494326Merge branch 'u/mkoeppe/fix_tox_docker_builds' of git://trac.sagemath.org/sage into public/33878
2e21858Merge #33878
d7923b7build/pkgs/kiwisolver: Update to 1.4.3
b5ceaf0build/pkgs/cppy: Update to 1.2.1
mkoeppe commented 2 years ago
comment:27

With 3.11.0b3, another problem with fpylll

  gcc -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -Isrc/fpylll/fplll -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/lib/python3.11/site-packages/cysignals -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/lib/python3.11/site-packages/numpy/core/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11 -c build/src/fpylll/fplll/integer_matrix.cpp -o build/temp.macosx-12.4-x86_64-cpython-311/build/src/fpylll/fplll/integer_matrix.o -std=c++11 -g -O2
  In file included from build/src/fpylll/fplll/integer_matrix.cpp:47:
  In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/Python.h:38:
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/pyport.h:47:24: error: cannot cast from type 'int' to pointer type '_object *'
                  return static_cast<type>(const_cast<expr_type &>(expr));
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  build/src/fpylll/fplll/integer_matrix.cpp:22142:9: note: in instantiation of function template specialization '(anonymous namespace)::_Py_CAST_impl<_object *, int>' requested here
          PyTuple_SET_ITEM(args, 0, 0);
          ^
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/cpython/tupleobject.h:40:49: note: expanded from macro 'PyTuple_SET_ITEM'
      PyTuple_SET_ITEM(_PyObject_CAST(op), index, _PyObject_CAST(value))
                                                  ^
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/object.h:107:28: note: expanded from macro '_PyObject_CAST'
  #define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
                             ^
  /Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/pyport.h:51:32: note: expanded from macro '_Py_CAST'
  #  define _Py_CAST(type, expr) _Py_CAST_impl<type>(expr)
                                 ^
  1 error generated.
  error: command '/usr/bin/gcc' failed with exit code 1
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

ef1dfa2Merge tag '9.7.beta5' into t/33842/test_ticket__python_3_11
4b0ba29Merge tag '9.7.beta6' into t/33842/test_ticket__python_3_11
7082f0ebuild/pkgs/python3: Update to 3.11.0b4
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from b5ceaf0 to 7082f0e

mkoeppe commented 2 years ago
comment:29

Replying to @mkoeppe:

With 3.11.0b3, another problem with fpylll

This problem has disappeared with 3.11.0b4

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 7082f0e to 74221f2

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

74221f2build/pkgs/python3: Update to 3.11.0b5
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

393d695build/pkgs/python3: Update to 3.11.0rc1
9cf7930Merge tag '9.7.beta8' into t/33842/test_ticket__python_3_11
a76a718Bump python_requires to >=3.8, <3.12
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 74221f2 to a76a718

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a76a718 to 4366cf2

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

4366cf2Merge tag '9.8.beta1' into t/33842/test_ticket__python_3_11
tornaria commented 2 years ago
comment:34

Patch for python-3.11 (see python-3.11.patch at https://github.com/void-linux/void-packages/commit/6229f313450ecae88743b4d5e99da2ed4de44e07):

--- a/src/sage/cpython/cython_metaclass.h
+++ b/src/sage/cpython/cython_metaclass.h
@@ -66,7 +66,7 @@
         }

         /* Now, set t.__class__ to metaclass */
-        Py_TYPE(t) = metaclass;
+        Py_SET_TYPE(t, metaclass);
         PyType_Modified(t);
     }
     else
--- a/src/sage/symbolic/ginac/numeric.cpp
+++ b/src/sage/symbolic/ginac/numeric.cpp
@@ -52,7 +52,6 @@
 #define register
 #define PY_SSIZE_T_CLEAN
 #include <Python.h>
-#include <longintrepr.h>
 #include "flint/fmpz.h"
 #include "flint/fmpz_factor.h"

--- a/src/sage/libs/gmp/pylong.pyx
+++ b/src/sage/libs/gmp/pylong.pyx
@@ -32,7 +32,7 @@
 from .mpz cimport *

 cdef extern from *:
-    Py_ssize_t* Py_SIZE_PTR "&Py_SIZE"(object)
+    void Py_SET_SIZE(object, Py_ssize_t)
     int hash_bits """
         #ifdef _PyHASH_BITS
         _PyHASH_BITS         /* Python 3 */
@@ -57,10 +57,8 @@
     mpz_export(L.ob_digit, NULL,
             -1, sizeof(digit), 0, PyLong_nails, z)
     if mpz_sgn(z) < 0:
-        # Set correct size (use a pointer to hack around Cython's
-        # non-support for lvalues).
-        sizeptr = Py_SIZE_PTR(L)
-        sizeptr[0] = -pylong_size
+        # Set correct size
+        Py_SET_SIZE(L, -pylong_size)
     return L
mkoeppe commented 2 years ago
comment:35

I think we need to conditionalize this - Py_SET_TYPE, Py_SET_SIZE were added in 3.9, but we still support 3.8

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 4366cf2 to 2df72df

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

99de8a3Merge tag '9.8.beta2' into t/33842/test_ticket__python_3_11
9042e6bbuild/pkgs/python3: Update to 3.11.0
a379ea3build/pkgs/cypari: Update to 2.1.3
34c5c94build/pkgs/cypari/patches/trashcan.patch: Remove
de38bacsrc/sage: Apply python-3.11.patch from https://github.com/void-linux/void-packages/commit/6229f313450ecae88743b4d5e99da2ed4de44e07
2df72dfbuild/pkgs/cython/patches/trashcan.patch: Remove
mkoeppe commented 2 years ago
comment:38

Next error (on startup):

  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/src/sage/misc/decorators.py", line 35, in <module>
    from inspect import ArgSpec
ImportError: cannot import name 'ArgSpec' from 'inspect' (/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0/lib/python3.11/inspect.py)
mkoeppe commented 2 years ago
comment:39

Also

  File "sage/misc/cachefunc.pyx", line 2821, in sage.misc.cachefunc.CachedMethod.__get__
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/src/sage/misc/sageinspect.py", line 1720, in sage_getargspec
    return inspect.ArgSpec(*_sage_getargspec_cython(source))
           ^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'ArgSpec'