search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.37k stars 466 forks source link

upgrade to ipython 7 #28197

Closed fchapoton closed 4 years ago

fchapoton commented 5 years ago

Upgrade ipython and related packages:

All of these package versions support python >= 3.6 (note #29033).

Tarballs: see checksums.ini [upstream_url]. (To configure Sage to download from the upstream URLs, use ./configure --enable-download-from-upstream-url)

CC: @embray @jdemeyer @slel @kiwifb @timokau @antonio-rojas

Component: packages: standard

Keywords: upgrade

Author: Jonathan Kliem, John Palmieri, Antonio Rojas

Branch: e21a7b0

Reviewer: John Palmieri, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/28197

fchapoton commented 5 years ago

Branch: public/ticket/28197

fchapoton commented 5 years ago

Description changed:

--- 
+++ 
@@ -1 +1,5 @@
 WARNING: this is python3-only
+
+* upgraded prompt-toolkit-2.0.9.tar.gz
+
+* pip-install backcall
fchapoton commented 5 years ago
comment:1

This is basically working, but maybe without preparsing..

New commits:

0d4f0c7adaptation to ipython 7 : first tentative
fchapoton commented 5 years ago

Commit: 0d4f0c7

jhpalmieri commented 5 years ago
comment:3

Should this depend on #28190?

fchapoton commented 5 years ago

Dependencies: #28190

jplab commented 5 years ago

Description changed:

--- 
+++ 
@@ -1,5 +1,19 @@
 WARNING: this is python3-only
+
+* Upgrade ipython to 7.6.1
+
+```
+$ wget -P upstream https://github.com/ipython/ipython/releases/download/7.6.1/ipython-7.6.1.tar.gz
+```

 * upgraded prompt-toolkit-2.0.9.tar.gz

+To download the tarball:
+
+```
+$ wget -P upstream --output-document=prompt-toolkit-2.0.9.tar.gz https://github.com/prompt-toolkit/python-prompt-toolkit/archive/2.0.9.tar.gz
+```
+
 * pip-install backcall
+
+
jplab commented 5 years ago

Description changed:

--- 
+++ 
@@ -16,4 +16,19 @@

 * pip-install backcall

+Current issues (8.9.beta3):

+```
+sage: installed_packages()['ipython']                                                                                                                                                                              
+'7.6.1'
+sage: 2^3                                                                                                                                                                                                          
+1
+sage: 2**3                                                                                                                                                                                                         
+8
+sage: n=4                                                                                                                                                                                                          
+sage: type(n)                                                                                                                                                                                                      
+<class 'int'>
+```
+
+The preparsing is not working.
+
jplab commented 5 years ago

Description changed:

--- 
+++ 
@@ -16,19 +16,3 @@

 * pip-install backcall

-Current issues (8.9.beta3):
-
-```
-sage: installed_packages()['ipython']                                                                                                                                                                              
-'7.6.1'
-sage: 2^3                                                                                                                                                                                                          
-1
-sage: 2**3                                                                                                                                                                                                         
-8
-sage: n=4                                                                                                                                                                                                          
-sage: type(n)                                                                                                                                                                                                      
-<class 'int'>
-```
-
-The preparsing is not working.
-
jplab commented 5 years ago
comment:7

Current issues (8.9.beta3):

sage: installed_packages()['ipython']                                                                                                                                                                              
'7.6.1'
sage: 2^3                                                                                                                                                                                                          
1
sage: 2**3                                                                                                                                                                                                         
8
sage: n=4                                                                                                                                                                                                          
sage: type(n)                                                                                                                                                                                                      
<class 'int'>

The preparsing is not working since it has been commented out.

jplab commented 5 years ago
comment:8

Trying to launch ipython one get:

jplabbe@blackbomb:~/sage$ sage -ipython
Traceback (most recent call last):
...
pkg_resources.DistributionNotFound: The 'jedi>=0.10' distribution was not found and is required by ipython

So one has to install the jedi module as well.

fchapoton commented 5 years ago

Description changed:

--- 
+++ 
@@ -16,3 +16,5 @@

 * pip-install backcall

+* pip-install jedi
+
antonio-rojas commented 5 years ago
comment:11

A patch that makes sage work with ipython 7, including the preparser, is available at [1]. I haven't bothered to make it backwards compatible with older ipython, and it probably needs some polishing, feel free to take it from there.

[1] https://aur.archlinux.org/cgit/aur.git/tree/sagemath-ipython7.patch?h=sagemath-python3-git

embray commented 5 years ago
comment:12

(For lack of a better component; perhaps we should add a "REPL" component...?)

jhpalmieri commented 4 years ago
comment:15

This is Python 3 only. Are there specific plans for when we drop Python 2 support from Sage? Version 9.1?

antonio-rojas commented 4 years ago
comment:16

ipython 7.10 breaks many more tests due to it no longer sorting dicts. Some tests output is actually random now. Updated patch at https://aur.archlinux.org/cgit/aur.git/tree/sagemath-ipython7.patch?h=sagemath-git

embray commented 4 years ago
comment:17

Ticket retargeted after milestone closed

mwageringel commented 4 years ago
comment:18

The new jedi completion engine seems to have some problems. For example with IPython 7.6.1, invoking tab completion like

sage: 1 + <TAB>

causes the interpreter to freeze for about 1.5 minutes (apparently, a largish cache is created at ~/.cache/jedi/). After that, several deprecated functions are imported into global scope causing deprecation warnings to be printed, such as:

Importing absolute_igusa_invariants_kohel from here is deprecated. If you need to use it, please import it directly from sage.schemes.hyperelliptic_curves.invariants
See https://trac.sagemath.org/28064 for details.
  return getattr(handle.access, attribute)(*args, **kwargs)

This also seems to slow down exiting Sage.

As for the display of dictionaries during doctests, I suggest to open a separate ticket to remove the sorting. This can be done independently from the IPython upgrade, once Python 2 support is dropped by Sage (in 9.1 I assume).

mwageringel commented 4 years ago

Description changed:

--- 
+++ 
@@ -11,7 +11,7 @@
 To download the tarball:

-$ wget -P upstream --output-document=prompt-toolkit-2.0.9.tar.gz https://github.com/prompt-toolkit/python-prompt-toolkit/archive/2.0.9.tar.gz +$ wget --output-document=upstream/prompt_toolkit-2.0.9.tar.gz https://github.com/prompt-toolkit/python-prompt-toolkit/archive/2.0.9.tar.gz


 * pip-install backcall
fchapoton commented 4 years ago
comment:19

a preparation step at #28948

antonio-rojas commented 4 years ago
comment:20

I've opened #29042 for the dict sorting changes

antonio-rojas commented 4 years ago

Changed dependencies from #28190 to #28190 #29042

kiwifb commented 4 years ago
comment:21

I have the following that I don't really know what to do with

sage -t --long /usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py", line 352, in sage.repl.ipython_extension.SageMagics.cython
Failed example:
    shell.run_cell('''
    %%cython
    def f():
        print('test')
    ''')
Expected nothing
Got:
    UsageError: Line magic function `%%cython` not found.
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py", line 357, in sage.repl.ipython_extension.SageMagics.cython
Failed example:
    f()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_extension.SageMagics.cython[3]>", line 1, in <module>
        f()
    NameError: name 'f' is not defined
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py", line 385, in sage.repl.ipython_extension.SageMagics.fortran
Failed example:
    shell.run_cell('''
    %%fortran
    C FILE: FIB1.F
          SUBROUTINE FIB(A,N)
    C
    C     CALCULATE FIRST N FIBONACCI NUMBERS
    C
          INTEGER N
          REAL*8 A(N)
          DO I=1,N
             IF (I.EQ.1) THEN
                A(I) = 0.0D0
             ELSEIF (I.EQ.2) THEN
                A(I) = 1.0D0
             ELSE
                A(I) = A(I-1) + A(I-2)
             ENDIF
          ENDDO
          END
    C END FILE FIB1.F
    ''')
Expected nothing
Got:
      File "<ipython-input-1-ab346d8f879f>", line 3
        C FILE: FIB1.F
             ^
    SyntaxError: invalid syntax
    <BLANKLINE>
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py", line 406, in sage.repl.ipython_extension.SageMagics.fortran
Failed example:
    fib
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_extension.SageMagics.fortran[3]>", line 1, in <module>
        fib
    NameError: name 'fib' is not defined
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py", line 410, in sage.repl.ipython_extension.SageMagics.fortran
Failed example:
    fib(a, 10)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_extension.SageMagics.fortran[6]>", line 1, in <module>
        fib(a, Integer(10))
    NameError: name 'fib' is not defined
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py", line 411, in sage.repl.ipython_extension.SageMagics.fortran
Failed example:
    a
Expected:
    array([  0.,   1.,   1.,   2.,   3.,   5.,   8.,  13.,  21.,  34.])
Got:
    array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
**********************************************************************
2 items had failures:
   2 of   5 in sage.repl.ipython_extension.SageMagics.cython
   4 of   9 in sage.repl.ipython_extension.SageMagics.fortran
    [80 tests, 6 failures, 2.14 s]
----------------------------------------------------------------------
sage -t --long /usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py  # 6 doctests failed

It all comes down from the first test failing because of the %%cython magic function not being found. Any clue as to what to do to fix that?

antonio-rojas commented 4 years ago
comment:22

Replying to @kiwifb:

It all comes down from the first test failing because of the %%cython magic function not being found. Any clue as to what to do to fix that?

Broken preparsing? it works fine here with the patch at comment:16

kiwifb commented 4 years ago
comment:23

Because I was using ipython-7.5 so far I was using the patch at comment:11 (with small variations between sage-9.0 and 9.1.beta). I'll check the preparsing in comment:16.

kiwifb commented 4 years ago
comment:24

I should have asked. When you said it is fine, is it sage-9.0 or 9.1.betaX? I am working on 9.1.beta9 right now.

antonio-rojas commented 4 years ago
comment:25

Replying to @kiwifb:

I should have asked. When you said it is fine, is it sage-9.0 or 9.1.betaX? I am working on 9.1.beta9 right now.

Both. The patch for 9.0 is at https://git.archlinux.org/svntogit/community.git/tree/trunk/sagemath-ipython7.patch?h=packages/sagemath

kiwifb commented 4 years ago
comment:26

Nope. I must be missing something subtle or something in my stack has a bug.

antonio-rojas commented 4 years ago

Changed dependencies from #28190 #29042 to #28190 #29042 #29428

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 0d4f0c7 to 289e603

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

289e603trac 28197: IPython 7.13.0, with new required packages jedi, backcall
jhpalmieri commented 4 years ago
comment:30

Here is a branch for IPython 7.13, which requires new packages jedi and backcall, to give people something new to play with, especially since we're getting close to starting the 9.2 release cycle.

I don't see the problem reported at #29428, or at least the small example there doesn't cause problems for me. The preparser doesn't work, though. I'm about to try the patch in comment:16, after which I'll add it to the branch. (Is there a missing or an extra single quote on line 47 of that patch?)

jhpalmieri commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,20 +1,13 @@
 WARNING: this is python3-only

-* Upgrade ipython to 7.6.1
+Upgrade ipython to 7.13.0 and related packages

-```
-$ wget -P upstream https://github.com/ipython/ipython/releases/download/7.6.1/ipython-7.6.1.tar.gz
-```

-* upgraded prompt-toolkit-2.0.9.tar.gz
+Files:

-To download the tarball:
+* https://pypi.io/packages/source/i/ipython/ipython-7.13.0.tar.gz
+* https://pypi.io/packages/source/i/ipykernel/ipykernel-5.2.1.tar.gz
+* https://pypi.io/packages/source/b/backcall/backcall-0.1.0.tar.gz
+* https://pypi.io/packages/source/j/jedi/jedi-0.17.0.tar.gz
+* https://pypi.io/packages/source/p/prompt_toolkit/prompt_toolkit-3.0.5.tar.gz

-```
-$ wget --output-document=upstream/prompt_toolkit-2.0.9.tar.gz https://github.com/prompt-toolkit/python-prompt-toolkit/archive/2.0.9.tar.gz
-```
-
-* pip-install backcall
-
-* pip-install jedi
-
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

31bb55ctrac 28197: fix preparser for new IPython: patch taken from
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 289e603 to 31bb55c

jhpalmieri commented 4 years ago
comment:33

Here is the preparser patch from comment:16 (although without the lines changing sys.getrecursionlimit()). When merged with #29042, I see these doctest failures:

sage -t --long src/sage/tests/cmdline.py  # 3 doctests failed
sage -t --long src/sage/crypto/mq/sr.py  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/multi_polynomial_sequence.py  # 1 doctest failed
sage -t --long src/sage/misc/package.py  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/pbori.pyx  # 1 doctest failed
sage -t --long src/sage/sat/solvers/dimacs.py  # 1 doctest failed
sage -t --long src/doc/en/reference/sat/index.rst  # 1 doctest failed
sage -t --long src/sage/sat/converters/polybori.py  # 1 doctest failed
sage -t --long src/sage/sat/boolean_polynomials.py  # 1 doctest failed
      File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/IPYTHON/sage-9.1.rc1/local/lib/python3.7/site-packages/brial/gbrefs.py", line 9, in <module>
        import imp
    ...
    DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
Failed example:
    installed_packages()  # optional - build
Expected:
    {...'alabaster': ...'pynac': ...}

pynac appeared before alabaster in the actual output, unfortunately.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from 31bb55c to cd347e1

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

cd347e1trac 28197: add new package parso
antonio-rojas commented 4 years ago
comment:35

Replying to @jhpalmieri:

Here is the preparser patch from comment:16 (although without the lines changing sys.getrecursionlimit()). When merged with #29042, I see these doctest failures:

Uhm, for some reason I haven't been getting emails from this ticket (and many others). The brial issues are fixed in 1.2.6, sage's package seems to be quite outdated (latest version is 1.2.8)

When I ported the preparser, I removed quite a few tests from SagePromptTransformer(), those should probably be brought back somewhere.

jhpalmieri commented 4 years ago
comment:36

BRiAl update at #29658

jhpalmieri commented 4 years ago

Changed dependencies from #28190 #29042 #29428 to #28190 #29042 #29428 #29658

antonio-rojas commented 4 years ago

Changed dependencies from #28190 #29042 #29428 #29658 to #28190 #29042 #29428 #29658 #29774

antonio-rojas commented 4 years ago
comment:38

Opened #29774 to deal with the multiple deprecation warnings thrown by ipython with jedi>=0.16

kliem commented 4 years ago
comment:39

This seems to basically work now. I'm testing it on top of #29042 now and for example with ubuntu eoan I'm getting only one new failure:

https://github.com/kliem/sage-test-27122/runs/742530683

File "src/sage/rings/qqbar.py", line 8096, in sage.rings.qqbar.ANBinaryExpr.exactify
Failed example:
    import sys; sys.getrecursionlimit()
Expected:
    1000
Got:
    3000
**********************************************************************
File "src/sage/rings/qqbar.py", line 8102, in sage.rings.qqbar.ANBinaryExpr.exactify
Failed example:
    sys.getrecursionlimit()
Expected:
    1000
Got:
    3000

I will report further results, once the tests are done.

mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,13 +1,6 @@
 WARNING: this is python3-only

-Upgrade ipython to 7.13.0 and related packages
+Upgrade ipython to 7.13.0 and related packages: ipython-7.13.0, ipykernel-5.2.1, backcall-0.1.0, jedi-0.17.0, prompt_toolkit-3.0.5

+Tarballs: see `checksums.ini` [`upstream_url`]. (To configure Sage to download from the upstream URLs, use `./configure --enable-download-from-upstream-url`)

-Files:
-
-* https://pypi.io/packages/source/i/ipython/ipython-7.13.0.tar.gz
-* https://pypi.io/packages/source/i/ipykernel/ipykernel-5.2.1.tar.gz
-* https://pypi.io/packages/source/b/backcall/backcall-0.1.0.tar.gz
-* https://pypi.io/packages/source/j/jedi/jedi-0.17.0.tar.gz
-* https://pypi.io/packages/source/p/prompt_toolkit/prompt_toolkit-3.0.5.tar.gz
-
mkoeppe commented 4 years ago

Description changed:

--- 
+++ 
@@ -1,6 +1,11 @@
-WARNING: this is python3-only
+Upgrade ipython and related packages: 
+* https://pypi.org/project/ipython/ 7.13.0 (current as of 2020-06-05: 7.15.0)
+* https://pypi.org/project/ipykernel/ 5.2.1 (current as of 2020-06-05: 5.3.0)
+* https://pypi.org/project/backcall/ 0.1.0 (current as of 2020-06-05)
+* https://pypi.org/project/jedi/ 0.17.0 (current as of 2020-06-05)
+* https://pypi.org/project/prompt-toolkit/ 3.0.5 (current as of 2020-06-05)

-Upgrade ipython to 7.13.0 and related packages: ipython-7.13.0, ipykernel-5.2.1, backcall-0.1.0, jedi-0.17.0, prompt_toolkit-3.0.5
+All of these package versions support python >= 3.6 (note #29033).

 Tarballs: see `checksums.ini` [`upstream_url`]. (To configure Sage to download from the upstream URLs, use `./configure --enable-download-from-upstream-url`)
kliem commented 4 years ago
comment:42

Except for this qqbar test, all tests pass on my ubuntu bionic (testing this ticket on top of #29042).

As noted already tab completion is broken and #29774 fixes this.

jhpalmieri commented 4 years ago
comment:43

Is this coming from jedi/app/__init__.py, the line

sys.setrecursionlimit(3000)

?

I don't think the actual limit is important, just that it doesn't change, so I propose the following:

diff --git a/src/sage/rings/qqbar.py b/src/sage/rings/qqbar.py
index 7fc0cb9917..f38195cdcc 100644
--- a/src/sage/rings/qqbar.py
+++ b/src/sage/rings/qqbar.py
@@ -8093,14 +8093,14 @@ class ANBinaryExpr(ANDescr):
         do this by increasing the recursion level at each step and
         decrease it before we return::

-            sage: import sys; sys.getrecursionlimit()
-            1000
+            sage: import sys
+            sage: old_limit = sys.getrecursionlimit()
             sage: s = SymmetricFunctions(QQ).schur()
             sage: a=s([3,2]).expand(8)(flatten([[QQbar.zeta(3)^d for d in range(3)], [QQbar.zeta(5)^d for d in range(5)]]))
             sage: a.exactify(); a # long time
             0
-            sage: sys.getrecursionlimit()
-            1000
+            sage: sys.getrecursionlimit() == old_limit
+            True

         """
         import sys
jhpalmieri commented 4 years ago
comment:44

I ran into a problem building Sage because it tried to build backcall before pip was installed. So here is a new branch. (It doesn't include the proposed change from comment:43).

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 4 years ago

Changed commit from cd347e1 to a9594b5