search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.44k stars 480 forks source link

Metaticket: Add support for python 3.6+ #15530

Closed ohanar closed 4 years ago

ohanar commented 10 years ago

How to build sage with python3:

make configure
./configure --with-python=3
make

Beware that the second step above should be redone after every make distclean.

For progress on making all doctests pass, see #26212 and #28298

In order to support python 3.7, the following needs to be fixed:

  1. 15510 - upgrade to a newer version of setuptools

  2. 15511 - upgrade to a newer version of rpy2

  3. 15512 - upgrade to a newer version of sympy

  4. 14854 - upgrade to a newer version of pycrypto

  5. 15532 - upgrade to a newer version of networkx

  6. 15537 - fix csage to work with python3

  7. 15539 - switch from using PIL to Pillow

  8. 15540 - trivial python3 fixes to a few spkgs

  9. 15541 - fix sage-location and sage-download-file for python3

  10. 15593 - remove sqlalchemy

  11. 17591 - remove gdmodule

  12. 15620 - Stop using StandardError

  13. 15755 - upgrade cython to version 0.20.1

  14. 15807 - upgrade mpmath to version 0.18

  15. 15980 - meta-ticket for python3 compatibility of the sage library (stage 1)

  16. 16052 - meta-ticket for python3 compatibility of the sage library (stage 2)

  17. 17854 - remove c_lib

  18. 18437 - fix polybori and python 3

  19. 18537 - fix Pynac at least building against python 3

  20. 17607 - add python 3 package

  21. 27024 - make sagetex python3 compatible

Possibly relevant timeline from PEP 373:

Being the last of the 2.x series, 2.7 will have an extended period of maintenance. The current plan is to support it for at least 10 years from the initial 2.7 release. This means there will be bugfix releases until 2020.

Depends on #15510 Depends on #15511 Depends on #15512 Depends on #15531 Depends on #15532 Depends on #15537 Depends on #15539 Depends on #15540 Depends on #15541 Depends on #15807 Depends on #15980 Depends on #18537 Depends on #16052 Depends on #18437 Depends on #27024

Component: python3

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/15530

kiwifb commented 9 years ago
comment:46

First attempt to get scons to install polybori as a python 3.4 thing failed. SConstruct is relying on the version of python running scons as being the version you should use I think. That leads to a number of odd behavior and misdetections very fast.

ohanar commented 9 years ago

Description changed:

--- 
+++ 
@@ -16,8 +16,8 @@
 14. #15807 - upgrade mpmath to version 0.18
 15. #15980 - meta-ticket for python3 compatibility of the sage library (stage 1)
 16. #16052 - meta-ticket for python3 compatibility of the sage library (stage 2)
-17. upgrade scons to a version that supports python3 (does not yet exist) or get rid of scons (assuming #17854, scons is only used by polybori)
-18. upgrade polybori to a version that supports python3 (does not yet exist)
+17. #17854 - remove c_lib
+18. #18437 - fix polybori and python 3
 19. #17607 - add python 3 package

 ---
ohanar commented 9 years ago
comment:47

fyi, I started working on an autotools based build system for polybori at https://github.com/ohanar/PolyBoRi/tree/autotools. I'll probably work on it a bit more next weekend during SD 64.25, but the important bits (the c++ libraries) are currently working with only a couple of hacks. What remains (for sage at least) is just a little cleanup and throwing in the bit of the python bindings we use. I've created #18437 for future discussion of polybori and python 3.

rwst commented 9 years ago

Changed dependencies from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, pynac-0.3.8 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537

jdemeyer commented 9 years ago
comment:49

Does now every standard package except polybori support Python 3?

kiwifb commented 9 years ago
comment:50

I think so. The current pexpect would be the other problematic one but the current has been patched to use six to support python 3 if we don't upgrade it to 3.3 + patch (#10295).

a-andre commented 9 years ago
comment:51

Replying to @jdemeyer:

Does now every standard package except polybori support Python 3?

The Sphinx release notes for 1.3b1 mention Add support for Python 3.4. So I assume this version is not supported by the Sphinx 1.2.2 which is currently shipped with Sage. #18497

kiwifb commented 9 years ago
comment:52

Replying to @a-andre:

Replying to @jdemeyer:

Does now every standard package except polybori support Python 3?

The Sphinx release notes for 1.3b1 mention Add support for Python 3.4. So I assume this version is not supported by the Sphinx 1.2.2 which is currently shipped with Sage. #18497

That's odd, Gentoo permits you to build and install sphinx 1.2.2 against python 3.4 without any patch. It is a stable version so I'll assume that it passes its test suite at least with python 3.4. But going to 1.3.1 would be good anyway.

ohanar commented 9 years ago
comment:53

Replying to @jdemeyer:

Does now every standard package except polybori support Python 3?

Pynac doesn't really support Python 3 (it builds against Python 3, but it is almost certainly broken, since it can only really be tested when sage can be built against Python 3).

kcrisman commented 9 years ago

Changed dependencies from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437

kcrisman commented 9 years ago

Description changed:

--- 
+++ 
@@ -18,7 +18,8 @@
 16. #16052 - meta-ticket for python3 compatibility of the sage library (stage 2)
 17. #17854 - remove c_lib
 18. #18437 - fix polybori and python 3
-19. #17607 - add python 3 package
+19. #18537 - fix Pynac at least building against python 3
+20. #17607 - add python 3 package

 ---
jdemeyer commented 8 years ago

Description changed:

--- 
+++ 
@@ -26,9 +26,3 @@
 Possibly relevant timeline from [PEP 373](https://www.python.org/dev/peps/pep-0373/):

 Being the last of the 2.x series, 2.7 will have an extended period of maintenance. The current plan is to support it for at least 10 years from the initial 2.7 release. This means there will be bugfix releases until 2020.
-
-Planned future release dates:
-
-* 2.7.10 June 2015
-* beyond this date, releases as needed
-
fchapoton commented 8 years ago

Description changed:

--- 
+++ 
@@ -22,6 +22,15 @@
 20. #17607 - add python 3 package

 ---
+How to try to make sage with python3:
+
+```
+export SAGE_PYTHON3=yes
+make build
+```
+(preferably inside a python3 virtualenv)
+
+---

 Possibly relevant timeline from [PEP 373](https://www.python.org/dev/peps/pep-0373/):
fchapoton commented 8 years ago

Description changed:

--- 
+++ 
@@ -28,7 +28,6 @@
 export SAGE_PYTHON3=yes
 make build

-(preferably inside a python3 virtualenv)

embray commented 6 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-In order to support python 3.3, the following needs to be fixed:
+In order to support python 3.6, the following needs to be fixed:

 1. #15510 - upgrade to a newer version of setuptools
 2. #15511 - upgrade to a newer version of rpy2
embray commented 6 years ago
comment:59

Changed ticket title since we're now targeting Python 3.6 at a minimum (though most of the Python 3 work could probably go back to older Python 3's since everything is still compatible with Python 2.7).

I guess we can leave this open as there are still 2 meta-tickets associated with it that are open...

fchapoton commented 6 years ago

Description changed:

--- 
+++ 
@@ -22,10 +22,11 @@
 20. #17607 - add python 3 package

 ---
-How to try to make sage with python3:
+How to build sage with python3:

-export SAGE_PYTHON3=yes +make configure +./configure --with-python=3 make build

embray commented 6 years ago
comment:61

I recently ran the doctests on my python3 branch (which contains many fixes not in develop yet, though I've yet to incorporate all the existing open tickets for python3 which might lead to more improvements). I got roughly 540 failing modules.

Out of curiosity I thought I'd try a little experiment and ran the test log through the following:

$ sort logs/ptest.log | uniq -c | sort -nr

This just gives the text that appears most frequently in the test output. Obviously the top examples are generic things like "6461 Traceback (most recent call last):"

But beyond that there are some interesting things that might help prioritize things to fix (and maybe some of these are already fixed in open tickets). The top 20 error messages this brings up (excluding NameErrors) are:

Of course, from this experiment there's often no easy way to see what the context is to these errors, an some of them might just occur hundreds of times in one module and not have a wide-reaching effect. But it's worth looking into. Grepping the test log for individual error messages helps give a better idea of where they might come from.

embray commented 6 years ago
comment:62

Another potentially interesting heuristic, though I haven't tested how effective this is yet is something like:

$ grep 'doctest.*failed' logs/ptest.log | sort -k 5 -nr | head -20

showing me the modules with the most failures.

On my latest python3 branch (which has all the currently open python3 tickets merged in, as well as some other fixes) I have:

sage -t src/sage/misc/explain_pickle.py  # 71 doctests failed
sage -t src/sage/calculus/riemann.pyx  # 67 doctests failed
sage -t src/sage/libs/lcalc/lcalc_Lfunction.pyx  # 58 doctests failed
sage -t src/sage/combinat/free_dendriform_algebra.py  # 50 doctests failed
sage -t src/sage/combinat/words/finite_word.py  # 47 doctests failed
sage -t src/doc/en/thematic_tutorials/sandpile.rst  # 45 doctests failed
sage -t src/sage/matrix/compute_J_ideal.py  # 44 doctests failed
sage -t src/sage/graphs/generic_graph.py  # 41 doctests failed
sage -t src/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed
sage -t src/sage/sandpiles/sandpile.py  # 34 doctests failed
sage -t src/sage/combinat/hillman_grassl.py  # 34 doctests failed
sage -t src/sage/combinat/cluster_algebra_quiver/quiver.py  # 33 doctests failed
sage -t src/sage/combinat/finite_state_machine.py  # 32 doctests failed
sage -t src/sage/rings/polynomial/real_roots.pyx  # 29 doctests failed
sage -t src/sage/coding/source_coding/huffman.py  # 26 doctests failed
sage -t src/sage/combinat/diagram_algebras.py  # 25 doctests failed
sage -t src/sage/matroids/matroid.pyx  # 23 doctests failed
sage -t src/sage/combinat/rigged_configurations/kr_tableaux.py  # 23 doctests failed
sage -t src/sage/combinat/cluster_algebra_quiver/cluster_seed.py  # 22 doctests failed
sage -t src/sage/graphs/graph.py  # 20 doctests failed

With the exception of "explain_pickle" which is kind of a special case (and one I'm not sure what to do with), my guess is that modules with many failing tests indicate something that is deeply broken enough which, if fixed, might have far reaching consequences both within that module itself, and other modules that depend on it.

Of course this isn't always going to be the case--sometimes the failures might be very localized. But this gives some idea of where the most effort is needed.

embray commented 6 years ago
comment:63

Well, based on the example above, I found one one-line bug in sage.calculus.riemann (#25974) which fixed all the tests in that module, as well as 7 other modules. Still very anecdotal, but at least one good result from that heuristic.

kiwifb commented 6 years ago
comment:64

In sage-on-gentoo I get in the following trouble a lot with python3

      File "sage/libs/gap/util.pyx", line 199, in sage.libs.gap.util.initialize (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_6/build/cythonized/sage/libs/gap/util.c:4452)
        s = str_to_bytes(gap_root(), FS_ENCODING, "surrogateescape")
      File "sage/libs/gap/util.pyx", line 171, in sage.libs.gap.util.gap_root (/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_6/build/cythonized/sage/libs/gap/util.c:4232)
        gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]
    TypeError: 'filter' object is not subscriptable

But I am guessing that it is not seen in vanilla sage because I apply the following patch

diff --git a/sage/libs/gap/util.pyx b/sage/libs/gap/util.pyx
index 97703ff..48dc617 100644
--- a/sage/libs/gap/util.pyx
+++ b/sage/libs/gap/util.pyx
@@ -23,7 +23,7 @@ from .element cimport *
 from sage.cpython.string import FS_ENCODING
 from sage.cpython.string cimport str_to_bytes, char_to_str
 from sage.interfaces.gap_workspace import prepare_workspace_dir
-from sage.env import SAGE_LOCAL, GAP_ROOT_DIR
+from sage.env import SAGE_LOCAL

 ############################################################################
@@ -167,9 +167,6 @@ def gap_root():
         '/home/vbraun/opt/sage-5.3.rc0/local/gap/latest'
     """
     import os.path
-    if os.path.exists(GAP_ROOT_DIR):
-        return GAP_ROOT_DIR
-    print('The gap-4.5.5.spkg (or later) seems to be not installed!')
     gap_sh = open(os.path.join(SAGE_LOCAL, 'bin', 'gap')).read().splitlines()
     gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]
     gapdir = gapdir.split('"')[1]

vanilla never shows the problem because it uses GAP_ROOT_DIR, which I find an hindrance as it presuppose a way of installing gap, the alternate code that comes after works well enough - at least with python2. But obviously not with python3.

Any suggestions on fixing this?

fchapoton commented 6 years ago
comment:65

replace

gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]

by

gapdir = next(dir for dir in gap_sh if dir.strip().startswith('GAP_DIR'))

? Please make a ticket.

kiwifb commented 6 years ago
comment:66

OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.

embray commented 6 years ago
comment:67

Replying to @kiwifb:

OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.

Perhaps it should be?

kiwifb commented 6 years ago
comment:68

Replying to @embray:

Replying to @kiwifb:

OK will try that and create a ticket when I can. This has escaped detection so far because it is not doctested.

Perhaps it should be?

Definitely.

kiwifb commented 6 years ago
comment:69

Replying to @fchapoton:

replace

gapdir = filter(lambda dir:dir.strip().startswith('GAP_DIR'), gap_sh)[0]

by

gapdir = next(dir for dir in gap_sh if dir.strip().startswith('GAP_DIR'))

? Please make a ticket.

That works, but to be complete, it will also need some doctest. It is now #26665.

fchapoton commented 5 years ago

Description changed:

--- 
+++ 
@@ -1,3 +1,5 @@
+For progress on making all doctests pass, see #26212
+
 In order to support python 3.6, the following needs to be fixed:

 1. #15510 - upgrade to a newer version of setuptools
dimpase commented 5 years ago

Changed dependencies from #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437 to #15510, #15511, #15512, #15531, #15532, #15537, #15539, #15540, #15541, #15807, #15980, #18537, #16052, #18437, #27024

dimpase commented 5 years ago

Description changed:

--- 
+++ 
@@ -22,6 +22,7 @@
 18. #18437 - fix polybori and python 3
 19. #18537 - fix Pynac at least building against python 3
 20. #17607 - add python 3 package
+21. #27024 - make sagetex python3 compatible

 ---
 How to build sage with python3:
dimpase commented 5 years ago
comment:72

Shouldn't this ticket maintain a branch of with all the positively reviewed py3-related tickets?

embray commented 5 years ago
comment:73

That might be useful: I do something like that manually anyways.

kwankyu commented 5 years ago

Description changed:

--- 
+++ 
@@ -30,7 +30,7 @@

make configure ./configure --with-python=3 -make build +make


 ---
fchapoton commented 5 years ago

Description changed:

--- 
+++ 
@@ -32,6 +32,7 @@
 ./configure --with-python=3
 make

+Beware that the configure step above should be redone after every make distclean.

fchapoton commented 5 years ago

Description changed:

--- 
+++ 
@@ -1,3 +1,15 @@
+---
+How to build sage with python3:
+
+```
+make configure
+./configure --with-python=3
+make
+```
+Beware that the second step above should be redone after every `make distclean`.
+
+---
+
 For progress on making all doctests pass, see #26212

 In order to support python 3.6, the following needs to be fixed:
@@ -24,18 +36,6 @@
 20. #17607 - add python 3 package
 21. #27024 - make sagetex python3 compatible

----
-How to build sage with python3:
-
-```
-make configure
-./configure --with-python=3
-make
-```
-Beware that the configure step above should be redone after every `make distclean`.
-
----
-
 Possibly relevant timeline from [PEP 373](https://www.python.org/dev/peps/pep-0373/):

 Being the last of the 2.x series, 2.7 will have an extended period of maintenance. The current plan is to support it for at least 10 years from the initial 2.7 release. This means there will be bugfix releases until 2020.
fchapoton commented 5 years ago

Description changed:

--- 
+++ 
@@ -10,9 +10,9 @@

 ---

-For progress on making all doctests pass, see #26212
+For progress on making all doctests pass, see #26212 and #28298

-In order to support python 3.6, the following needs to be fixed:
+In order to support python 3.7, the following needs to be fixed:

 1. #15510 - upgrade to a newer version of setuptools
 2. #15511 - upgrade to a newer version of rpy2
dimpase commented 5 years ago
comment:78

What are we going to do with animation support on Python 3? It seems it is broken - the only (non-console, not sure about console though) possibility seems to be via jupyter+j(s)mol, and this has not moved far: https://github.com/jupyter/notebook/issues/1872

See also the recent where people complain about broken animation on 8.9. https://groups.google.com/d/msg/sage-devel/OepzKdldklU/uf2eRVWuDAAJ

jhpalmieri commented 5 years ago
comment:79

Dima, can you provide more details? For the second paragraph, the sage-devel thread gives an example of something which doesn't work in the legacy Sage notebook (their example starts with the command notebook()). So that's not a Python 3 problem. It also doesn't seem to be discussing animation, but maybe I'm missing something. The problem in the first paragraph is about the Jupyter notebook, and I see the same problem with both Python 2 and Python 3.

dimpase commented 5 years ago
comment:80

yes, the sage-devel thread discusses mostly sagenb, but the question is how to do the job in jupyter instead, and this appears to be a blocker, as this is a serious regression.

Disclaimer: I do not know how to do animations in Sage. If someone who knows looked into this, it would be great.

jhpalmieri commented 5 years ago
comment:81

Okay, it still feels like a Jupyter and/or JMol/threejs issue rather than a Python 3 issue.

dimpase commented 5 years ago
comment:82

it is a Python 3 issue, as one does not have sagenb there, while sagenb (at least before Sage 8.9) was able to do animations.

kcrisman commented 5 years ago
comment:83

Regarding the animation issue, see my tests on the sage-devel thread just now. It does somehow seem to be an integration between sagenb and new j(s)mol thing. As for "how to do animations", all they meant was "make a 3d graphic, and then try to make it movable by clicking the little circle in jmol", not using an actual animation command. I'd encourage a few others to try using viewer='jmol' in Jupyter in the 9.0 series to see what happens.

mkoeppe commented 4 years ago
comment:84

Is anything left to be done here or can the ticket be closed?

dimpase commented 4 years ago
comment:85

see #30053 for a blocker for 3.x, for x<7.

mkoeppe commented 4 years ago
comment:86

Let's close this metaticket to signal the completion of the Python 3 transition.

dimpase commented 4 years ago
comment:87

ok, done

dimpase commented 4 years ago

Reviewer: Dima Pasechnik