search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.47k stars 486 forks source link

Upgrade to Python 2.7.13 #22037

Closed jdemeyer closed 7 years ago

jdemeyer commented 7 years ago

Tarball: https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz

Note: upgrade to Python 2.7.12 done in #19735.

CC: @simon-king-jena @hivert @slel @EmmanuelCharpentier

Component: packages: standard

Author: Jeroen Demeyer

Branch/Commit: 5aeaf5c

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/22037

jdemeyer commented 7 years ago
comment:45

My first impression is that this new version of Python is significantly slower than 2.7.12: I'm seeing several doctest timeouts on a system where I usually don't get any. Granted, my system is loaded from doing things so it might be a false positive. Need to check...

jdemeyer commented 7 years ago
comment:46

Replying to @embray:

Ah, I think Florent was telling me about this a couple weeks ago. Wish I had been CC'd on this--I've spent a lot of time in Python's class initialization code and probably could have helped. Let me know if I can still help on this.

The main question remains: did they break tp_new on other Python versions? The patch got reverted for 2.7.13 but it's not clear whether or not it was applied to some 3.y.x version.

jdemeyer commented 7 years ago
comment:47

It worries me that nothing in the Python docs nor in any PEP describes how tp_new is inherited. In my opinion, https://bugs.python.org/issue5322 makes a significant change which should be subject to a PEP. However, neither the old nor new behaviour is described anywhere. This makes it harder to argue which behaviour is correct.

embray commented 7 years ago
comment:48

Part of the problem is that tp_new is a detail of the CPython API, and while there are PEPs that deal with CPython there aren't many (not enough IMO given that it is the reference implementation). Point being though, this code is very particular to the C API and doesn't particularly impact how Python should work from a pure language design perspective. But I agree, this should be better documented.

My guess is just that in the wild there aren't that many hand-written C types outside of Numpy and ones generated by Cython. And it's even less common for them to be used in a multiple inheritance scenario. But just because it isn't common doesn't mean it should be broken or undocumented, so I completely agree. After all this problem was pointed out in 2009 so it's not exactly unheard of.

embray commented 7 years ago
comment:49

FWIW the most relevant PEP to this https://www.python.org/dev/peps/pep-0253/, but I haven't read it in a long time so I forget whether or not it addresses this specific issue, especially given:

The PEP no longer accurately describes the implementation.

jdemeyer commented 7 years ago
comment:50

Yes, I found that PEP when the discussion about https://bugs.python.org/issue5322 started but it doesn't answer that specific question.

embray commented 7 years ago
comment:51

Ah, okay. Not surprising--it also says in the PEP:

While the details of initializing a PyTypeObject structure haven't been documented as such, they are easily gleaned from the examples in the source code, and I am assuming that the reader is sufficiently familiar with the traditional way of creating new Python types in C.

In fairness, those details predate the PEP process in the first place so it's not surprising that there's no PEP for it. But it would be nice to have something better documented in the C-API docs. It should be part of the API.

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago

First attempt at installing Python 2.7.13

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago
comment:53

Attachment: python2-2.7.13.p0.log.gz

No good, alas... After merging the relevant branch in my current branch (containing R 3.3.2 and the necessary helper packages for xz and pcre), I tried time ./sage -i -c python2. This took a long time, mostly spent on waiting on failing tests :

real    48m11,852s
user    6m46,312s
sys 1m12,012s

At the end of the log file (see attached file), I get :

353 tests OK.
6 tests failed:
    test_gdb test_site test_ssl test_sysconfig test_urllib2
    test_urllib2_localnet
1 test altered the execution environment:
    test_import
41 tests skipped:
    test_aepack test_al test_applesingle test_bsddb test_bsddb185
    test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dbm test_dl test_gdbm test_gl test_imageop test_imgfile
    test_kqueue test_linuxaudiodev test_macos test_macostools
    test_msilib test_ossaudiodev test_pep277 test_scriptpackages
    test_smtpnet test_socketserver test_startfile test_sunaudiodev
    test_timeout test_tk test_ttk_guionly test_unicode_file
    test_urllib2net test_urllibnet test_winreg test_winsound
    test_zipfile64
4 skips unexpected on linux2:
    test_bsddb test_bsddb3 test_dbm test_gdbm
Makefile:877: recipe for target 'test' failed

Curiously, notwhithstanding the reported problems on the ssl module, pip seems to be able to function somehow :

charpent@asus16-ec:/usr/local/sage-7$ sage -pip search tralala
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
charpent@asus16-ec:/usr/local/sage-7$ sage -pip search terminado
terminado (0.6)  - Terminals served to term.js using Tornado websockets
  INSTALLED: 0.6 (latest)
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(and I get pip advertisements again, source of pseudo-failures in tests...)

I doubt that it is worth make ptestlong (which monopolizes my machine for a long nice hour, nowadays...).

I do not know enough about the specifics of Python in Sage to fel justified in positioning ths ticket as needs_wokj, but I'm sorely tempted.

I'll keep my branch in this tate for a while to see what may crop up.

HTH,

jdemeyer commented 7 years ago
comment:54

Replying to @EmmanuelCharpentier:

I do not know enough about the specifics of Python in Sage to fel justified in positioning ths ticket as needs_wokj, but I'm sorely tempted.

Please elaborate. Does this ticket break anything?

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago
comment:55

Replying to @jdemeyer:

Replying to @EmmanuelCharpentier:

I do not know enough about the specifics of Python in Sage to fel justified in positioning ths ticket as needs_wokj, but I'm sorely tempted.

Please elaborate. Does this ticket break anything?

A cursory, extremely superficial, test(a couple lines in Sage, a bit more in IPython + sympy) didn't show major immediate problems shuch as a crash. But the six bailed test units make me wary.

I just launched make ptestlong and I see sage recompiling python2 and brial. Answer in a couple of hours...

jhpalmieri commented 7 years ago
comment:56

At some times in the past, the python2 package has not passed its test suite. Does the current version pass? That is, are you seeing a regression or the status quo?

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago
comment:57

It might be a regression, but I'm not sure :

make ptestlong is currently stalled, with a suspicious number of defunct (zombie) subprocesses. From ps axf, I get (extract) :

26497 ?        Ssl    1:03  \_ /usr/lib/gnome-terminal/gnome-terminal-server
14353 pts/3    Ss     0:00  |   \_ bash
20219 pts/3    S+     0:00  |   |   \_ make ptestlong
 3447 pts/3    S+     0:34  |   |       \_ python /usr/local/sage-7/src/bin/sage-runtests -p --all --long --logfile=logs/ptestlong.log
 3486 pts/3    S+     0:01  |   |           \_ python /usr/local/sage-7/src/bin/sage-cleaner
15843 pts/3    S      0:01  |   |           \_ python /usr/local/sage-7/src/bin/sage-runtests -p --all --long --logfile=logs/ptestlong.log
15844 pts/0    Ssl+   0:06  |   |               \_ /usr/local/sage-7/local/bin/giac --sage
15845 pts/3    Z      0:00  |   |               \_ [python] <defunct>
16021 ?        Zs     0:00  |   |               \_ [giac] <defunct>
16022 pts/3    Z      0:00  |   |               \_ [python] <defunct>
16032 pts/3    Z      0:00  |   |               \_ [python] <defunct>
16033 ?        Zs     0:00  |   |               \_ [giac] <defunct>
16034 pts/3    Z      0:00  |   |               \_ [python] <defunct>
16043 pts/3    Z      0:00  |   |               \_ [python] <defunct>
17896 pts/4    Ss     0:00  |   \_ bash
25263 pts/4    R+     0:00  |       \_ ps axf

I'll let it go for a while, and kill it, say, tomorrow morning (it's about 22:05 locally).

But this smells. A look at the log (from another terminal) gives me some known bugs, and some new :

File "src/sage/rings/polynomial/multi_polynomial_ideal.py", line 3532, in sage.r
ings.polynomial.multi_polynomial_ideal.NCPolynomialIdeal.groebner_basis
Failed example:
    ideal(J.transformed_basis()).change_ring(P).interreduced_basis()  # optional - giacpy_sage
Expected:
    [a - 60*c^3 + 158/7*c^2 + 8/7*c - 1, b + 30*c^3 - 79/7*c^2 + 3/7*c, c^4 - 10/21*c^3 + 1/84*c^2 + 1/84*c]
Got:
    [7*a - 420*c^3 + 158*c^2 + 8*c - 7, 7*b + 210*c^3 - 79*c^2 + 3*c, 84*c^4 - 40*c^3 + c^2 + c]

This one is already known.

File "src/sage/homology/simplicial_complex.py", line 2812, in sage.homology.simp
licial_complex.SimplicialComplex.is_cohen_macaulay
Failed example:
    X.is_cohen_macaulay(ZZ)
Expected:
    False
Got:
    [Errno 2] No such file or directory: '/home/charpent/.sage/temp/asus16-ec/17311/dir_ZhULbV/17421.out'
    False

Might be a race condition : from yet another terminal :

charpent@asus16-ec:/usr/local/sage-7$ sage -t --long --warn-long 109.9 src/sage/homology/simplicial_complex.py
Running doctests with ID 2017-01-04-22-08-34-3e41af20.
Git branch: testR
Using --optional=database_gap,giacpy_sage,mpir,python2,sage,xz
Doctesting 1 file.
sage -t --long --warn-long 109.9 src/sage/homology/simplicial_complex.py
    [588 tests, 3.25 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 3.4 seconds
    cpu time: 3.0 seconds
    cumulative wall time: 3.3 seconds
sage -t --long --warn-long 109.9 src/sage/combinat/root_system/cartan_type.py
    Bad exit: 2
**********************************************************************
Tests run before process (pid=17761) failed:
sage: T = CartanType(['A', 4]) ## line 18 ##
sage: T ## line 19 ##
['A', 4]
sage: DynkinDiagram(T) ## line 24 ##
O---O---O---O
1   2   3   4   
A4
sage: DynkinDiagram(['A',4]) ## line 33 ##
O---O---O---O
1   2   3   4   
A4
sage: DynkinDiagram('A4') ## line 37 ##
O---O---O---O
1   2   3   4   
A4
sage: T.dynkin_diagram() ## line 41 ##
O---O---O---O
1   2   3   4   
A4
sage: RootSystem(T) ## line 52 ##
Root system of type ['A', 4]
sage: W = WeylGroup(T) ## line 57 ##

This one is new to me. But again, it might be a race condition :

charpent@asus16-ec:/usr/local/sage-7$ sage -t --long --warn-long 109.9 src/sage/combinat/root_system/cartan_type.py
Running doctests with ID 2017-01-04-22-10-59-8ee52f1c.
Git branch: testR
Using --optional=database_gap,giacpy_sage,mpir,python2,sage,xz
Doctesting 1 file.
sage -t --long --warn-long 109.9 src/sage/combinat/root_system/cartan_type.py
    [446 tests, 1.36 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 1.5 seconds
    cpu time: 1.1 seconds
    cumulative wall time: 1.4 seconds

And that's all for now. But make ptestlong is still stalled at :

sage -t --long --warn-long 109.9 local/lib/python2.7/site-packages/sagenb/testing/selenium/__init__.py
    [0 tests, 0.00 s]
sage -t --long --warn-long 109.9 src/sage/interfaces/giac.py

If you feel it useful, I'll attach the resulting ptestlong.log after tomorrow's kill.

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago
comment:58

Replying to @jhpalmieri:

At some times in the past, the python2 package has not passed its test suite. Does the current version pass?

No. Six test units failed (see above) .

That is, are you seeing a regression or the status quo?

I don't know if these test units used to pass before...

HTH,

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago

Attachment: ptestlong.log.gz

result of make ptestlong, first attempt on 2.7.13

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago
comment:59

New data point : after about 1 hour, the test of giac inteface timed out, liberating the whole thing (the zombie processes disappeared).

The resulting ptestlong.log has been attached, just in case...

Interestingly, this may be a race condition : the very same test passes standalone in about 1.33 seconds :

charpent@asus16-ec:/usr/local/sage-7$ sage -t --long --warn-long 109.9 src/sage/interfaces/giac.py
Running doctests with ID 2017-01-04-22-51-02-b275d916.
Git branch: testR
Using --optional=database_gap,giacpy_sage,mpir,python2,sage,xz
Doctesting 1 file.
sage -t --long --warn-long 109.9 src/sage/interfaces/giac.py
    [157 tests, 1.33 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 6.4 seconds
    cpu time: 0.6 seconds
    cumulative wall time: 1.3 seconds

So we have two, possibly three problems :

Advice ?

jhpalmieri commented 7 years ago
comment:60

Replying to @EmmanuelCharpentier:

Replying to @jhpalmieri:

At some times in the past, the python2 package has not passed its test suite. Does the current version pass?

No. Six test units failed (see above) .

I meant, do the unit tests pass for 2.7.10 (= the current version)?

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago
comment:61

Replying to @jhpalmieri:

[ Snip...]

I meant, do the unit tests pass for 2.7.10 (= the current version)?

Current is 2.7.12 (since 7.5rc0).

354 tests OK.
5 tests failed:
    test_gdb test_site test_sysconfig test_urllib2
    test_urllib2_localnet
42 tests skipped:
    test_aepack test_al test_applesingle test_bsddb test_bsddb185
    test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dbm test_dl test_gdbm test_gl test_imageop test_imgfile
    test_kqueue test_linuxaudiodev test_macos test_macostools
    test_msilib test_ossaudiodev test_pep277 test_scriptpackages
    test_smtpnet test_socketserver test_ssl test_startfile
    test_sunaudiodev test_timeout test_tk test_ttk_guionly
    test_unicode_file test_urllib2net test_urllibnet test_winreg
    test_winsound test_zipfile64
5 skips unexpected on linux2:
    test_bsddb test_bsddb3 test_dbm test_gdbm test_ssl

So there is more lossage (test_ssl passed in 2.7.12).

7822f248-ba56-45f1-ab3d-4de7482bdf9f commented 7 years ago
comment:62

Replying to @EmmanuelCharpentier:

Going back to 7.5beta6 (which had 2.7.10, I get :

4 tests failed:
    test_gdb test_site test_sysconfig test_urllib2_localnet
42 tests skipped:
    test_aepack test_al test_applesingle test_bsddb test_bsddb185
    test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dbm test_dl test_gdbm test_gl test_imageop test_imgfile
    test_kqueue test_linuxaudiodev test_macos test_macostools
    test_msilib test_ossaudiodev test_pep277 test_scriptpackages
    test_smtpnet test_socketserver test_ssl test_startfile
    test_sunaudiodev test_timeout test_tk test_ttk_guionly
    test_unicode_file test_urllib2net test_urllibnet test_winreg
    test_winsound test_zipfile64
5 skips unexpected on linux2:
    test_bsddb test_bsddb3 test_dbm test_gdbm test_ssl

So test_urllib2 was lost between 2.7.10 and 2.7.12. And, BTW, test_ssl didn't pass in 2.7.12 : it was marked as unexpected.

HTH,

jdemeyer commented 7 years ago
comment:63

There is some significant speed regression going from 2.7.12 to 2.7.13.

I doctested the directory src/sage/manifolds 3 times with each Python version on an otherwise idle system.

With 2.7.12, I got

manifolds-2.7.12.log:Total time for all tests: 619.3 seconds
manifolds-2.7.12.log:Total time for all tests: 625.6 seconds
manifolds-2.7.12.log:Total time for all tests: 616.8 seconds

With 2.7.13:

manifolds-2.7.13.log:Total time for all tests: 645.1 seconds
manifolds-2.7.13.log:Total time for all tests: 640.7 seconds
manifolds-2.7.13.log:Total time for all tests: 646.0 seconds

I don't know what to think of this...

jdemeyer commented 7 years ago
comment:64

Blocker because of #22147.

vbraun commented 7 years ago

Reviewer: Volker Braun

vbraun commented 7 years ago
comment:65

Since we need this to compile on OSX...

remaining issues should be moved to separate tickets.

vbraun commented 7 years ago

Changed branch from u/jdemeyer/upgrade_to_python_2_7_13 to 5aeaf5c