search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.46k stars 485 forks source link

Upgrade: Python 3.8.5 #27754

Closed slel closed 4 years ago

slel commented 5 years ago

This is to upgrade to Python 3.8.x.

Tarball: see checksums.ini

Previous and next Python upgrades:

See also:

Depends on #28197 Depends on #29042 Depends on #29861 Depends on #30162 Depends on #30225 Depends on #30127 Depends on #30144 Depends on #30345 Depends on #30365 Depends on #30140 Depends on #30351

Upstream: Reported upstream. Developers acknowledge bug.

CC: @fchapoton @embray @kiwifb @jdemeyer @mkoeppe @slel @timokau @isuruf @kliem @tscrim @heluani @sheerluck

Component: packages: standard

Keywords: days101

Author: Jeroen Demeyer, John Palmieri, Matthias Koeppe, Antonio Rojas

Branch: f1cda12

Reviewer: Matthias Koeppe, François Bissey

Issue created by migration from https://trac.sagemath.org/ticket/27754

slel commented 5 years ago
comment:1

If people want to start testing, Python 3.8.0a3 is out:

jdemeyer commented 5 years ago
comment:2

I think it's a bit early for testing. I'd rather wait for the first beta, which is supposed to be released later this month.

slel commented 5 years ago
comment:3

Python 3.8.0b1 is out:

jdemeyer commented 5 years ago
comment:4

Note: this requires a Cython upgrade.

slel commented 5 years ago
comment:5

Cython upgrade needs review at #27886.

slel commented 5 years ago

Changed keywords from none to days101

slel commented 5 years ago

Dependencies: #28012

jdemeyer commented 5 years ago
comment:7

I'll have a look at this today.

jdemeyer commented 5 years ago

Changed dependencies from #28012 to #27886

jdemeyer commented 5 years ago

Description changed:

--- 
+++ 
@@ -3,3 +3,4 @@
 - [PEP569 -- Python 3.8 Release Schedule](https://www.python.org/dev/peps/pep-0569/)
 - [What's new in Python 3.8](https://docs.python.org/3.8/whatsnew/3.8.html)

+**Tarball**: https://www.python.org/ftp/python/3.8.0/Python-3.8.0b1.tar.xz
jdemeyer commented 5 years ago

Author: Jeroen Demeyer

jdemeyer commented 5 years ago

Branch: u/jdemeyer/upgrade__python_3_8_x

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 5 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

46837b7Python 3.8.0
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 5 years ago

Commit: 46837b7

jdemeyer commented 5 years ago

Description changed:

--- 
+++ 
@@ -4,3 +4,9 @@
 - [What's new in Python 3.8](https://docs.python.org/3.8/whatsnew/3.8.html)

 **Tarball**: https://www.python.org/ftp/python/3.8.0/Python-3.8.0b1.tar.xz
+
+Patches added:
+- https://github.com/python/cpython/pull/13959
+- https://github.com/python/cpython/pull/14193
+
+It is likely that these will be accepted by the next beta
jdemeyer commented 5 years ago

Changed dependencies from #27886 to none

jdemeyer commented 5 years ago
comment:12

Actually, thanks to two patches which are likely going to be accepted upstream, it no longer depends on the Cython upgrade.

jdemeyer commented 5 years ago

Upstream: Reported upstream. Developers acknowledge bug.

jdemeyer commented 5 years ago
comment:14

There is a mysterious build failure of matplotlib:

Processing /home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
  Created temporary directory: /tmp/pip-req-build-vq4okaxf
  Added file:///home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src to build tracker '/tmp/pip-req-tracker-xcv165b_'
  Running setup.py (path:/tmp/pip-req-build-vq4okaxf/setup.py) egg_info for package from file:///home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-vq4okaxf/setup.py", line 172, in <module>
        result = package.check()
      File "/tmp/pip-req-build-vq4okaxf/setupext.py", line 963, in check
        import numpy
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
        from . import core
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/__init__.py", line 16, in <module>
        from . import multiarray
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/multiarray.py", line 12, in <module>
        from . import overrides
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/overrides.py", line 8, in <module>
        from numpy.compat._inspect import getargspec
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/compat/__init__.py", line 14, in <module>
        from . import py3k
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/compat/py3k.py", line 15, in <module>
        from pathlib import Path, PurePath
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/pathlib.py", line 391, in <module>
        class _NormalAccessor(_Accessor):
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/pathlib.py", line 415, in _NormalAccessor
        link_to = os.link
    AttributeError: module 'os' has no attribute 'link'

But when running Python normally, os.link does exist:

jdemeyer@sage4:~/sage-python3$ ./sage --python3
Python 3.8.0b1 (default, Jun 19 2019, 16:54:26) 
[GCC 4.9.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.link
<built-in function link>

I have no clue how this could happen.

jhpalmieri commented 5 years ago
comment:15

I tried this with OS X and ran into a problem with Pillow:

  File "setup.py", line 316, in build_extensions
    _add_directory(library_dirs, sysroot+"/usr/lib")
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Here is the problem code from Pillow's setup.py:

sysroot = sysconfig.get_config_var('Py_MACOS_SYSROOT')
_add_directory(library_dirs, sysroot+"/usr/lib")

Python 3.7.3 does this:

Python 3.7.3 (default, Jun 17 2019, 15:38:22) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('Py_MACOS_SYSROOT')
'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk'
>>>

The new Python does this:

Python 3.8.0b1 (default, Jun 19 2019, 11:09:54) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('Py_MACOS_SYSROOT')
>>>
jhpalmieri commented 5 years ago
comment:16

I see the same problem with matplotlib. For laughs, I put a print(dir(os)) command right before the line link_to = os.link. The output is identical to what I get when I do import os; dir(os) from sage --python3, except it is missing link.

If I run make -k, I also see problems building the Sage library, by the way: lots of errors of the form

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
jdemeyer commented 5 years ago
comment:17

Replying to @jhpalmieri:

I see the same problem with matplotlib. For laughs, I put a print(dir(os)) command right before the line link_to = os.link. The output is identical to what I get when I do import os; dir(os) from sage --python3, except it is missing link.

That makes no sense. Somebody is fooling with us...

Could you try also print(os)? Maybe there is a different os module shadowing the standard one or something...

Just in case, I checked that the problem unfortunately persists after make distclean.

And a Google search for this error doesn't find anything.

jhpalmieri commented 5 years ago
comment:18

I'm guessing the problem is coming from matplotlib's setup.py:

import os
try:
    del os.link
except AttributeError:
    pass
jhpalmieri commented 5 years ago
comment:19

My question is why it worked with Python 3.7.3 but not 3.8.0.

jdemeyer commented 5 years ago
comment:20

Replying to @jhpalmieri:

My question is why it worked with Python 3.7.3 but not 3.8.0.

A different implementation of pathlib: https://bugs.python.org/issue26978

jdemeyer commented 5 years ago

Description changed:

--- 
+++ 
@@ -9,4 +9,6 @@
 - https://github.com/python/cpython/pull/13959
 - https://github.com/python/cpython/pull/14193

-It is likely that these will be accepted by the next beta
+It is likely that these will be accepted by the next beta.
+
+This breaks matplotlib: https://github.com/matplotlib/matplotlib/issues/14580
jdemeyer commented 5 years ago
comment:21

I reported this to matplotlib: https://github.com/matplotlib/matplotlib/issues/14580

jdemeyer commented 5 years ago

Dependencies: #28023

jdemeyer commented 5 years ago

Description changed:

--- 
+++ 
@@ -10,5 +10,3 @@
 - https://github.com/python/cpython/pull/14193

 It is likely that these will be accepted by the next beta.
-
-This breaks matplotlib: https://github.com/matplotlib/matplotlib/issues/14580
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 5 years ago

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

296092ematplotlib 2.2.4
cc78db2Python 3.8.0
7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 5 years ago

Changed commit from 46837b7 to cc78db2

jdemeyer commented 5 years ago
comment:24

With matplotlib fixes, I'm getting these errors at runtime:

Exception ignored in: <sage.misc.weak_dict.WeakValueDictEraser object at 0x7f1bf60b5460>
Traceback (most recent call last):
  File "sage/misc/weak_dict.pyx", line 219, in sage.misc.weak_dict.WeakValueDictEraser.__call__ (build/cythonized/sage/misc/weak_dict.c:1968)
  File "sage/cpython/dict_del_by_value.pyx", line 375, in sage.cpython.dict_del_by_value.del_dictitem_by_exact_value (build/cythonized/sage/cpython/dict_del_by_value.c:2311)
  File "sage/cpython/dict_del_by_value.pyx", line 294, in sage.cpython.dict_del_by_value.ensure_allows_deletions (build/cythonized/sage/cpython/dict_del_by_value.c:2025)
AssertionError:
jdemeyer commented 5 years ago

Changed dependencies from #28023 to #28023, #28025

jdemeyer commented 5 years ago
comment:26

Also getargspec is completely broken, causing docbuild errors.

fchapoton commented 5 years ago
comment:27

see #27971 for getargspec ?

jdemeyer commented 5 years ago
comment:28

Replying to @fchapoton:

see #27971 for getargspec ?

I'll have to look at that. In any case, these seem to be new failures that did not exist on Python 3.7.

jhpalmieri commented 5 years ago
comment:29

Replying to @jhpalmieri:

I tried this with OS X and ran into a problem with Pillow: Here is the problem code from Pillow's setup.py:

sysroot = sysconfig.get_config_var('Py_MACOS_SYSROOT')
_add_directory(library_dirs, sysroot+"/usr/lib")

This is actually added in a Sage patch. It's failing because of the removal of macos_no_include.patch, which was added in #27631. I'll have to see if that patch is necessary anymore. If not, we need to unpatch Pillow.

jhpalmieri commented 5 years ago
comment:30

I still gets lots of errors as in comment:16. The build process for sagelib seems to go into an infinite loop. I just hit ctrl-C, and there are about 2400 instances of

************************************************************************
Error building the Sage library
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
  /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/logs/pkgs/sagelib-8.8.rc2.log
Describe your computer, operating system, etc.

in the log file, all connected with the same error message.

jdemeyer commented 5 years ago
comment:31

On which system are you seeing that? The build of the Sage library worked for me on Linux.

jhpalmieri commented 5 years ago
comment:32

Replying to @jdemeyer:

On which system are you seeing that? The build of the Sage library worked for me on Linux.

OS X, both 10.13 and 10.14.

jhpalmieri commented 5 years ago
comment:33

Replying to @jhpalmieri:

Replying to @jhpalmieri:

I tried this with OS X and ran into a problem with Pillow: Here is the problem code from Pillow's setup.py:

sysroot = sysconfig.get_config_var('Py_MACOS_SYSROOT')
_add_directory(library_dirs, sysroot+"/usr/lib")

This is actually added in a Sage patch. It's failing because of the removal of macos_no_include.patch, which was added in #27631. I'll have to see if that patch is necessary anymore. If not, we need to unpatch Pillow.

On an OS X 10.14 machine, one affected by #27631 (no /usr/include), pip builds just fine, so I think Python and pip are finding zlib, but Pillow does not.

If I keep the Pillow patch, it fails to build at all, with the error mentioned above. If I delete the Pillow patch and build, I see in its log file "The headers and library files could not be found for zlib".

jhpalmieri commented 5 years ago
comment:34

Attachment: sagelib-8.8.rc2.log

Here is the start of the log file. I've deleted more than 2000 repeated tracebacks.

jhpalmieri commented 5 years ago
comment:35

I think the problem is this (from the 3.8 changelog):

on macOS, the spawn start method is now used by default in multiprocessing

The documentation for multiprocessing then points to a bug report which discusses our old friend OBJC_DISABLE_INITIALIZE_FORK_SAFETY (#25921). So if we get multiprocessing to work properly on OS X, maybe we can remove the line

    export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

from sage-env.

In the meantime, I don't know how to get it working properly; using fork instead of spawn in src/setup.py is the only way I've found to proceed so far. I don't understand the distinctions in the multiprocessing documentation between fork and spawn to know what should be set differently to use spawn correctly.

jhpalmieri commented 5 years ago
comment:36

Using fork instead of spawn lets me build the Sage library:

diff --git a/src/setup.py b/src/setup.py
index 42ba3a63da..5e2d65f9cc 100755
--- a/src/setup.py
+++ b/src/setup.py
@@ -493,11 +493,12 @@ def execute_list_of_commands_in_parallel(command_list, nthreads):
         progress = progress_fmt.format(i+1, N)
         command_list[i] = command_list[i] + (progress,)

-    from multiprocessing import Pool
+    import multiprocessing as mp
+    mp.set_start_method('fork')
     # map_async handles KeyboardInterrupt correctly if an argument is
     # given to get().  Plain map() and apply_async() do not work
     # correctly, see Trac #16113.
-    pool = Pool(nthreads)
+    pool = mp.Pool(nthreads)
     result = pool.map_async(apply_func_progress, command_list, 1).get(99999)
     pool.close()
     pool.join()

I made a similar change to allow docbuilding to work. Now I get failures with docbuilding. First, lots of warnings:

..../local/lib/python3.8/site-packages/matplotlib/backends/backend_agg.py:531: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats

Then an error:

Traceback (most recent call last):
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
    main()
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 1731, in main
    builder()
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 352, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 548, in _wrapper
    build_many(build_ref_doc, L)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 289, in build_many
    ret = x.get(99999)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/multiprocessing/pool.py", line 768, in get
    raise self._value
OSError: WARNING: error while formatting arguments for sage.calculus.riemann.Riemann_Map.get_szego: bad operand type for unary -: 'NoneType'
jhpalmieri commented 5 years ago
comment:37

And of course I can't run any doctests because they use multiprocessing, and I haven't figured out how to tell doctesting (forker.py in particular) to use fork instead of spawn. Sage starts and some basic things work. The Jupyter notebook is giving me problems.

jdemeyer commented 5 years ago
comment:38

I also didn't manage to build the docs because the same error. I know it's a problem with sage_getargspec but I haven't checked further.

jhpalmieri commented 5 years ago
comment:39

I don't have a good solution for the problem with Pillow (comment:33). I have been deleting the patch and force installing zlib, which works for now.

jhpalmieri commented 5 years ago
comment:40

For the record: for the spawn vs. fork issue with OS X, I have had to make changes in three files: src/setup.py, src/sage_setup/docbuild/__init__.py, and Cython/Build/Dependencies.py. The changes are all pretty much the same; for example in docbuild/__init__.py:

diff --git a/src/sage_setup/docbuild/__init__.py b/src/sage_setup/docbuild/__init__.py
index 0f2700168a..6d1ff4e377 100644
--- a/src/sage_setup/docbuild/__init__.py
+++ b/src/sage_setup/docbuild/__init__.py
@@ -279,8 +279,9 @@ if not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3):
         # serialize this by taking the pool (and thus the call to fork()) out
         # completely: The call to Sphinx leaks memory, so we need to build each
         # document in its own process to control the RAM usage.
-        from multiprocessing import Pool
-        pool = Pool(NUM_THREADS, maxtasksperchild=1)
+        import multiprocessing
+        ctx = multiprocessing.get_context('fork')
+        pool = ctx.Pool(NUM_THREADS, maxtasksperchild=1)
         # map_async handles KeyboardInterrupt correctly. Plain map and
         # apply_async does not, so don't use it.
         x = pool.map_async(target, args, 1)
jhpalmieri commented 5 years ago
comment:41

I am also seeing the error "ImportError: cannot import name 'clock' from 'time' (unknown location)". I don't think this is coming from the Sage library, but rather from other packages, and in particular brial and sympy may need to be changed.

$ grep -R "from time import clock" local/lib/python3.8/
local/lib/python3.8//site-packages/sympy/plotting/pygletplot/plot_window.py:from time import clock
local/lib/python3.8//site-packages/brial/interpolate.py:from time import clock
local/lib/python3.8//site-packages/future/backports/test/pystone.py:from time import clock
local/lib/python3.8//site-packages/zmq/backend/cffi/_poll.py:    from time import clock as monotonic
local/lib/python3.8//site-packages/sage/coding/information_set_decoder.py:            from time import clock as process_time

Changing the import to something like

try:
    from time import perf_counter as clock # or process_time instead of perf_counter?
except ImportError:
    from time import clock

might work.

fchapoton commented 5 years ago
comment:42

Replying to @jhpalmieri:

I am also seeing the error "ImportError: cannot import name 'clock' from 'time' (unknown location)". I don't think this is coming from the Sage library, but rather from other packages, and in particular brial and sympy may need to be changed.

$ grep -R "from time import clock" local/lib/python3.8/
local/lib/python3.8//site-packages/sympy/plotting/pygletplot/plot_window.py:from time import clock
local/lib/python3.8//site-packages/brial/interpolate.py:from time import clock
local/lib/python3.8//site-packages/future/backports/test/pystone.py:from time import clock
local/lib/python3.8//site-packages/zmq/backend/cffi/_poll.py:    from time import clock as monotonic
local/lib/python3.8//site-packages/sage/coding/information_set_decoder.py:            from time import clock as process_time

Changing the import to something like

try:
    from time import perf_counter as clock # or process_time instead of perf_counter?
except ImportError:
    from time import clock

might work.