Closed jdemeyer closed 5 years ago
Author: Jeroen Demeyer
I don't think we should be in any hurry on this. All the last ~year's work of Python 3 porting has been targeting Python 3.6, and I think I'd rather stick with that before risking throwing everything out of whack again.
Between Python 3.6 and 3.7 I suspect the differences, if any, will be minor. But I'd rather try to stabilize on 3.6 first (to which we're quite close), then deal with those differences, rather than change Python versions again. Also I don't know that Python 3.7 has been ported to Cygwin yet (although it does contain a number of useful fixes for Cygwin, I do know).
That said, if you know any specific changes in Python 3.7 (other than the aforementioned Cygwin fixes) that would actively make our porting effort easier I'd consider it...
Jeroen has said elsewhere that islice will accept Sage integers instead of just ints.
I am again becoming tired and frustated by the slow pace of progress towards py3. And by the large amount of difficult work that remains..
Ah, I remember making an issue about that, but I don't think I followed what the resolution was.
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
1bfcdc8 | Upgrade to Python 3.7.0 |
Well there's this https://github.com/python/cpython/pull/1918 But unless there are other fixes I'm not seeing it's annoying that this was only fixed for islice
, where I believe there is other code affected by the same issue (e.g. range). Also we've already worked aroudn the majority of those cases (sigh...)
Replying to @fchapoton:
I am again becoming tired and frustated by the slow pace of progress towards py3. And by the large amount of difficult work that remains..
Unfortunately I'm not convinced that upgrading Python yet again is going to make that go any faster. In any case I don't actually have that much "difficult" work remaining. Just dozens of little issues. My Python 3 branch has < 600 modules failing, and most of it appears to minor issues (at one point I had it down to ~400 and I don't know why it ballooned up again, but one or two small changes in just the right places can do that with Sage...)
Also I'll add if you're not using a patched pynac you're going to have lots, lots more "serious" looking issues. We still need to get a pynac upgrade in (I've been manually installing pynac with my Python 3 fixes every time I re-build sage).
Replying to @embray:
That said, if you know any specific changes in Python 3.7 (other than the aforementioned Cygwin fixes) that would actively make our porting effort easier I'd consider it...
Replying to @embray:
I don't think we should be in any hurry on this. All the last ~year's work of Python 3 porting has been targeting Python 3.6, and I think I'd rather stick with that before risking throwing everything out of whack again.
First of all, I do not think that this 3.6 -> 3.7 will throw everything out of whack.
However, regardless of that, we'll have to upgrade sooner or later. I don't see the point of having a perfectly working Sage on an outdated Python version. And for the many issues that we still have to fix, it would be better to make sure that they work on Python 3.7 from the start.
I still think we have very hard remaining problems, among which
Attachment: python3-3.7.0.log
I get the following error
$ ./configure --with-python=3
...
$ make build
...
[python3-3.7.0] Testing importing of various modules...
[python3-3.7.0] Traceback (most recent call last):
[python3-3.7.0] File "<string>", line 1, in <module>
[python3-3.7.0] File "/home/vklein/odk/sage/local/var/tmp/sage/build/python3-3.7.0/src/Lib/ctypes/__init__.py", line 7, in <module>
[python3-3.7.0] from _ctypes import Union, Structure, Array
[python3-3.7.0] ModuleNotFoundError: No module named '_ctypes'
[python3-3.7.0] ctypes module failed to import
[python3-3.7.0] math module imported OK
[python3-3.7.0] hashlib module imported OK
[python3-3.7.0] crypt module imported OK
[python3-3.7.0] readline module imported OK
[python3-3.7.0] socket module imported OK
[python3-3.7.0] Error: One or more modules failed to import.
...
full python3.7 install log in attachement
Failure to build _ctypes
usually means a problem with ffi
(sometimes called libffi) may be there is minimum version requirement?
Indeed i don't get the error after installing libffi-dev
. Thanks !
Note : sage make build
works fine in python 3.6 without that.
next error during the build is :
Installing collected packages: ipykernel
Running setup.py install for ipykernel: started
Running command /home/vklein/odk/sage/local/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-qoxsao1r-build/setup.py';f=getattr(tokenize
, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" --no-user-cfg install --record /tmp/pip-vc23lh
pc-record/install-record.txt --single-version-externally-managed --compile
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-qoxsao1r-build/setup.py", line 91, in <module>
from ipykernel.kernelspec import write_kernel_spec, make_ipkernel_cmd, KERNEL_NAME
...
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/pexpect/spawnbase.py", line 224
def expect(self, pattern, timeout=-1, searchwindowsize=-1, async=False):
^
SyntaxError: invalid syntax
Running setup.py install for ipykernel: finished with status 'error'
Cleaning up...
Issue reference : https://github.com/pypa/pipenv/issues/956
To be honest we haven't updated pexpect in vanilla sage for a while. It may be a good idea to do that - in a separate ticket.
Do we need to update pexpect or to patch pipenv ?
No, pexpect
.
I am at pexpect-4.2.1
in sage-on-gentoo (without any patches) and sage is at 4.1.0 + some patches.
Replying to @kiwifb:
I am at
pexpect-4.2.1
in sage-on-gentoo (without any patches) and sage is at 4.1.0 + some patches.
The patches in Sage are for performance. Without those patches, pexpect
is really a lot slower.
Let's do that #25700.
Dependencies: #25700
Replying to @fchapoton:
I still think we have very hard remaining problems, among which
- sorting issues in graphs
- hashing and comparisons, especially for groups
- coercion framework have broken parts
Most of which Python 3.7 won't help with. The one area where it might is sorting of dicts, since dict insertion order is now preserved (hooray!)
Well, I'm not completely opposed to upgrading, just skeptical, and don't want to rush into it. When I get a chance I'll compare how my python3 branch fares with and without it.
Replying to @embray:
Well, I'm not completely opposed to upgrading, just skeptical, and don't want to rush into it. When I get a chance I'll compare how my python3 branch fares with and without it.
One of my goal reviewing this ticket is to produce a kind of diff between the doctests failures in 3.6 and 3.7. It can be helpful to decide what to do.
And now it's numpy's turn :
$ ./configure --with-python=3
...
$ make build
...
/home/vklein/odk/sage/local/lib/python3.7/distutils/dist.py:274: UserWarning: Un
known distribution option: 'define_macros'
warnings.warn(msg)
numpy/random/mtrand/mtrand.c: In function '__Pyx__ExceptionSave':
numpy/random/mtrand/mtrand.c:45208:19: error: 'PyThreadState {aka struct _ts}' h
as no member named 'exc_type'
*type = tstate->exc_type;
^
numpy/random/mtrand/mtrand.c:45209:20: error: 'PyThreadState {aka struct _ts}' h
as no member named 'exc_value'
*value = tstate->exc_value;
...
numpy's issue #10500. I am not sure what is the right call here i will try with numpy 1.14.5
When updating to numpy 1.14.5
with this following patch :
diff --git a/numpy/distutils/system_info.py b/numpy/distutils/system_info.py
index 93a8e6f..6e31d30 100644
--- a/numpy/distutils/system_info.py
+++ b/numpy/distutils/system_info.py
@@ -1740,7 +1740,7 @@ class blas_info(system_info):
lib = self.has_cblas(info)
if lib is not None:
info['language'] = 'c'
- info['libraries'] = [lib]
+ info['libraries'] = lib
info['define_macros'] = [('HAVE_CBLAS', None)]
self.set_info(**info)
@@ -1772,16 +1772,16 @@ class blas_info(system_info):
# check for cblas lib, and if not present check for blas lib.
try:
c.link_executable(obj, os.path.join(tmpdir, "a.out"),
- libraries=["cblas"],
+ libraries=["libraries"],
library_dirs=info['library_dirs'],
extra_postargs=info.get('extra_link_args', []))
- res = "cblas"
+ res = info["libraries"]
except distutils.ccompiler.LinkError:
c.link_executable(obj, os.path.join(tmpdir, "a.out"),
libraries=["blas"],
library_dirs=info['library_dirs'],
extra_postargs=info.get('extra_link_args', []))
- res = "blas"
+ res = ["blas"]
except distutils.ccompiler.CompileError:
res = None
finally:
I get the following error :
Found local metadata for numpy-1.14.5.p0
Using cached file /home/vklein/odk/sage/upstream/numpy-1.14.5.zip
numpy-1.14.5.p0
====================================================
Setting up build directory for numpy-1.14.5.p0
Finished extraction
Applying patches from ../patches...
Applying ../patches/numpy-1.14.5-no-hardcode-blas.patch
patching file numpy/distutils/system_info.py
****************************************************
Host system:
Linux tuono 4.13.0-45-generic #50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.10' --with-bugurl=file:///usr/s
hare/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suff
ix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=po
six --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcx
x-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --en
able-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/
usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 -
-with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share
/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --wit
h-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-g
nu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)
****************************************************
Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:
- `pip install .` (from a git repo or downloaded source
release)
- `pip install numpy` (last NumPy release on PyPi)
blas_opt_info:
blas_mkl_info:
Disabled blas_mkl_info: (MKLROOT is None)
Disabled blas_mkl_info: (MKLROOT is None)
customize UnixCCompiler
libraries mkl_rt not found in []
NOT AVAILABLE
blis_info:
customize UnixCCompiler
libraries blis not found in ['/home/vklein/odk/sage/local/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-li
nux-gnu']
NOT AVAILABLE
openblas_info:
customize UnixCCompiler
Disabled openblas_info: (OPENBLAS is None)
customize UnixCCompiler
libraries openblas not found in []
NOT AVAILABLE
atlas_3_10_blas_threads_info:
Disabled atlas_3_10_blas_threads_info: (PTATLAS is None)
customize UnixCCompiler
libraries tatlas not found in []
NOT AVAILABLE
atlas_3_10_blas_info:
Disabled atlas_3_10_blas_info: (ATLAS is None)
customize UnixCCompiler
libraries satlas not found in []
NOT AVAILABLE
atlas_blas_threads_info:
Disabled atlas_blas_threads_info: (PTATLAS is None)
customize UnixCCompiler
libraries ptf77blas,ptcblas,atlas not found in []
NOT AVAILABLE
atlas_blas_info:
Disabled atlas_blas_info: (ATLAS is None)
customize UnixCCompiler
libraries f77blas,cblas,atlas not found in []
NOT AVAILABLE
blas_info:
customize UnixCCompiler
customize UnixCCompiler
C compiler: gcc -DNDEBUG -g -fwrapv -O3 -Wall -D__CEPHES_COMPLEX_H -fPIC
creating /tmp/tmpyq7p80or/tmp
creating /tmp/tmpyq7p80or/tmp/tmpyq7p80or
compile options: '-I/home/vklein/odk/sage/local/include -c'
gcc: /tmp/tmpyq7p80or/source.c
gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/lib -llibraries -o /tmp/tmpyq7p80or/a.o
ut
/usr/bin/ld: cannot find -llibraries
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -llibraries
collect2: error: ld returned 1 exit status
gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/lib -lblas -o /tmp/tmpyq7p80or/a.out
Running from numpy source directory.
/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/system_info.py:624: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
self.calc_info()
/usr/bin/ld: cannot find -lblas
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lblas
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
File "/home/vklein/odk/sage/local/lib/python3.7/distutils/unixccompiler.py", line 197, in link
self.spawn(linker + ld_args)
File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/ccompiler.py", line 89,
in <lambda>
m = lambda self, *args, **kw: func(self, *args, **kw)
File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/ccompiler.py", line 152,
in CCompiler_spawn
raise DistutilsExecError('Command "%s" failed with exit status %d%s' % (cmd, s, msg))
distutils.errors.DistutilsExecError: Command "gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage
/local/lib -llibraries -o /tmp/tmpyq7p80or/a.out" failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/system_info.py", line 17
77, in has_cblas
extra_postargs=info.get('extra_link_args', []))
File "/home/vklein/odk/sage/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
debug, extra_preargs, extra_postargs, None, target_lang)
File "/home/vklein/odk/sage/local/lib/python3.7/distutils/unixccompiler.py", line 199, in link
raise LinkError(msg)
distutils.errors.LinkError: Command "gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/li
b -llibraries -o /tmp/tmpyq7p80or/a.out" failed with exit status 1
Advices or help are welcome. I get the same errors with a sage with python2.7.
Ok i will try.
Replying to @vinklein:
Ok i will try.
If you look in the setup.py
file of numpy
, you'll see
from setuptools import setup
if run_build:
from numpy.distutils.core import setup
cwd = os.path.abspath(os.path.dirname(__file__))
if not os.path.exists(os.path.join(cwd, 'PKG-INFO')):
# Generate Cython sources, unless building from source release
generate_cython()
So I guess that removing the PKG-INFO
file in spkg-install
will force recythonization.
Your trick almost work. The problem is that the numpy file tools/cythonize.py is not in numpy's source releases.
I confirm that numpy's install work if we remove PKG-INFO
and add cythonize.py
file.
Changed branch from u/jdemeyer/upgrade_to_python_3_7_0 to u/vklein/upgrade_to_python_3_7_0
Can we please deal with numpy in a separate ticket.
I don't know. The patch doesn't make sense without python 3.7.
Replying to @vinklein:
I don't know. The patch doesn't make sense without python 3.7.
It's needed for Python 3.7, but the fix is unrelated to it.
If you're patching the sources anyway, I would prefer to patch setup.py
instead of removing PKG-INFO
.
Sure ! It will be cleaner to patch setup.py
.
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
Changed dependencies from #25700 to #25700, #25755
scipy errors seems pretty similar to numpy's:
scipy/cluster/_vq.c: In function '__Pyx__ExceptionSave':
scipy/cluster/_vq.c:9728:19: error: 'PyThreadState {aka struct _ts}' has no member named 'exc_type'
*type = tstate->exc_type;
^
scipy/cluster/_vq.c:9729:20: error: 'PyThreadState {aka struct _ts}' has no member named 'exc_value'
*value = tstate->exc_value;
With likely the same cause.
In setup.py
file of scipy
:
if not os.path.exists(os.path.join(cwd, 'PKG-INFO')):
# Generate Cython sources, unless building from source release
generate_cython()
Changed dependencies from #25700, #25755 to #25700, #25755, #25759
Building sage with this ticket and it's dependencies works fine.
I have made two run of sage -t -a --long
, one with a sage python3.6 and one with this ticket. Comparing the logs of the two runs (and weakly idenfying a failure by the tuple (
nb doctests failures with python3.6: 42990
nb doctests failures with python3.7: 43643
new python3.7 failures : 1366
python3.6 failures not appearing with python 3.7 : 713
Two exemple failures among 1366 :
File "src/sage/combinat/finite_state_machine.py", line 10190, in sage.combinat.finite_state_machine.FiniteStateMachine.asymptotic_moments
Failed example:
moments = T.asymptotic_moments()
Exception raised:
Traceback (most recent call last):
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 573, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.combinat.finite_state_machine.FiniteStateMachine.asymptotic_moments[2]>", line 1, in <module>
moments = T.asymptotic_moments()
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/combinat/finite_state_machine.py", line 10540, in asymptotic_moments
if not final_component.digraph().is_aperiodic():
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/graphs/digraph.py", line 3330, in is_aperiodic
import networkx
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/__init__.py", line 114, in <module>
import networkx.generators
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/generators/__init__.py", line 13, in <module>
from networkx.generators.geometric import *
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/generators/geometric.py", line 27, in <module>
from scipy.spatial import cKDTree as KDTree
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/scipy/spatial/__init__.py", line 95, in <module>
from .qhull import *
File "qhull.pyx", line 2224, in init scipy.spatial.qhull
AttributeError: 'cython_function_or_method' object has no attribute '__func__'
sage -t --long src/sage/combinat/dyck_word.py
**********************************************************************
File "src/sage/combinat/dyck_word.py", line 873, in sage.combinat.dyck_word.DyckWord.plot
Failed example:
w.plot()
Expected:
Graphics object consisting of 1 graphics primitive
Got:
doctest:warning
File "/home/vklein/odk/sage/src/bin/sage-runtests", line 127, in <module>
err = DC.run()
File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/control.py", line 1172, in run
self.run_doctests()
...
File "/home/vklein/odk/sage/local/lib/python3.7/collections/__init__.py", line 52, in __getattr__
DeprecationWarning, stacklevel=2)
File "/home/vklein/odk/sage/local/lib/python3.7/warnings.py", line 99, in _showwarnmsg
msg.file, msg.line)
:
DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
Graphics object consisting of 1 graphics primitive
Note : To see the script and logs used https://github.com/vinklein/Trac25680
Changed dependencies from #25700, #25755, #25759 to #25700, #25755, #24766
Tarball: https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz
CC: @fchapoton @embray @kiwifb @mkoeppe @slel
Component: packages: standard
Keywords: upgrade
Author: Jeroen Demeyer
Branch:
5a09cf1
Reviewer: Vincent Klein, Erik Bray, Frédéric Chapoton
Issue created by migration from https://trac.sagemath.org/ticket/25680