Python 3 preparation: Adapt to the changed integer division semantics (/ vs. //)

3a5ab40a-5e15-47f8-8025-1665d51c0660 commented 10 years ago

See original ticket below. We can still use this as a meta-ticket for tracking all division-related changes needed for Python 3. Existing tickets include:

Progress Report

Modules needing work

Based on some grepping results, the following modules appear to contain a class that implements `__div__` but not `__truediv__` (the latter should be preferred in all cases):	Ticket	Module
#?????	src/sage/algebras/lie_algebras/lie_algebra_element.pyx
#?????	src/sage/categories/examples/lie_algebras.py
#24805	~~src/sage/ext/fast_callable.pyx~~ (this is just from a Python 2 only test case from the relevant ticket)
#?????	src/sage/groups/additive_abelian/qmodnz_element.py
#?????	src/sage/matrix/matrix_gfpn_dense.pyx
#?????	src/sage/rings/integer.pyx
#?????	src/sage/rings/rational.pyx
#?????	src/sage/structure/category_object.pyx

The new semantics can be enabled in Python 2 with from `future` import division.

This will require checking where to replace / by //

Changes according to libfuturize/fixes/division.py:

Add
from __future__ import division

This ticket is tracked as a dependency of meta-ticket ticket:15980.

To test what remains to be done, one can use:

export PYTHONIOENCODING="utf-8"
source ./local/bin/sage-env
python -Qnew ./local/bin/sage-runtests --all

CC: @embray @jdemeyer @kiwifb @tscrim @jhpalmieri @vinklein

Component: python3

Keywords: division

Issue created by migration from https://trac.sagemath.org/ticket/15995

3a5ab40a-5e15-47f8-8025-1665d51c0660 commented 10 years ago

comment:1

The first attempt went like this: futurize -w -f division src/.

This added from `future` import division to 411 py modules.

Running the doctests ./sage -t -p --all gave billions of lines of doctest failures ... - I had to cancel the run.

But this result should not have come as a surprise because I had not fixed anything yet ...

The task is to identify those / and /= that operate solely on ints and longs and then change them to // and //=.

The Python source distribution has a script Tools/scripts/finddiv.py which has identified similar 416 py modules as the above programs (it performs clearly better as a simple grep since it tokenizes the code). It also outputs those 2625 lines that contain the / and /= operators - but for all types of operands (the analysis is purely syntactical).

There is another script Tools/scripts/fixdiv.py that uses the output of

python -Qwarnall yourscripts 2>the-warnings to create a simple (commented) patch file for those division operators that only deal with ints and longs. This is possible since it uses the type information obtained by running the Python programs. This should be much more useful!

But currently I am stuck:

I don't know how to pass the option -Qwarnall to the Python interpreter when running the doctests
and the warning output screws up the doctests. It may be possible to use the warnings module to write the warnings to a file instead if stderr (by replacing warnings.showwarning() with a custom function).

3a5ab40a-5e15-47f8-8025-1665d51c0660 commented 10 years ago

comment:2

I finally manage to run the doctests with the Python command line option -3. This is how I did it:

...:~/sage-6.2.beta8$ . src/bin/sage-env
...:~/sage-6.2.beta8$ python -3 ./local/bin/sage-runtests  src/sage 1>logs/sage-warnings3-stdout.txt 2>logs/sage-warnings3-stderr.txt

A summary of stderr output is

nbr_lines= 133
   83  Overriding __eq__ blocks inheritance of __hash__ in 3.x
   27  classic int division
   18  type inequality comparisons not supported in 3.x
    2  the cmp argument is not supported in 3.x
    1  execfile() not supported in 3.x; use exec()
    1  file.softspace not supported in 3.x
    1  tmpfile has been removed in 3.x; use the tempfile module

and of stdout

nbr_lines= 39630
19386  type inequality comparisons not supported in 3.x
 8779  Overriding __eq__ blocks inheritance of __hash__ in 3.x
 4039  CObject type is not supported in 3.x. Please use capsule objects instead.
 2139  reduce() not supported in 3.x; use functools.reduce()
 2017  classic int division
 2000  the cmp argument is not supported in 3.x
  789  comparing unequal types not supported in 3.x
  262  __getitem__ not supported for exception classes in 3.x; use args attribute
   88  sys.exc_clear() not supported in 3.x; use except clauses
   57  in 3.x, __getslice__ has been removed; use __getitem__
   14  file.softspace not supported in 3.x
   12  the commands module has been removed in Python 3.0; use the subprocess module instead
    7  execfile() not supported in 3.x; use exec()
    7  tmpfile has been removed in 3.x; use the tempfile module
    5  dict inequality comparisons not supported in 3.x
    3  
    2  Use the Permutations object instead.
    2  This class is replaced by Matrix_modn_dense_float/Matrix_modn_dense_double.
    2  to_permutation() is deprecated. Use instead reading_word_permutation()
    2  this class is deprecated. Use RibbonShapedTableau instead
    2  BaseException.message has been deprecated as of Python 2.6
    2  The 'new' module has been removed in Python 3.0; use the 'types' module instead.
    2  this is done automatically by the doctest framework
    2  derangements() is deprecated. Use Derangements instead.
    1  The output of Transducer.cartesian_product will change. Please use Transducer.intersection for the original output.
    1  Substitution using function-call syntax and unnamed arguments is deprecated and will be removed from a future release of Sage; you can use named arguments instead, like EXPR(x=..., y=...)
    1  This method is deprecated. Use triangulate() instead.
    1  In 3.x, reload() is renamed to imp.reload()
    1  dict.has_key() not supported in 3.x; use the in operator
    1  The output of
    1  Substitution using
    1  This method is deprecated. Use triangulate().simplicial_complex() instead.
    1  Variable of integration should be specified explicitly.
    1  OrderedAlphabet is deprecated; use Alphabet instead.

Unfortunately typical stdout printout looks like

**********************************************************************
File "src/sage/modular/modsym/space.py", line 929, in sage.modular.modsym.space.ModularSymbolsSpace._q_expansion_module
Failed example:
    ModularSymbols(11, 2, base_ring=GF(4,'a')).cuspidal_submodule()._q_expansion_module(prec=4, algorithm="hecke")
Expected:
    Vector space of degree 4 and dimension 1 over Finite Field in a of size 2^2
    Basis matrix:
    [0 1 0 1]
Got:
    doctest:127: DeprecationWarning: type inequality comparisons not supported in 3.x
    doctest:511: DeprecationWarning: classic int division
    doctest:250: DeprecationWarning: type inequality comparisons not supported in 3.x
    doctest:585: DeprecationWarning: classic int division
    doctest:286: DeprecationWarning: classic int division
    doctest:396: DeprecationWarning: classic int division
    doctest:6464: DeprecationWarning: classic int division
    Vector space of degree 4 and dimension 1 over Finite Field in a of size 2^2
    Basis matrix:
    [0 1 0 1]
**********************************************************************

and I don't know how to correlate line likes doctest:511 with the original line.

The line in stderr are more useful, but are few and far between

~/sage-6.2.beta8/local/lib/python2.7/site-packages/sage/groups/perm_gps/cubegroup.py:368: DeprecationWarning: classic int division
  2:  [1/2,3/2,5/2], #  ub

a-andre commented 10 years ago

comment:4

Hi Wilfried, in #16371 I changed some / to //, maybe you can review it.

BTW: how you got the summary of stdout and stderr?

a-andre commented 10 years ago

Dependencies: #16371

3a5ab40a-5e15-47f8-8025-1665d51c0660 commented 10 years ago

Small Python script to create the summary

3a5ab40a-5e15-47f8-8025-1665d51c0660 commented 10 years ago

comment:5

Attachment: read_deprecation-warnings-v1b.py.gz

For the summary see the small attached Py script ;-)

I will look at your ticket.

3a5ab40a-5e15-47f8-8025-1665d51c0660 commented 9 years ago

comment:7

I created the ticket #18578 to address classes which define the special function __div__() (and __truediv__()). See also the comments 5 and 7 in #16731.

3a5ab40a-5e15-47f8-8025-1665d51c0660 commented 9 years ago

Changed dependencies from #16371 to #16371, #18578

jdemeyer commented 9 years ago

comment:8

One very small effort in the spirit of this ticket: #19535

jdemeyer commented 9 years ago

comment:9

Also #19536.

edd8e884-f507-429a-b577-5d554626c0fe commented 8 years ago

comment:10

Replying to @wluebbe:

But currently I am stuck:

I don't know how to pass the option -Qwarnall to the Python interpreter when running the doctests

Could you try the following:

From a terminal, run sage -sh

From this shell, run:

export PYTHONIOENCODING="utf-8"
python -Qwarnall $(which sage-runtests) <options>

Where <options> is something like --all --long (depending on the options you are using for doctesting) ?

fchapoton commented 8 years ago

Description changed:

--- 
+++ 
@@ -10,3 +10,11 @@

This ticket is tracked as a dependency of meta-ticket ticket:15980. + +To test what remains to be done, one can use: + + +export PYTHONIOENCODING="utf-8" +source ./local/bin/sage-env +python -Qnew ./local/bin/sage-runtests --all +

fchapoton commented 8 years ago

comment:13

one step done in #21127

fchapoton commented 7 years ago

comment:15

another step in #22767

fchapoton commented 7 years ago

comment:17

another step in #23731

fchapoton commented 6 years ago

comment:18

another step in #24548