py3: fix src/sage/misc/nested_class.pyx

[fchapoton]

traceback:

sage -t --long src/sage/misc/nested_class.pyx
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 39, in sage.misc.nested_class
Failed example:
    A1.A2.A3
Expected:
    <class sage.misc.nested_class.A3 at ...>
Got:
    <class 'sage.misc.nested_class.A1.A2.A3'>
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 42, in sage.misc.nested_class
Failed example:
    nested_pickle(A1)
Expected:
    <class sage.misc.nested_class.A1 at ...>
Got:
    <class 'sage.misc.nested_class.A1'>
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 45, in sage.misc.nested_class
Failed example:
    A1.A2
Expected:
    <class sage.misc.nested_class.A1.A2 at ...>
Got:
    <class 'sage.misc.nested_class.A1.A2'>
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 48, in sage.misc.nested_class
Failed example:
    A1.A2.A3
Expected:
    <class sage.misc.nested_class.A1.A2.A3 at ...>
Got:
    <class 'sage.misc.nested_class.A1.A2.A3'>
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 131, in sage.misc.nested_class.modify_for_nested_pickle
Failed example:
    getattr(module, 'A.B', 'Not found')
Expected:
    <class '__main__.X.A.B'>
Got:
    <class '__main__.A.B'>
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 133, in sage.misc.nested_class.modify_for_nested_pickle
Failed example:
    getattr(module, 'X.A.B', 'Not found')
Expected:
    <class '__main__.X.A.B'>
Got:
    <class '__main__.A.B'>
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 219, in sage.misc.nested_class.nested_pickle
Failed example:
    getattr(module, 'A.B', 'Not found')
Expected:
    <class __main__.A.B at ...>
Got:
    <class '__main__.A.B'>
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 259, in sage.misc.nested_class.NestedClassMetaclass
Failed example:
    A3.B.__name__
Expected:
    'A3.B'
Got:
    'B'
**********************************************************************
File "src/sage/misc/nested_class.pyx", line 261, in sage.misc.nested_class.NestedClassMetaclass
Failed example:
    getattr(sys.modules['__main__'], 'A3.B', 'Not found')
Expected:
    <class '__main__.A3.B'>
Got:
    'Not found'
**********************************************************************
4 items had failures:
   4 of  14 in sage.misc.nested_class
   2 of   6 in sage.misc.nested_class.NestedClassMetaclass
   2 of  23 in sage.misc.nested_class.modify_for_nested_pickle
   1 of   9 in sage.misc.nested_class.nested_pickle
    [68 tests, 9 failures, 1.77 s]

CC: @embray @jdemeyer @kiwifb

Component: python3

Author: Erik Bray, Kwankyu Lee

Branch/Commit: 562ff1e

Reviewer: Frédéric Chapoton, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/27692

[embray]

comment:42

Replying to @jhpalmieri:

After I fix the references in dyck_word.py, I get a ton of other "Duplicate citation" and "Duplicate explicit target names" from other files in sage/combinat. For example:

I didn't have this problem before, so is it possible it was recently introduced? Perhaps it's just another citation that needs to be moved to the module-level, or the master citations list?

[jhpalmieri]

comment:43

It may only appear if you do make doc-clean. My guess is that whatever is forcing us to move the references in dyck_word.py to the module-level is also going to force us to move other references. The references in FullyPackedLoop are in a class with the decoration @add_metaclass(InheritComparisonClasscallMetaclass). The reference Combe2019 is in a method with @cached_method. Do we have to move all references in classes, methods, with these decorations to the module-level?

[jhpalmieri]

comment:44

Another issue: in tableau_tuple.py we have

from sage.combinat.tableau import (Tableau, ...)

and later

class TableauTuple(CombinatorialElement):
    ...
    Element = Tableau

and now Sphinx thinks that all of the references in the Element class are duplicates of those from Tableau. This seems to be fixed by moving all of the references to the master bibliography file.

[jhpalmieri]

Changed branch from u/embray/python3/ticket-27692 to u/jhpalmieri/python3/ticket-27692

[jhpalmieri]

comment:46

I'm getting some doctest failures with Python 3. Does anyone else see them?

----------------------------------------------------------------------
sage -t --long src/sage/categories/category.py  # 5 doctests failed
sage -t --long src/sage/categories/primer.py  # 1 doctest failed
sage -t --long src/sage/categories/category_with_axiom.py  # 5 doctests failed
sage -t --long src/sage/categories/homsets.py  # 1 doctest failed
sage -t --long src/sage/categories/polyhedra.py  # 1 doctest failed
----------------------------------------------------------------------

For example:

sage -t --long src/sage/categories/homsets.py
**********************************************************************
File "src/sage/categories/homsets.py", line 56, in sage.categories.homsets.HomsetsCategory.default_super_categories
Failed example:
    type(H)
Expected:
    <class 'sage.categories.additive_monoids.AdditiveMonoids.Homsets_with_category'>
Got:
    <class 'sage.categories.additive_monoids.Homsets_with_category'>

---

New commits:

6ce1df6	trac 27692: pyflakes cleanup in sage_autodoc.py
4475f30	trac 27692: move some references to master bibliography file

[jhpalmieri]

Changed commit from 8337e67 to 4475f30

[fchapoton]

comment:47

I can confirm the same failure in categories/ with the present branch here.

[fchapoton]

comment:48

This has doctests failure with python3

[embray]

comment:49

Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).

[fchapoton]

comment:50

Is there any hope for progress here ?

EDIT: the branch is now red

[fchapoton]

comment:51

I have made #28105 for just moving the references.

[fchapoton]

comment:52

Erik, John ? The branch is now red. Is there any hope to make at least a partial progress on this file ?

[jhpalmieri]

comment:53

Replying to @embray:

There does appear to be a legitimate issue here w.r.t. Sphinx.

In the class QuasiSymmetricFunctions the nested class Fundamental also has an alias of just F. That is, in the class body it has:

F = Fundamental

just as a shortcut. This does not affect the qualified name of the class, which is still correct:

sage: QuasiSymmetricFunctions.Fundamental.__qualname__
'QuasiSymmetricFunctions.Fundamental'

but for some reason Sphinx is outputting the docs for QuasiSymmetricFunctions with this nested class listed as just "F" instead of "Fundamental", whereas in the current docs it's listed correctly as "Fundamental".

I don't see this problem, at least with a Python 3 build. I see

F
    alias of QuasiSymmetricFunctions.Fundamental

class Fundamental(QSym)
    ...

Without this branch, the first two lines are missing, but everything else looks the same.

[jhpalmieri]

comment:54

On the other hand, with this branch I see a new py3 doctest failure:

File "src/sage/misc/c3_controlled.pyx", line 449, in sage.misc.c3_controlled.CmpKey
Failed example:
    Sets().Infinite()._cmp_key
Expected:
    (4, ...)
Got:
    (0, 40)
**********************************************************************
1 item had failures:
   1 of  27 in sage.misc.c3_controlled.CmpKey

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

67f6132	py3: Make NestedClassMetaclass do nothing on Python 3
5f63e7d	Fix up sage_autodoc to work with normal Python 3 nested classes with qualnames,
e8a9b9f	trac 27692: pyflakes cleanup in sage_autodoc.py

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 4475f30 to e8a9b9f

[jhpalmieri]

comment:56

Rebased to 8.9.beta2.

[jhpalmieri]

comment:57

In my opinion, if the new doctest failure in c3_controlled.pyx can be fixed, then this is ready to go. I don't know what's causing it, though.

[fchapoton]

comment:58

This seems to be an issue with class name of Sets().Infinite()

in the flags tables, one sees

{'FacadeSets': 1,
 'FiniteSets': 2,
 'Sets.Infinite': 4,
 'EnumeratedSets': 8,

where the second line is the only line with a dot inside.

[jhpalmieri]

comment:59

It may be more complicated than that. I made this change:

diff --git a/src/sage/misc/c3_controlled.pyx b/src/sage/misc/c3_controlled.pyx
index 323086fb8e..9b404e8b70 100644
--- a/src/sage/misc/c3_controlled.pyx
+++ b/src/sage/misc/c3_controlled.pyx
@@ -375,7 +375,7 @@ from sage.structure.dynamic_class import dynamic_class
 ##############################################################################

 cdef tuple atoms = ("FacadeSets",
-                    "FiniteSets", "Sets.Infinite", "EnumeratedSets", "SetsWithGrading",
+                    "FiniteSets", "InfiniteSets", "EnumeratedSets", "SetsWithGrading",
                     "Posets", "LatticePosets", "Crystals", "AdditiveMagmas",
                     "FiniteDimensionalModules", "GradedModules", "ModulesWithBasis",
                     "Magmas", "Semigroups", "Monoids", "PermutationGroups",

and then added a statement to print flags. I got this:

{'FacadeSets': 1, 'FiniteSets': 2, 'InfiniteSets': 4, 'EnumeratedSets': 8, 'SetsWithGrading': 16, 'Posets': 32, 'LatticePosets': 64, 'Crystals': 128, 'AdditiveMagmas': 256, 'FiniteDimensionalModules': 512, 'GradedModules': 1024, 'ModulesWithBasis': 2048, 'Magmas': 4096, 'Semigroups': 8192, 'Monoids': 16384, 'PermutationGroups': 32768, 'MagmasAndAdditiveMagmas': 65536, 'Rngs': 131072, 'Domains': 262144, 'HopfAlgebras': 524288}
sage: Sets().Finite()._cmp_key
(2, 55)
sage: Sets().Infinite()._cmp_key
(0, 40)
sage: Sets().Enumerated()._cmp_key
(8, 42)

So although InfiniteSets is now in flags with value of 4, the _cmp_key is still 0 for Sets().Infinite().

[jhpalmieri]

comment:60

Without this branch:

sage: C = Sets().Infinite()
sage: C.__class__.__name__
'Sets.Infinite_with_category'

With this branch:

sage: C = Sets().Infinite()
sage: C.__class__.__name__
'Infinite_with_category'

Changing Sets.Infinite to just Infinite in atoms fixes the doctest, but it doesn't feel like the right thing to do.

[jhpalmieri]

comment:61

This seems like a direct consequence of the change in nested_class.pyx. Do we do:

if PY2:
    ... define atoms using Sets.Infinite ...
else:
    ... define atoms using Infinite ...

Seems ugly.

[jhpalmieri]

comment:62

For example:

diff --git a/src/sage/misc/c3_controlled.pyx b/src/sage/misc/c3_controlled.pyx
index 323086fb8e..2c7cd0f573 100644
--- a/src/sage/misc/c3_controlled.pyx
+++ b/src/sage/misc/c3_controlled.pyx
@@ -369,13 +369,19 @@ from sage.misc.classcall_metaclass import ClasscallMetaclass, typecall
 from sage.misc.cachefunc import cached_function, cached_method
 from sage.misc.lazy_attribute import lazy_attribute
 from sage.structure.dynamic_class import dynamic_class
+from six import PY2

 ##############################################################################
 # Implementation of the total order between categories
 ##############################################################################

+if PY2:
+    infinitesets = "Sets.Infinite"
+else:
+    infinitesets = "Infinite"
+
 cdef tuple atoms = ("FacadeSets",
-                    "FiniteSets", "Sets.Infinite", "EnumeratedSets", "SetsWithGrading",
+                    "FiniteSets", infinitesets, "EnumeratedSets", "SetsWithGrading",
                     "Posets", "LatticePosets", "Crystals", "AdditiveMagmas",
                     "FiniteDimensionalModules", "GradedModules", "ModulesWithBasis",
                     "Magmas", "Semigroups", "Monoids", "PermutationGroups",

I don't like it, but it works.

[kwankyu]

New commits:

4214075	py3: Make NestedClassMetaclass do nothing on Python 3
291d584	Move this reference to the module level.
8337e67	Fix up sage_autodoc to work with normal Python 3 nested classes with qualnames,
6ce1df6	trac 27692: pyflakes cleanup in sage_autodoc.py
4475f30	trac 27692: move some references to master bibliography file
78ab9c5	Merge branch 'develop'
fb6d5fa	Keep NestedClassMetaClass functional even with Python 3

[kwankyu]

Changed branch from u/jhpalmieri/python3/ticket-27692 to public/27692

[kwankyu]

Changed commit from e8a9b9f to fb6d5fa

[kwankyu]

comment:64

In the commit fb6d5fa, I followed different approach to solve the issue. I keep NestedClassMetaclass as it was even with python3, but fixes doctests appropriately for both python2 and python3. See the note in the module docstring.

I think this approach is simple and the right one at this stage.

To a reviewer: I am not completely sure about what I wrote in the note. Please correct the note.

[jhpalmieri]

comment:66

Tests pass for me. The approach seems okay to me, but I am not an expert on this aspect of the code. What do others think? @embray?

[kwankyu]

comment:67

I add that Erik's approach should be considered when we drop the support on python2 and remove the nested class module altogether.

[jhpalmieri]

comment:68

It would be nice to improve the documentation. This is not a new addition, but what do "save_global" and "load_global" refer to? I guess they are methods in Python's pickle module, but they are undocumented, which makes them less than ideal as reference points. (Fixing this could certainly wait until another ticket.)

[jhpalmieri]

comment:69

The only issue here is the note added by klee. Is it correct? If so, I think this is ready to go.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

04645f1

Correct the note

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from fb6d5fa to 04645f1

[kwankyu]

comment:71

Replying to @jhpalmieri:

It would be nice to improve the documentation. This is not a new addition, but what do "save_global" and "load_global" refer to? I guess they are methods in Python's pickle module, but they are undocumented, which makes them less than ideal as reference points. (Fixing this could certainly wait until another ticket.)

They are C functions in cPickle module (actually cPickle.c), and not visible to a user.

[kwankyu]

comment:72

In the last commit, I corrected the note, after investigating cPickle of python2 and the pickle module of python3. So I am now sure that what it says is correct.

I think this ticket is now good to go.

[jhpalmieri]

comment:73

Replying to @kwankyu:

Replying to @jhpalmieri:

It would be nice to improve the documentation. This is not a new addition, but what do "save_global" and "load_global" refer to? I guess they are methods in Python's pickle module, but they are undocumented, which makes them less than ideal as reference points. (Fixing this could certainly wait until another ticket.)

They are C functions in cPickle module (actually cPickle.c), and not visible to a user.

Exactly. Not only not visible to a user, but as I said, not documented anywhere, so having them as part of our documentation is not helpful.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

040265b

Remove mentioning undocumented functions in cPickle

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 04645f1 to 040265b

[kwankyu]

comment:75

Replying to @jhpalmieri:

Replying to @kwankyu:

Replying to @jhpalmieri:

It would be nice to improve the documentation. This is not a new addition, but what do "save_global" and "load_global" refer to? I guess they are methods in Python's pickle module, but they are undocumented, which makes them less than ideal as reference points. (Fixing this could certainly wait until another ticket.)

They are C functions in cPickle module (actually cPickle.c), and not visible to a user.

Exactly. Not only not visible to a user, but as I said, not documented anywhere, so having them as part of our documentation is not helpful.

Ok. Then let's remove them here.

[fchapoton]

Changed author from Erik Bray to Erik Bray, Kwankyu Lee

[fchapoton]

Changed reviewer from Frédéric Chapoton to Frédéric Chapoton, John Palmieri, Kwankyu Lee

[fchapoton]

comment:76

I think that it is more than time to move on here.

Given that John and Kwankyu said "good to go", I am setting to positive

[vbraun]

comment:77

Now it fails on py2 with:

[dochtml] OSError: /var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py:docstring of sage.combinat.tableau.Tableaux.Element.promotion:40: WARNING: Duplicate explicit target name: "hai1992".

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

a1e3a4c	Merge branch 'develop' into t/27692/public/27692
86df126	trac 27692: fix one citation

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 040265b to 86df126

[jhpalmieri]

comment:79

This should fix the problem.

[vbraun]

comment:80

Now the pdf docs are broken

[docpdf] Underfull \hbox (badness 10000) in paragraph at lines 4766--4767
[docpdf] \T1/ptm/m/n/10 Bases: [][]\T1/pcr/m/n/10 sage.categories.category_with_axiom.
[docpdf] 
[docpdf] ! LaTeX Error: Too deeply nested.
[docpdf] 
[docpdf] See the LaTeX manual or LaTeX Companion for explanation.
[docpdf] Type  H <return>  for immediate help.
[docpdf]  ...                                              
[docpdf]                                                   
[docpdf] l.4779 \begin{itemize}
[docpdf]                       
[docpdf] ? 
[docpdf] ! Emergency stop.
[docpdf]  ...                                              

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 86df126 to 2f3fc35

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

13eb46f	Merge branch 'develop'
fb30abb	Merge branch 'public/27692'
fefe75e	Fix class documenter in sage_autodoc
2f3fc35	Recover some original docstrings

[kwankyu]

comment:82

Seems to have fixed pdf docs.