doctesting sage/graphs/graph_generators.py fails in prefix but not in gentoo

[strogdon]

See https://github.com/cschwan/sage-on-gentoo/issues/482 for the initial report of the failure. The failure:

sage -t --long usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py
**********************************************************************
File "usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py", line 566, in sage.graphs.graph_generators.GraphGenerators
Failed example:
    graphs_list.show_graphs(L) # long time
Expected nothing
Got:
    doctest:warning
      File "/storage/strogdon/gentoo-rap/usr/lib/python-exec/python2.7/sage-runtests", line 98, in <module>
        err = DC.run()
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/control.py", line 1099, in run
        self.run_doctests()
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/control.py", line 824, in run_doctests
        self.dispatcher.dispatch()
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 1777, in dispatch
        self.parallel_dispatch()
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 1667, in parallel_dispatch
        w.start()  # This might take some time
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 1943, in start
        super(DocTestWorker, self).start()
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start
        self._popen = Popen(self)
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/multiprocessing/forking.py", line 126, in __init__
        code = process_obj._bootstrap()
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 1916, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 2218, in __call__
        result = runner.run(test)
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 663, in run
        return self._run(test, compileflags, out)
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 518, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 888, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.graphs.graph_generators.GraphGenerators[7]>", line 1, in <module>
        graphs_list.show_graphs(L) # long time
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/graphs/graph_list.py", line 366, in show_graphs
        to_graphics_array(graph_slice, **kwds).show()
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/plot/graphics.py", line 3725, in show
        dm.display_immediately(self, **kwds)
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 831, in display_immediately
        plain_text, rich_output = self._rich_output_formatter(obj, rich_repr_kwds)
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 623, in _rich_output_formatter
        rich_output = self._call_rich_repr(obj, rich_repr_kwds)
      File "/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 590, in _call_rich_repr
        RichReprWarning,
    :
    RichReprWarning: Exception in _rich_repr_ while displaying object: No module named Tkinter
    Graphics Array of size 4 x 4
**********************************************************************
File "usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py", line 575, in sage.graphs.graph_generators.GraphGenerators
Failed example:
    graphs_list.show_graphs(L)              # long time
Expected nothing
Got:
    Graphics Array of size 6 x 4
    Graphics Array of size 3 x 4
**********************************************************************
1 item had failures:
   2 of  33 in sage.graphs.graph_generators.GraphGenerators
    [88 tests, 2 failures, 17.93 s]
----------------------------------------------------------------------
sage -t --long usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py  # 2 doctests failed
----------------------------------------------------------------------

[strogdon]

The following works as expected on both Gentoo and vanilla Sage:

sage: L = list(graphs(5, lambda G: G.size() <= 4))
sage: graphs_list.show_graphs(L)
Launched png viewer for Graphics Array of size 4 x 4
sage: graphs_list.to_graphics_array(L).show()
Launched png viewer for Graphics Array of size 4 x 4
sage: show(graphs_list.to_graphics_array(L))
Launched png viewer for Graphics Array of size 4 x 4

The png files are generated and appear in a viewer.

[strogdon]

The above commands in Prefix:

sage: L = list(graphs(5, lambda G: G.size() <= 4))
sage: graphs_list.show_graphs(L)
/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/display_manager.py:590: RichReprWarning: Exception in _rich_repr_ while displaying object: No module named Tkinter
  RichReprWarning,
Graphics Array of size 4 x 4
sage: graphs_list.to_graphics_array(L).show()
Graphics Array of size 4 x 4
sage: show(graphs_list.to_graphics_array(L))
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-4-0ccb3bbd1c35> in <module>()
----> 1 show(graphs_list.to_graphics_array(L))

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/pretty_print.pyc in show(*args, **kwds)
    256         args[0].show()
    257         return
--> 258     pretty_print(*args, **kwds)

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/pretty_print.pyc in pretty_print(*args, **kwds)
    227             pass
    228         elif len(args) == 1:
--> 229             dm.display_immediately(*args, **kwds)
    230         else:
    231             SequencePrettyPrinter(*args, **kwds).pretty_print()

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/display_manager.pyc in display_immediately(self, obj, **rich_repr_kwds)
    829             1/2
    830         """
--> 831         plain_text, rich_output = self._rich_output_formatter(obj, rich_repr_kwds)
    832         self._backend.display_immediately(plain_text, rich_output)
    833 

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/display_manager.pyc in _rich_output_formatter(self, obj, rich_repr_kwds)
    631         if rich_output is None:
    632             rich_output = self._preferred_text_formatter(
--> 633                 obj, plain_text=plain_text, **rich_repr_kwds)
    634         # promote output container types to backend-specific containers
    635         plain_text = self._promote_output(plain_text)

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/display_manager.pyc in _preferred_text_formatter(self, obj, plain_text, **kwds)
    533             return out
    534         if want == 'latex' and OutputLatex in supported:
--> 535             out = self._backend.latex_formatter(obj, **kwds)
    536             if type(out) is not OutputLatex:
    537                 raise OutputTypeException('backend returned wrong output type, require Latex')

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/repl/rich_output/backend_base.pyc in latex_formatter(self, obj, **kwds)
    482             mathjax = MathJax().eval(obj, mode='plain', combine_all=True)
    483         else:
--> 484             mathjax = MathJax().eval(obj, mode='plain', combine_all=False)
    485         from sage.repl.rich_output.output_basic import OutputLatex
    486         return OutputLatex(str(mathjax))

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/misc/latex.pyc in eval(self, x, globals, locals, mode, combine_all)
   1950         """
   1951         # Get a regular LaTeX representation of x
-> 1952         x = latex(x, combine_all=combine_all)
   1953 
   1954         # The following block, hopefully, can be removed in some future MathJax.

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/misc/latex.pyc in __call__(self, x, combine_all)
    920         """
    921         if has_latex_attr(x):
--> 922             return LatexExpr(x._latex_())
    923         try:
    924             f = latex_table[type(x)]

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/plot/graphics.pyc in _latex_(self, dpi, figsize, axes, **args)
   3678         """
   3679         tmpfilename = tmp_filename(ext='.pgf')
-> 3680         self.save(filename=tmpfilename, **args)
   3681         with open(tmpfilename, "r") as tmpfile:
   3682                 latex_list = tmpfile.readlines()

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/plot/graphics.pyc in save(self, filename, dpi, figsize, axes, **kwds)
   3632             subplot = figure.add_subplot(rows, cols, i)
   3633             g.matplotlib(filename, figure=figure, sub=subplot,
-> 3634                          verify=do_verify, axes = axes, **kwds)
   3635         g.save(filename, dpi=dpi, figure=figure, sub=subplot,
   3636                verify=do_verify, axes=axes, **kwds)

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/plot/graphics.pyc in matplotlib(self, filename, xmin, xmax, ymin, ymax, figsize, figure, sub, axes, axes_labels, axes_labels_size, fontsize, frame, verify, aspect_ratio, gridlines, gridlinesstyle, vgridlinesstyle, hgridlinesstyle, show_legend, legend_options, axes_pad, ticks_integer, tick_formatter, ticks, title, title_pos, base, scale, typeset)
   2620                     pass
   2621             g.set_options(opts)
-> 2622             g._render_on_subplot(subplot)
   2623             if hasattr(g, '_bbox_extra_artists'):
   2624                 self._bbox_extra_artists.extend(g._bbox_extra_artists)

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/sage/plot/scatter_plot.pyc in _render_on_subplot(self, subplot)
    129             Graphics object consisting of 1 graphics primitive
    130         """
--> 131         from matplotlib.pyplot import scatter
    132         options = self.options()
    133         p = subplot.scatter(self.xdata, self.ydata, alpha=options['alpha'],

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/pyplot.py in <module>()
    112 
    113 from matplotlib.backends import pylab_setup
--> 114 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    115 
    116 _IP_REGISTERED = None

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/backends/__init__.pyc in pylab_setup()
     30     # imports. 0 means only perform absolute imports.
     31     backend_mod = __import__(backend_name,
---> 32                              globals(),locals(),[backend_name],0)
     33 
     34     # Things we pull in from all backends

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_tkagg.py in <module>()
      4 
      5 from matplotlib.externals import six
----> 6 from matplotlib.externals.six.moves import tkinter as Tk
      7 from matplotlib.externals.six.moves import tkinter_filedialog as FileDialog
      8 

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/externals/six.pyc in load_module(self, fullname)
    197         mod = self.__get_module(fullname)
    198         if isinstance(mod, MovedModule):
--> 199             mod = mod._resolve()
    200         else:
    201             mod.__loader__ = self

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/externals/six.pyc in _resolve(self)
    111 
    112     def _resolve(self):
--> 113         return _import_module(self.mod)
    114 
    115     def __getattr__(self, attr):

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/externals/six.pyc in _import_module(name)
     78 def _import_module(name):
     79     """Import module, returning the module after the last dot."""
---> 80     __import__(name)
     81     return sys.modules[name]
     82 

ImportError: No module named Tkinter

In the first two

sage: graphs_list.show_graphs(L)
sage: graphs_list.to_graphics_array(L).show()

png files are opened but they are empty. In the third

sage: show(graphs_list.to_graphics_array(L))

a png and a pgf file are opened but they also are empty. My suspicion is that there is a path issue somewhere?

[kiwifb]

Ok, so site-packages/tkinter exists? Which version of six is installed?

[strogdon]

dev-python/six-1.10.0 which I believe is the only one available.

[kiwifb]

Is this matplotlib 2.0.2?

[strogdon]

No it's 1.5.3-r2.

[strogdon]

I'm not sure what vanilla sage uses to generate png files. That component could be missing on my Prefix.

[strogdon]

I can replicate this error on Gentoo by removing matplotlib and then testing sage -t --long /usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py.

[strogdon]

I believe this

      5 from matplotlib.externals import six
----> 6 from matplotlib.externals.six.moves import tkinter as Tk

will always fail with

ImportError: No module named Tkinter

whenever Sage is built when there are no tk libraries installed. I do not have tk libraries installed on my Prefix. But I have a Gentoo Sage build where dev-lang/tk is not installed and the graph_generator.py test passes. I wonder why in Prefix is Sage trying to import tkinter?

[strogdon]

It's probably matplotlib that needs to be linked against tk for there to be no ImportError I don't know about Sage.

[kiwifb]

Stab in the dark, do you have tk or tcl installed on the host of the prefix?

[strogdon]

OK, I believe this should not fail at all

sage: import matplotlib.pyplot
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-6f467123fe04> in <module>()
----> 1 import matplotlib.pyplot

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/pyplot.py in <module>()
    112 
    113 from matplotlib.backends import pylab_setup
--> 114 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    115 
    116 _IP_REGISTERED = None

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/backends/__init__.pyc in pylab_setup()
     30     # imports. 0 means only perform absolute imports.
     31     backend_mod = __import__(backend_name,
---> 32                              globals(),locals(),[backend_name],0)
     33 
     34     # Things we pull in from all backends

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_tkagg.py in <module>()
      4 
      5 from matplotlib.externals import six
----> 6 from matplotlib.externals.six.moves import tkinter as Tk
      7 from matplotlib.externals.six.moves import tkinter_filedialog as FileDialog
      8 

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/externals/six.pyc in load_module(self, fullname)
    197         mod = self.__get_module(fullname)
    198         if isinstance(mod, MovedModule):
--> 199             mod = mod._resolve()
    200         else:
    201             mod.__loader__ = self

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/externals/six.pyc in _resolve(self)
    111 
    112     def _resolve(self):
--> 113         return _import_module(self.mod)
    114 
    115     def __getattr__(self, attr):

/storage/strogdon/gentoo-rap/usr/lib64/python2.7/site-packages/matplotlib/externals/six.pyc in _import_module(name)
     78 def _import_module(name):
     79     """Import module, returning the module after the last dot."""
---> 80     __import__(name)
     81     return sys.modules[name]
     82 

ImportError: No module named Tkinter

[strogdon]

Yes tk/tcl is installed on the host.

[kiwifb]

I think matplotlib may be able to see the host install and get misconfigured. I'll try to have a closer look at that scenario.

[strogdon]

On gentoo:

sage: from matplotlib.backends import pylab_setup
sage: pylab_setup()
(<module 'matplotlib.backends.backend_qt4agg' from '/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_qt4agg.pyc'>,
 <function new_figure_manager at 0x7fd3825d9140>,
 <function draw_if_interactive at 0x7fd387133b18>,
 <matplotlib.backends.backend_qt5.Show object at 0x7fd3825d3710>)

so it is using the qt5 backend. But in Prefix it is trying to use the tkagg backend.

[strogdon]

This fixed it

strogdon@blitzen ~ $ cat ~/matplotlibrc 
backend : agg

See https://github.com/matplotlib/matplotlib/issues/7115/. So there must be some rationale for setting the backend?

[kiwifb]

Yes of course those problems seem to always be fixable that way. I think we need to have that documented somewhere.

Ultimately I think the root cause is interference between prefix and host. I need to get a prefix somewhere again to sort that out.

[strogdon]

I rebuilt matplotlib-1.5.3-r2 on gentoo with no useflags and the default backend is tkagg. I don't have tk/tcl installed. So this might be something new with matplotlib.

[strogdon]

And if I have done things correctly the default backend for matplotlib-2.0.2 is agg. So perhaps there is a bug in version 1.5.3?

[strogdon]

See https://github.com/matplotlib/matplotlib/pull/7530 for some history.

[kiwifb]

After checking the runtime check is not in 1.5.3 so it is never checking for tkinter. It is still weird that things work out of the box in Gentoo without tk. The default matplotlibrc is also listing tkagg as the default. So it must be able to fall back more reliably on pure Gentoo than prefix.

Is there a /etc/matplotlibrc in the host or inside the prefix?

[strogdon]

No there is no /etc/matplotlibrc file and I have no such file in my prefix home folder either. If I rebuild matplotlib-1.5.3-r2 on my Gentoo laptop without any useflags the subject test fails since then the default backend is tkagg. Now this Gentoo laptop does not have tk installed.

[kiwifb]

There must be more

fbissey@moonloop ~ $ eix dev-lang/tk
* dev-lang/tk
     Available versions:  
     (0)    8.5.17(0/8.5) ~8.6.5(0/8.6) ~8.6.5-r1(0/8.6) 8.6.6(0/8.6) ~8.6.7(0/8.6)
     (8.5)  ~8.5.17-r100
       {aqua debug test (+)threads truetype xscreensaver ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Homepage:            http://www.tcl.tk/
     Description:         Tk Widget Set

fbissey@moonloop ~ $ eix dev-lang/tcl
* dev-lang/tcl
     Available versions:  
     (0)    8.5.17(0/8.5) ~8.6.5(0/8.6) 8.6.6(0/8.6) ~8.6.7(0/8.6)
     (8.5)  ~8.5.17-r100
       {debug (+)threads ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Homepage:            http://www.tcl.tk/
     Description:         Tool Command Language

fbissey@moonloop ~ $ sage -t --long /usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py
too many failed tests, not using stored timings
Running doctests with ID 2017-09-05-17-25-27-87f62151.
Using --optional=optional,sage
Doctesting 1 file.
sage -t --long /usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py
    [88 tests, 9.64 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 9.7 seconds
    cpu time: 9.5 seconds
    cumulative wall time: 9.6 seconds
fbissey@moonloop ~ $ eix -I matplotlib
[U] dev-python/matplotlib
     Available versions:  1.4.3 (~)1.5.3-r1 (~)1.5.3-r2 (~)2.0.2 **9999 {cairo doc examples excel fltk gtk gtk2 gtk3 latex pyside qt4 qt5 test tk wxwidgets PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6"}
     Installed versions:  1.5.3-r2(21:17:22 25/08/17)(cairo -doc -examples -excel -fltk -gtk2 -gtk3 -latex -pyside -qt4 -qt5 -test -tk -wxwidgets PYTHON_TARGETS="python2_7 python3_4 python3_6 -python3_5")
     Homepage:            http://matplotlib.org/
     Description:         Pure python plotting library with matlab like syntax

Do you have any graphical backend enabled?

[strogdon]

No backend

eix -I matplotlib
[U] dev-python/matplotlib
     Available versions:  1.4.3 (~)1.5.3-r1 (~)1.5.3-r2 (~)2.0.2 **9999 {cairo doc examples excel fltk gtk gtk2 gtk3 latex pyside qt4 qt5 test tk wxwidgets PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6"}
     Installed versions:  1.5.3-r2(12:06:31 AM 09/05/2017)(-cairo -doc -examples -excel -fltk -gtk2 -gtk3 -latex -pyside -qt4 -qt5 -test -tk -wxwidgets PYTHON_TARGETS="python2_7 python3_4 -python3_5 -python3_6")
     Homepage:            http://matplotlib.org/
     Description:         Pure python plotting library with matlab like syntax

I will check later today to see what happens with cairo enabled.

[kiwifb]

Hum... I do have cairo. May be that makes a difference.

[strogdon]

No cairo didn't make any difference here. Just so we're on the same page:

steven@hp-envy ~ $ eix -I matplotlib
[U] dev-python/matplotlib
     Available versions:  1.4.3 (~)1.5.3-r1 (~)1.5.3-r2 (~)2.0.2 **9999 {cairo doc examples excel fltk gtk gtk2 gtk3 latex pyside qt4 qt5 test tk wxwidgets PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6"}
     Installed versions:  1.5.3-r2(07:30:28 AM 09/05/2017)(cairo -doc -examples -excel -fltk -gtk2 -gtk3 -latex -pyside -qt4 -qt5 -test -tk -wxwidgets PYTHON_TARGETS="python2_7 python3_4 -python3_5 -python3_6")
     Homepage:            http://matplotlib.org/
     Description:         Pure python plotting library with matlab like syntax

steven@hp-envy ~ $ eix dev-lang/tk
* dev-lang/tk
     Available versions:  
     (0)    8.5.17(0/8.5) ~8.6.5(0/8.6) ~8.6.5-r1(0/8.6) 8.6.6(0/8.6) ~8.6.7(0/8.6)
     (8.5)  ~8.5.17-r100
       {aqua debug test (+)threads truetype xscreensaver ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Homepage:            http://www.tcl.tk/
     Description:         Tk Widget Set

steven@hp-envy ~ $ eix dev-lang/tcl
* dev-lang/tcl
     Available versions:  
     (0)    8.5.17(0/8.5) ~8.6.5(0/8.6) 8.6.6(0/8.6) ~8.6.7(0/8.6)
     (8.5)  ~8.5.17-r100
       {debug (+)threads ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Homepage:            http://www.tcl.tk/
     Description:         Tool Command Language

steven@hp-envy ~ $ sage -t --long /usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py
Running doctests with ID 2017-09-05-11-45-53-4b5e7201.
Using --optional=optional,sage
Doctesting 1 file.
sage -t --long --warn-long 123.8 /usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py
**********************************************************************
File "/usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py", line 566, in sage.graphs.graph_generators.GraphGenerators
Failed example:
    graphs_list.show_graphs(L) # long time
Expected nothing
Got:
    doctest:warning
      File "/usr/lib/python-exec/python2.7/sage-runtests", line 98, in <module>
        err = DC.run()
      File "/usr/lib64/python2.7/site-packages/sage/doctest/control.py", line 1099, in run
        self.run_doctests()
      File "/usr/lib64/python2.7/site-packages/sage/doctest/control.py", line 824, in run_doctests
        self.dispatcher.dispatch()
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 1777, in dispatch
        self.parallel_dispatch()
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 1667, in parallel_dispatch
        w.start()  # This might take some time
      File "/usr/lib64/python2.7/site-packages/sage/doctest/forker.py", line 1943, in start
        super(DocTestWorker, self).start()
      File "/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start
        self._popen = Popen(self)
      File "/usr/lib64/python2.7/multiprocessing/forking.py", line 126, in __init__
        code = process_obj._bootstrap()
      File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
        self.run()

...

So clearly something else is up. Strange.

[strogdon]

Forcing the install of matplotlib-2.0.2 which should have https://github.com/matplotlib/matplotlib/pull/7530

steven@hp-envy ~ $ eix -I matplotlib
[I] dev-python/matplotlib
     Available versions:  1.4.3 (~)1.5.3-r1 (~)1.5.3-r2 (~)2.0.2 **9999 {cairo doc examples excel fltk gtk gtk2 gtk3 latex pyside qt4 qt5 test tk wxwidgets PYTHON_TARGETS="python2_7 python3_4 python3_5 python3_6"}
     Installed versions:  2.0.2(12:01:38 PM 09/05/2017)(cairo -doc -examples -excel -gtk2 -gtk3 -latex -pyside -qt4 -qt5 -test -tk -wxwidgets PYTHON_TARGETS="python2_7 python3_4 -python3_5 -python3_6")
     Homepage:            http://matplotlib.org/
     Description:         Pure python plotting library with matlab like syntax

I get

steven@hp-envy ~ $ sage -t --long /usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py
Running doctests with ID 2017-09-05-12-02-07-1b11fe02.
Using --optional=optional,sage
Doctesting 1 file.
sage -t --long --warn-long 123.8 /usr/lib64/python2.7/site-packages/sage/graphs/graph_generators.py
    [88 tests, 10.22 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 10.3 seconds
    cpu time: 10.1 seconds
    cumulative wall time: 10.2 seconds

[strogdon]

I now have 50+ failures in 8.1.beta4 associated with the tkagg backend.

sage: import matplotlib
sage: from matplotlib.backends import _tkagg
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-874272889e03> in <module>()
----> 1 from matplotlib.backends import _tkagg

ImportError: No module named Tkinter

The inability to import Tkinter is the real reason for the failures. This module is provided by python with USE=tk. This useflag of course pulls in tk/tcl. So, I don't see how the graph_generator.py doctest will ever pass with the tkagg backend unless python is built with the tk useflag. Is it possible that there is a Tkinter module under /usr/lib64/python2.7/lib-tk on your system?

[kiwifb]

No, I definitely don't. There should be a bit from six that says that Tkinter should be changed to tkinter which in turn should be installed by future.

[strogdon]

The way I dealt with this tkinter issue on my machines was to apply the patch https://github.com/matplotlib/matplotlib/commit/7322046282ce40446b5970dfe2b7e0ac820bbec2 on top of matplotlib-1.5.3-r2. As near as I can determine the patch prevents setting as default the tkagg backend when runtime dependencies are not met.

[kiwifb]

Dependencies that are very nebulous since we don't seem to be able to figure out what they are.

[kiwifb]

Interestingly debian does some patching that I believe is related https://anonscm.debian.org/cgit/debian-science/packages/sagemath.git/tree/debian/patches/u1-fix-dont-require-DISPLAY.patch

[strogdon]

Hmm. Things certainly go bonkers if I unset DISPLAY on Gentoo and doctest sage/plot/plot.py and sage/probability/probability_distribution.pyx. However, in Prefix the doctests pass when the DISPLAY is unset?

[kiwifb]

You need a gui interface set for DISPLAY to be an issue actually.

[strogdon]

That's what I thought but I can remotely log onto a Gentoo box, unset the DISPLAY and the tests fail.

[kiwifb]

I do most of my testing remotely, and DISPLAY is unset. The only useflag I set for matplotlib is cairo.

[strogdon]

Perhaps that's it. On my Gentoo, in addition to cairo, I have the qt4 useflag but no useflags in Prefix.

[strogdon]

I'm not sure this was ever resolved, but with MPL-2.1.0-r2 I no longer see this issue. I haven't changed anything except the version of matplotlib. I suspect that once MPL-2.1.0 is the default matplotlib for supported sage-on-gentoo Sage versions it won't be an issue for anyone.

[kiwifb]

Indeed. The stuff for the doctesting framework I pushed for MPL-2.1.0 upstream will certainly fix this kind of issues.