Import _tkinter after sage complains about libfontconfig library

[miguelmarco]

I am experiencing the following error:

┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.3, Release Date: 2018-08-03                     │
│ Type "notebook()" for the browser-based notebook interface.        │
│ Type "help()" for help.                                            │
└────────────────────────────────────────────────────────────────────┘
sage: plot(sin)
/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py:592: RichReprWarning: Exception in _rich_repr_ while displaying object: /usr/lib64/libfontconfig.so.1: undefined symbol: FT_Done_MM_Var
  RichReprWarning,
Graphics object consisting of 1 graphics primitive

I could trace the culprit to

sage: _.show()
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-e1aa3d6e2a32> in <module>()
----> 1 _.show()

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/misc/decorators.pyc in wrapper(*args, **kwds)
    484             kwds[self.name + "options"] = suboptions
    485 
--> 486             return func(*args, **kwds)
    487 
    488         #Add the options specified by @options to the signature of the wrapped

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/plot/graphics.pyc in show(self, **kwds)
   1984         from sage.repl.rich_output import get_display_manager
   1985         dm = get_display_manager()
-> 1986         dm.display_immediately(self, **kwds)
   1987 
   1988     def xmin(self, xmin=None):

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.pyc in display_immediately(self, obj, **rich_repr_kwds)                                                                                                   
    833             1/2
    834         """
--> 835         plain_text, rich_output = self._rich_output_formatter(obj, rich_repr_kwds)
    836         self._backend.display_immediately(plain_text, rich_output)
    837 

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.pyc in _rich_output_formatter(self, obj, rich_repr_kwds)                                                                                                  
    623         has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
    624         if has_rich_repr:
--> 625             rich_output = self._call_rich_repr(obj, rich_repr_kwds)
    626         if isinstance(rich_output, OutputPlainText):
    627             plain_text = rich_output

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.pyc in _call_rich_repr(self, obj, rich_repr_kwds)                                                                                                         
    581         if rich_repr_kwds:
    582             # do not ignore errors from invalid options
--> 583             return obj._rich_repr_(self, **rich_repr_kwds)
    584         try:
    585             return obj._rich_repr_(self)

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/plot/graphics.pyc in _rich_repr_(self, display_manager, **kwds)
    881             if output_container in display_manager.supported_output():
    882                 return display_manager.graphics_from_save(
--> 883                     self.save, kwds, file_ext, output_container)
    884 
    885     def __str__(self):

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.pyc in graphics_from_save(self, save_function, save_kwds, file_extension, output_container, figsize, dpi)                                                 
    711         if dpi is not None:
    712             kwds['dpi'] = dpi
--> 713         save_function(filename, **kwds)
    714         from sage.repl.rich_output.buffer import OutputBuffer
    715         buf = OutputBuffer.from_file(filename)

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/misc/decorators.pyc in wrapper(*args, **kwds)
    484             kwds[self.name + "options"] = suboptions
    485 
--> 486             return func(*args, **kwds)
    487 
    488         #Add the options specified by @options to the signature of the wrapped

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/plot/graphics.pyc in save(self, filename, **kwds)
   3149             rc_backup = (rcParams['ps.useafm'], rcParams['pdf.use14corefonts'],
   3150                          rcParams['text.usetex']) # save the rcParams
-> 3151             figure = self.matplotlib(**options)
   3152             # You can output in PNG, PS, EPS, PDF, PGF, or SVG format, depending
   3153             # on the file extension.

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/sage/plot/graphics.pyc in matplotlib(self, filename, xmin, xmax, ymin, ymax, figsize, figure, sub, axes, axes_labels, axes_labels_size, fontsize, frame, verify, aspect_ratio, gridlines, gridlinesstyle, vgridlinesstyle, hgridlinesstyle, show_legend, legend_options, axes_pad, ticks_integer, tick_formatter, ticks, title, title_pos, base, scale, stylesheet, typeset)                                                                            
   2526             ticks = (ticks, None)
   2527 
-> 2528         import matplotlib.pyplot as plt
   2529         if stylesheet not in plt.style.available:
   2530             stylesheet = 'classic'

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/matplotlib/pyplot.py in <module>()
    111 ## Global ##
    112 
--> 113 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    114 
    115 _IP_REGISTERED = None

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/matplotlib/backends/__init__.pyc in pylab_setup(name)
     58     # imports. 0 means only perform absolute imports.
     59     backend_mod = __import__(backend_name, globals(), locals(),
---> 60                              [backend_name], 0)
     61 
     62     # Things we pull in from all backends

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/matplotlib/backends/backend_tkagg.py in <module>()
      4 
      5 import six
----> 6 from six.moves import tkinter as Tk
      7 from six.moves import tkinter_filedialog as FileDialog
      8 

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/six.pyc in load_module(self, fullname)
    201         mod = self.__get_module(fullname)
    202         if isinstance(mod, MovedModule):
--> 203             mod = mod._resolve()
    204         else:
    205             mod.__loader__ = self

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/six.pyc in _resolve(self)
    113 
    114     def _resolve(self):
--> 115         return _import_module(self.mod)
    116 
    117     def __getattr__(self, attr):

/home/mmarco/sage-8.3/local/lib/python2.7/site-packages/six.pyc in _import_module(name)
     80 def _import_module(name):
     81     """Import module, returning the module after the last dot."""
---> 82     __import__(name)
     83     return sys.modules[name]
     84 

/home/mmarco/sage-8.3/local/lib/python2.7/lib-tk/Tkinter.py in <module>()
     37     # Attempt to configure Tcl/Tk without requiring PATH
     38     import FixTk
---> 39 import _tkinter # If this fails your Python may not be configured for Tk
     40 tkinter = _tkinter # b/w compat for export
     41 TclError = _tkinter.TclError

ImportError: /usr/lib64/libfontconfig.so.1: undefined symbol: FT_Done_MM_Var

Indeed, the same kind of error appears just trying to import _tkinter.

However, in a clean sage -python session, importing _tkinter does not give any problem. Furthermore, if in a clean python session I run

from sage.all import *
import _tkinter

the error happens, but it doesn't happen if I change the order of imports (that is, importing _tkinter before the sage library is ok.

I could circumvent the problem by adding a line

import _tkinter

at the file sage/all.py before the line

from sage.matrix.all import *

So, for some reason, it seems like some code in the matrix module messes in a subtle way with something that tkinter expects.

CC: @jdemeyer @NathanDunfield @williamstein @fchapoton @saraedum @embray @slel

Component: graphics

Author: Miguel Marco

Issue created by migration from https://trac.sagemath.org/ticket/26231

[embray]

comment:1

Normally Sage does not even include Tkinter. It is not included by default when building Python, normally, and IIRC you have to explicitly enable it when building Python which we don't for Sage.

If you're doing some import Tkinter then you must be pulling it in from some other Python, which is not supported behavior. This is especially problematic if your _tkinter was linked against system libraries (in this case, indirectly, libfreetype most likely) that conflict with versions included in Sage.

I would propose "wontfix" for this since Sage doesn't include Tkinter (perhaps it should be that's another question...)

[embray]

comment:3

Sorry, I didn't fully read the first part of your message. I thought for some reason you were explicitly trying to use Tkinter directly. Regardless, the solution is still basically the same: It shouldn't be importing _tkinter at all unless you have some bad PYTHONPATH or something.

Also maybe a ~/.config/matplotlib setting the TkAgg backend by default? I feel like Sage shouldn't be loading the default matplotlib config as it could conflict with other matplotlibs on the system.

[miguelmarco]

comment:4

Oddly enough, the bug appears without me manually importing tkinter. Just trying to plot a graph, on a clean sage install (compiled from the source tarball).

Maybe at some point matplotlib (or some other component) tries to use tkinter to show plots?

[embray]

comment:5

(In fact, sage-env does set export MPLCONFIGDIR="$DOT_SAGE/matplotlib-1.5.1". Probably the sage.env module should do the same if MPLCONFIGDIR is not otherwise set.)

[miguelmarco]

comment:6

I have removed the .local/matplotlib directory (it was empty anyways), and checked that I didn't have the PYTHONPATH variable set. The problem persists.

I also have the same feeling: for some reason Sage is trying to use tkinter to show the plots, but can't find exactly what is triggering that behaviour.

In any case, I think there are two orthogonal problems:

1. Sage trying to use tkinter, when it shouldn't
2. Importing the sage library breaks the ability to import other python libraries.

[embray]

comment:7

The latter isn't a real problem; it's expected. The former is strange but I think it's something odd on your system configuration. Starting with the fact that you can apparently import an _tkinter module at all. If you do:

sage -python -c 'import _tkinter; print(_tkinter)'

what does it show?

[miguelmarco]

comment:8

I get this:

mmarco@localhost ~/sage-8.3 $ ./sage -python -c 'import _tkinter; print(_tkinter)'
<module '_tkinter' from '/home/mmarco/sage-8.3/local/lib/python2.7/lib-dynload/_tkinter.so'>

[embray]

comment:9

Perhaps I'm wrong then. It looks like maybe Python will build the _tkinter module automatically if it happens to find working !Tcl/Tk libs somewhere.

Maybe I would err towards not doing that, since it can result in inconsistent builds of Sage (some that have Tkinter, some that don't). Unless we explicitly support it we should disable it. Also matplotlib should be configured by Sage to use the 'agg' backend for outputting plots.

[miguelmarco]

comment:10

That makes sense to me.

[NathanDunfield]

comment:11

The tkinter module is part of the standard Python library and has been since the 1990s. Thus Python builds _tkinter if it can find the Tcl/Tk library and headers, just as it does the bz2 module or any of its other standard modules that depend on some optional library or another. I would argue very strongly against disabling building tkinter for Sage. Admittedly, we use it heavily in SnapPy so I am no doubt biased, but except for servers that run headless tkinter is usually included with Python or trivial to install.

[NathanDunfield]

comment:12

Some thoughts on the initial report. What happens if you do

sage: %matplotlib

to list the current backend? Historically, I would have expected the response to be Using matplotlib backend: agg, independent of whether Tkinter is working, but I suspect you're going to get tk or tkagg.

I see that Sage recently upgraded to matplotlib 2.1.0 from the 1.* series. One thing they changed in 2.* is that the TkAgg backend is always built, even when Tk is not available or simply not working. This is because they figured out how to build the TkAgg backend without having the Tk library installed at build time. See can have the effect of making TkAgg the default backend, see https://github.com/matplotlib/matplotlib/pull/7530 Now, they claim to have fixed that particular instance, but likely this is some variant.

One solution might be to configure Sage so that it sets the default backend back to Agg. I'm not expert enough to know the best place to do this, but assuming the Sage interpreter is importing matplotlib at some stage, then you could probably just do:

matplotlib.use('Agg')

right after that.

[slel]

comment:13

It is general that when building Python, which Python modules get built depends on ambient functionality:

• the tkinter module given ambient !Tcl/Tk functionality, as Nathan pointed out,
• the ssl module given ambient SSL functionality --- crucial to let us "pip install" from PyPI!

!Tcl/Tk capability, if present, adds some extra functionality in Sage.

See the references to !Tcl/Tk in the Sage documentation:

• https://doc.sagemath.org/html/en/installation/source.html#tcl-tk
• https://doc.sagemath.org/html/en/faq/faq-usage.html#how-to-get-sage-s-python-to-recognize-my-system-s-tcl-tk-install
• https://doc.sagemath.org/html/en/constructions/plotting.html#openmath

It can be more important for some optional or external packages, such as SnapPy.

[miguelmarco]

comment:14

Replying to @NathanDunfield:

Some thoughts on the initial report. What happens if you do

sage: %matplotlib

to list the current backend? Historically, I would have expected the response to be Using matplotlib backend: agg, independent of whether Tkinter is working, but I suspect you're going to get tk or tkagg.

sage: %matplotlib
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-040f64586c53> in <module>()
----> 1 get_ipython().magic(u'matplotlib')

/home/mmarco/sage/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s)
   2158         magic_name, _, magic_arg_s = arg_s.partition(' ')
   2159         magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2160         return self.run_line_magic(magic_name, magic_arg_s)
   2161 
   2162     #-------------------------------------------------------------------------

/home/mmarco/sage/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_line_magic(self, magic_name, line)                                                                                                                         
   2079                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2080             with self.builtin_trap:
-> 2081                 result = fn(*args,**kwargs)
   2082             return result
   2083 

<decorator-gen-105> in matplotlib(self, line)

/home/mmarco/sage/local/lib/python2.7/site-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/home/mmarco/sage/local/lib/python2.7/site-packages/IPython/core/magics/pylab.pyc in matplotlib(self, line)
     98             print("Available matplotlib backends: %s" % backends_list)
     99         else:
--> 100             gui, backend = self.shell.enable_matplotlib(args.gui)
    101             self._show_matplotlib_backend(args.gui, backend)
    102 

/home/mmarco/sage/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in enable_matplotlib(self, gui)
   2948                 gui, backend = pt.find_gui_and_backend(self.pylab_gui_select)
   2949 
-> 2950         pt.activate_matplotlib(backend)
   2951         pt.configure_inline_support(self, backend)
   2952 

/home/mmarco/sage/local/lib/python2.7/site-packages/IPython/core/pylabtools.pyc in activate_matplotlib(backend)
    306     matplotlib.rcParams['backend'] = backend
    307 
--> 308     import matplotlib.pyplot
    309     matplotlib.pyplot.switch_backend(backend)
    310 

/home/mmarco/sage/local/lib/python2.7/site-packages/matplotlib/pyplot.py in <module>()
    111 ## Global ##
    112 
--> 113 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    114 
    115 _IP_REGISTERED = None

/home/mmarco/sage/local/lib/python2.7/site-packages/matplotlib/backends/__init__.pyc in pylab_setup(name)
     58     # imports. 0 means only perform absolute imports.
     59     backend_mod = __import__(backend_name, globals(), locals(),
---> 60                              [backend_name], 0)
     61 
     62     # Things we pull in from all backends

/home/mmarco/sage/local/lib/python2.7/site-packages/matplotlib/backends/backend_tkagg.py in <module>()
      4 
      5 import six
----> 6 from six.moves import tkinter as Tk
      7 from six.moves import tkinter_filedialog as FileDialog
      8 

/home/mmarco/sage/local/lib/python2.7/site-packages/six.pyc in load_module(self, fullname)
    201         mod = self.__get_module(fullname)
    202         if isinstance(mod, MovedModule):
--> 203             mod = mod._resolve()
    204         else:
    205             mod.__loader__ = self

/home/mmarco/sage/local/lib/python2.7/site-packages/six.pyc in _resolve(self)
    113 
    114     def _resolve(self):
--> 115         return _import_module(self.mod)
    116 
    117     def __getattr__(self, attr):

/home/mmarco/sage/local/lib/python2.7/site-packages/six.pyc in _import_module(name)
     80 def _import_module(name):
     81     """Import module, returning the module after the last dot."""
---> 82     __import__(name)
     83     return sys.modules[name]
     84 

/home/mmarco/sage/local/lib/python2.7/lib-tk/Tkinter.py in <module>()
     37     # Attempt to configure Tcl/Tk without requiring PATH
     38     import FixTk
---> 39 import _tkinter # If this fails your Python may not be configured for Tk
     40 tkinter = _tkinter # b/w compat for export
     41 TclError = _tkinter.TclError

ImportError: /usr/lib64/libfontconfig.so.1: undefined symbol: FT_Done_MM_Var

[embray]

comment:15

Replying to @NathanDunfield:

but except for servers that run headless tkinter is usually included with Python or trivial to install

That's a big "BUT". Most of our CI infrastructure and servers like SageCell and its variants are effectively "headless". If Tkinter is to be supported by Sage that means Sage has to include its own Tcl/Tk as well since balancing the system versions of those libraries with Sage's other bundled dependencies would be very tricky in many cases (not that I think that situation is a good thing, but it it is what it is).

I would argue that Tkinter should not be supported by sage-the-distribution until and unless we have our house in better order w.r.t. other packaging issues. That, or you'll have to package Tcl/Tk for Sage, which is perhaps not entirely unreasonable, but someone else will have to do that. It should also be optional of course since it would not be useful in most server cases.

[embray]

comment:16

Replying to @NathanDunfield:

One solution might be to configure Sage so that it sets the default backend back to Agg. I'm not expert enough to know the best place to do this, but assuming the Sage interpreter is importing matplotlib at some stage, then you could probably just do:

matplotlib.use('Agg')

right after that.

I generally agree that this should be the case. We already do exactly this when running the doctest suite. However, forcibly doing this is still problematic since it could override possibly intentional user settings. If there's a way to force agg to be the default backend while possibly still respecting a custom mplconfig that would be ideal probably.

[embray]

comment:17

Does anyone know why Sage packages libfreetype in the first place? Because that's probably the source of the problem here.

[NathanDunfield]

comment:18

Replying to @embray:

Does anyone know why Sage packages libfreetype in the first place?

Unfortunately, I believe it is a requirement of matplotlib itself. Specifically, a dependency of matplotlib/ft2font.so. It is also used by pillow/PIL.

[NathanDunfield]

comment:19

Replying to @embray:

If there's a way to force agg to be the default backend while possibly still respecting a custom mplconfig that would be ideal probably.

Yes, that seems like the best solution. I've never had any difficulty overriding the defaults when using matplotlib in Sage, and I do this fairly often. Perhaps we should just try one or two ways of making agg the default and see if any of them work... ;-)

[embray]

comment:20

Replying to @NathanDunfield:

Replying to @embray:

Does anyone know why Sage packages libfreetype in the first place?

Unfortunately, I believe it is a requirement of matplotlib itself. Specifically, a dependency of matplotlib/ft2font.so. It is also used by pillow/PIL.

Probably another good candidate for defaulting to the system lib where possible, rather than using the Sage version.

[slel]

comment:21

Replying to @embray:

Replying to @NathanDunfield:

Replying to @embray:

Does anyone know why Sage packages libfreetype in the first place?

Unfortunately, I believe it is a requirement of matplotlib itself. Specifically, a dependency of matplotlib/ft2font.so. It is also used by pillow/PIL.

Probably another good candidate for defaulting to the system lib where possible, rather than using the Sage version.

• spkg-configure.m4 for freetype: done in #27168, #30014
• use Python packages from the system: #29023