matplotlib failing to import when running from Unicode path

hraesvelgr commented 7 years ago

downloaded https://github.com/pyfa-org/Pyfa/releases/tag/v1.28.1

but graph is still greyed out. tried to change the "gui" file in "library zip", but 7zip refused this with an error.

so no graphs for me?

@blitzmann says:

WORKAROUND: This is a known issue for the graphing library that we ship with (matplotlib v1.4.3). It's been fixed in a higher version, however we haven't been able to update this library just yet. In the meantime, please ensure that pyfa runs from a non-Unicode directory (ASCII only)

Ebag333 commented 7 years ago

You can either use an earlier version, or wait until the next build. There should be one soon (tm).

What OS are you using?

hraesvelgr commented 7 years ago

win 7, 64 bit.

blitzmann commented 7 years ago

tried to change the "gui" file in "library zip"

I'm not sure what this means... gui is a directory that contains our GUI definitions. :P

Can you take a screenshot of Help > About as well as the graph menu being disabled? Additionally, can you open the library.zip/gui/graphFrame.py and post the contents (feel free to extract library.zip to your desktop or something)? I'm grasping at straws here, there really shouldn't be a situation in which graphs are disabled for the 1.28.1 windows release - everything is bundled correctly and should for all intents and purposes work for everyone.

Also check %userprofile%/.pyfa/ for log files (Pyfa*.log) and post them here

hraesvelgr commented 7 years ago

@blitzmann i tried this solution: https://github.com/pyfa-org/Pyfa/issues/1001#issuecomment-280865070

logfile: https://gist.github.com/hraesvelgr/1566c1e4a74d1dd51afc202512b4695c

grafik

content of the graphframe-data:

# =============================================================================
# Copyright (C) 2010 Diego Duclos
#
# This file is part of pyfa.
#
# pyfa is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyfa is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyfa.  If not, see <http://www.gnu.org/licenses/>.
# =============================================================================

import os
from logbook import Logger

# noinspection PyPackageRequirements
import wx

from service.fit import Fit
import gui.display
import gui.mainFrame
import gui.globalEvents as GE
from gui.graph import Graph
from gui.bitmapLoader import BitmapLoader
import traceback

pyfalog = Logger(__name__)

try:
    import matplotlib as mpl

    mpl_version = int(mpl.__version__[0]) or -1
    if mpl_version >= 2:
        mpl.use('wxagg')
        mplImported = True
    else:
        mplImported = False
    from matplotlib.patches import Patch

    from matplotlib.backends.backend_wxagg import FigureCanvasWxAgg as Canvas
    from matplotlib.figure import Figure

    graphFrame_enabled = True
    mplImported = True
except ImportError as e:
    pyfalog.warning("Matplotlib failed to import.  Likely missing or incompatible version.")
    mpl_version = -1
    Patch = mpl = Canvas = Figure = None
    graphFrame_enabled = False
    mplImported = False
except Exception:
    # We can get exceptions deep within matplotlib. Catch those.  See GH #1046
    tb = traceback.format_exc()
    pyfalog.critical("Exception when importing Matplotlib. Continuing without importing.")
    pyfalog.critical(tb)
    mpl_version = -1
    Patch = mpl = Canvas = Figure = None
    graphFrame_enabled = False
    mplImported = False

class GraphFrame(wx.Frame):
    def __init__(self, parent, style=wx.DEFAULT_FRAME_STYLE | wx.NO_FULL_REPAINT_ON_RESIZE | wx.FRAME_FLOAT_ON_PARENT):
        global graphFrame_enabled
        global mplImported
        global mpl_version

        self.legendFix = False

        if not graphFrame_enabled:
            pyfalog.warning("Matplotlib is not enabled. Skipping initialization.")
            return

        try:
            cache_dir = mpl._get_cachedir()
        except:
            cache_dir = os.path.expanduser(os.path.join("~", ".matplotlib"))

        cache_file = os.path.join(cache_dir, 'fontList.cache')

        if os.access(cache_dir, os.W_OK | os.X_OK) and os.path.isfile(cache_file):
            # remove matplotlib font cache, see #234
            os.remove(cache_file)
        if not mplImported:
            mpl.use('wxagg')

        graphFrame_enabled = True
        if int(mpl.__version__[0]) < 1:
            print("pyfa: Found matplotlib version ", mpl.__version__, " - activating OVER9000 workarounds")
            print("pyfa: Recommended minimum matplotlib version is 1.0.0")
            self.legendFix = True

        mplImported = True

        wx.Frame.__init__(self, parent, title=u"pyfa: Graph Generator", style=style, size=(520, 390))

        i = wx.IconFromBitmap(BitmapLoader.getBitmap("graphs_small", "gui"))
        self.SetIcon(i)
        self.mainFrame = gui.mainFrame.MainFrame.getInstance()
        self.CreateStatusBar()

        self.mainSizer = wx.BoxSizer(wx.VERTICAL)
        self.SetSizer(self.mainSizer)

        sFit = Fit.getInstance()
        fit = sFit.getFit(self.mainFrame.getActiveFit())
        self.fits = [fit] if fit is not None else []
        self.fitList = FitList(self)
        self.fitList.SetMinSize((270, -1))

        self.fitList.fitList.update(self.fits)

        self.graphSelection = wx.Choice(self, wx.ID_ANY, style=0)
        self.mainSizer.Add(self.graphSelection, 0, wx.EXPAND)

        self.figure = Figure(figsize=(4, 3))

        rgbtuple = wx.SystemSettings.GetColour(wx.SYS_COLOUR_BTNFACE).Get()
        clr = [c / 255. for c in rgbtuple]
        self.figure.set_facecolor(clr)
        self.figure.set_edgecolor(clr)

        self.canvas = Canvas(self, -1, self.figure)
        self.canvas.SetBackgroundColour(wx.Colour(*rgbtuple))

        self.subplot = self.figure.add_subplot(111)
        self.subplot.grid(True)

        self.mainSizer.Add(self.canvas, 1, wx.EXPAND)
        self.mainSizer.Add(wx.StaticLine(self, wx.ID_ANY, wx.DefaultPosition, wx.DefaultSize, wx.LI_HORIZONTAL), 0,
                           wx.EXPAND)

        self.gridPanel = wx.Panel(self)
        self.mainSizer.Add(self.gridPanel, 0, wx.EXPAND)

        dummyBox = wx.BoxSizer(wx.VERTICAL)
        self.gridPanel.SetSizer(dummyBox)

        self.gridSizer = wx.FlexGridSizer(0, 4)
        self.gridSizer.AddGrowableCol(1)
        dummyBox.Add(self.gridSizer, 0, wx.EXPAND)

        for view in Graph.views:
            view = view()
            self.graphSelection.Append(view.name, view)

        self.graphSelection.SetSelection(0)
        self.fields = {}
        self.select(0)
        self.sl1 = wx.StaticLine(self, wx.ID_ANY, wx.DefaultPosition, wx.DefaultSize, wx.LI_HORIZONTAL)
        self.mainSizer.Add(self.sl1, 0, wx.EXPAND)
        self.mainSizer.Add(self.fitList, 0, wx.EXPAND)

        self.fitList.fitList.Bind(wx.EVT_LEFT_DCLICK, self.removeItem)
        self.mainFrame.Bind(GE.FIT_CHANGED, self.draw)
        self.Bind(wx.EVT_CLOSE, self.close)

        self.Fit()
        self.SetMinSize(self.GetSize())

    def handleDrag(self, type, fitID):
        if type == "fit":
            self.AppendFitToList(fitID)

    def close(self, event):
        self.fitList.fitList.Unbind(wx.EVT_LEFT_DCLICK, handler=self.removeItem)
        self.mainFrame.Unbind(GE.FIT_CHANGED, handler=self.draw)
        event.Skip()

    def getView(self):
        return self.graphSelection.GetClientData(self.graphSelection.GetSelection())

    def getValues(self):
        values = {}
        for fieldName, field in self.fields.iteritems():
            values[fieldName] = field.GetValue()

        return values

    def select(self, index):
        view = self.getView()
        icons = view.getIcons()
        labels = view.getLabels()
        sizer = self.gridSizer
        self.gridPanel.DestroyChildren()
        self.fields.clear()

        # Setup textboxes
        for field, defaultVal in view.getFields().iteritems():

            textBox = wx.TextCtrl(self.gridPanel, wx.ID_ANY, style=0)
            self.fields[field] = textBox
            textBox.Bind(wx.EVT_TEXT, self.onFieldChanged)
            sizer.Add(textBox, 1, wx.EXPAND | wx.ALIGN_CENTER_VERTICAL | wx.ALL, 3)
            if defaultVal is not None:
                if not isinstance(defaultVal, basestring):
                    defaultVal = ("%f" % defaultVal).rstrip("0")
                    if defaultVal[-1:] == ".":
                        defaultVal += "0"

                textBox.ChangeValue(defaultVal)

            imgLabelSizer = wx.BoxSizer(wx.HORIZONTAL)
            if icons:
                icon = icons.get(field)
                if icon is not None:
                    static = wx.StaticBitmap(self.gridPanel)
                    static.SetBitmap(icon)
                    imgLabelSizer.Add(static, 0, wx.ALL | wx.ALIGN_CENTER_VERTICAL, 1)

            if labels:
                label = labels.get(field)
                label = label if label is not None else field
            else:
                label = field

            imgLabelSizer.Add(wx.StaticText(self.gridPanel, wx.ID_ANY, label), 0,
                              wx.LEFT | wx.RIGHT | wx.ALIGN_CENTER_VERTICAL, 3)
            sizer.Add(imgLabelSizer, 0, wx.ALIGN_CENTER_VERTICAL)
        self.draw()

    def draw(self, event=None):
        global mpl_version

        values = self.getValues()
        view = self.getView()
        self.subplot.clear()
        self.subplot.grid(True)
        legend = []

        for fit in self.fits:
            try:
                success, status = view.getPoints(fit, values)
                if not success:
                    # TODO: Add a pwetty statys bar to report errors with
                    self.SetStatusText(status)
                    return

                x, y = success, status

                self.subplot.plot(x, y)
                legend.append(fit.name)
            except:
                pyfalog.warning("Invalid values in '{0}'", fit.name)
                self.SetStatusText("Invalid values in '%s'" % fit.name)
                self.canvas.draw()
                return

        if mpl_version < 2:
            if self.legendFix and len(legend) > 0:
                leg = self.subplot.legend(tuple(legend), "upper right", shadow=False)
                for t in leg.get_texts():
                    t.set_fontsize('small')

                for l in leg.get_lines():
                    l.set_linewidth(1)

            elif not self.legendFix and len(legend) > 0:
                leg = self.subplot.legend(tuple(legend), "upper right", shadow=False, frameon=False)
                for t in leg.get_texts():
                    t.set_fontsize('small')

                for l in leg.get_lines():
                    l.set_linewidth(1)
        elif mpl_version >= 2:
            legend2 = []
            legend_colors = {
                0: "blue",
                1: "orange",
                2: "green",
                3: "red",
                4: "purple",
                5: "brown",
                6: "pink",
                7: "grey",
            }

            for i, i_name in enumerate(legend):
                try:
                    selected_color = legend_colors[i]
                except:
                    selected_color = None
                legend2.append(Patch(color=selected_color, label=i_name), )

            if len(legend2) > 0:
                leg = self.subplot.legend(handles=legend2)
                for t in leg.get_texts():
                    t.set_fontsize('small')

                for l in leg.get_lines():
                    l.set_linewidth(1)

        self.canvas.draw()
        self.SetStatusText("")
        if event is not None:
            event.Skip()

    def onFieldChanged(self, event):
        self.draw()

    def AppendFitToList(self, fitID):
        sFit = Fit.getInstance()
        fit = sFit.getFit(fitID)
        if fit not in self.fits:
            self.fits.append(fit)

        self.fitList.fitList.update(self.fits)
        self.draw()

    def removeItem(self, event):
        row, _ = self.fitList.fitList.HitTest(event.Position)
        if row != -1:
            del self.fits[row]
            self.fitList.fitList.update(self.fits)
            self.draw()

class FitList(wx.Panel):
    def __init__(self, parent):
        wx.Panel.__init__(self, parent)
        self.mainSizer = wx.BoxSizer(wx.VERTICAL)
        self.SetSizer(self.mainSizer)

        self.fitList = FitDisplay(self)
        self.mainSizer.Add(self.fitList, 1, wx.EXPAND)
        fitToolTip = wx.ToolTip("Drag a fit into this list to graph it")
        self.fitList.SetToolTip(fitToolTip)

class FitDisplay(gui.display.Display):
    DEFAULT_COLS = ["Base Icon",
                    "Base Name"]

    def __init__(self, parent):
        gui.display.Display.__init__(self, parent)

hraesvelgr commented 7 years ago

logfile ist too big to post (character-limit)

Ebag333 commented 7 years ago

Use gist (link up at the top

hraesvelgr commented 7 years ago

done.

Ebag333 commented 7 years ago

You have to share the link with us. We won't know what it is. :)

hraesvelgr commented 7 years ago

https://gist.github.com/hraesvelgr/1566c1e4a74d1dd51afc202512b4695c

Lunk already has been shared with all the other data you needed :) https://github.com/pyfa-org/Pyfa/issues/1075#issuecomment-289808783

blitzmann commented 7 years ago

[2017-03-28 06:35:45.830000] CRITICAL: gui.graphFrame: Traceback (most recent call last):
  File "C:\Users\André\Documents\eve trade tools\pyfa\library.zip\gui\graphFrame.py", line 37, in <module>
    import matplotlib as mpl
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 1100, in <module>
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 947, in rc_params
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 789, in matplotlib_fname
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 325, in wrapper
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 693, in _get_data_path_cached
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 661, in _get_data_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 13: ordinal not in range(128)

Here's the error. It's due to having unicode characters in the path (André). I don't think we're providing matplotlib with any of our paths, so this may very well be an issue with matplotlib internally... I'll verify tonight and try to work on a fix. :)

blitzmann commented 7 years ago

@Ebag333 excellent work btw on the logging stuff. Wouldn't have this error without it. Kudos :D

Ebag333 commented 7 years ago

Here's the error. It's due to having unicode characters in the path (André). I don't think we're providing matplotlib with any of our paths, so this may very well be an issue with matplotlib internally... I'll verify tonight and try to work on a fix. :)

I'm actually working on the locale tests currently. They're all sorts of messed up because of the revert. I suspect that it has to do with the unicode chars, but some stuff in git thinks they exist and some doesn't. Super frustrating.

Since you haven't submitted a PR on the unicode fix I'll probably tackle that next (unless you have an active branch working on it?)

blitzmann commented 7 years ago

@hraesvelgr also, as a work around, don't use pyfa from a directory with unicode characters. If you install pyfa with the .exe, it should install to Program Files, which should work around the issue and give you support for graphs.

For future reference for researching this issue, quick google results:

http://stackoverflow.com/questions/30095006/python-unicode-decode-error-when-importing-matplotlib https://github.com/matplotlib/matplotlib/issues/3618/ https://github.com/matplotlib/matplotlib/pull/3487

blitzmann commented 7 years ago

@Ebag333

Locale tests for pyfa won't really help for this issue if it's an internal matplotlib, which I suspect it may be. This is happening during the import of matplotlib, which tells me that it's something that are doing. Again, will have more info once I look into it.

Ebag333 commented 7 years ago

Um, no, they wouldn't.

path = os.sep.join([os.path.dirname(__file__), 'mpl-data'])

They don't set __file__ to be unicode, so that's why it breaks.

Quick search doesn't turn up an issue: https://github.com/matplotlib/matplotlib/issues

hraesvelgr commented 7 years ago

So... The issue is because of my name (i.e. the name of the directory) and a problem with ascii-table?

Ok, and if i simply change the name of my directory with "simple" letters?

blitzmann commented 7 years ago

python2, which pyfa is written on, has shit support for unicode characters (rather, the applications have shit support if unicode is not kept in mind while developing). matplotlib has these same limitations - if their development team doesn't keep in mind unicode paths, then it's easy to run into problem, as seems to be the case here

Ok, and if i simply change the name of my directory with "simple" letters?

I strongly advise against changing the actual directory name. There are a lot of things on your system that utilize this and will break if you just arbitrarily change it. :)

Here are your options until this can be verified and fixed (or determined unfixable for some reason):

Use the pyfa installer and install to the Program Files (x86) directory (which is the default path). I'm not sure if different locales have a different name for this directory
Run pyfa from somewhere else on the C:\ drive if you have permissions to do so. So, for example, C:\pyfa\.
Run pyfa off another drive (hard drive, flash drive, what have you), so that the path is simply drive:\pyfa\

I'll keep progress posted in this thread as to what I find out. Thanks :)

hraesvelgr commented 7 years ago

Much appreciated!

blitzmann commented 7 years ago

Can confirm this is a matplotlib issue. The issue has been fixed (and I've tested personally that it works) in later versions of matplotlib, v1.5.0+. Unfortunately pyfa ships with v1.4.3.

https://github.com/matplotlib/matplotlib/commit/3ba4f5a48fbcb9507f3aeb2a8dc17cd82672db6c https://github.com/matplotlib/matplotlib/commit/7c12fa0f938c917faaf5a7b9f9876d2e1b9f34d0

Updating matplotlib is on the list of things to do when we roll out new binaries, so this should be solved at that time. Or I can look into releasing new binaries just for Windows (much easier than OS X).

Until any of that happens, though, if you're interested in using graphing support, please ensure that pyfa doesn't run from a directory which contains Unicode characters.

blitzmann commented 6 years ago

I'm going to assume this issue is not longer a thing with the new 2.x versions for the following reasons:

Python 3 support, which deals with unicode paths better
Updated to matplotlib 2.x, the original issue was that the version were were running didn't have the fix in it, and later versions did.

Going to close this issue as solved, if something else creeps up, please open a new issue :)

ine16 commented 6 years ago

On windows 7 32 bit and matplotlib 2.2.2 the problem still persists

blitzmann commented 6 years ago

@ine16 can you provide some more details? Are you sure it's due to locale issue, or possibly another reason why graph option might be grayed out?

pyfa-org / Pyfa

matplotlib failing to import when running from Unicode path #1075