Matplotlib 2.2.3 fails to launch a plot window, under SageMath 8.4

fusion809 commented 5 years ago

Issue description

Under SageMath 8.4 (@timokau) I have noticed that my plots in Matplotlib are failing to launch. Here is what I am running in SageMath (doesn't seem to matter if I run it in the notebook interface, or from the command-line):

x,y,z=var('x,y,z')

# Next we define the parameters
sigma=10
rho=28
beta=8/3

# The Lorenz equations
lorenz=[sigma*(y-x),x*(rho-z)-y,x*y-beta*z]

# Time and initial conditions
N=250000
tmax=250
h=tmax/N
t=srange(0,tmax+h,h)
ics=[0,1,1]
sol=desolve_odeint(lorenz,ics,t,[x,y,z],rtol=1e-13,atol=1e-14)
X=sol[:,0]
Y=sol[:,1]
Z=sol[:,2]

# Plot the result
from mpl_toolkits.mplot3d import axes3d
from matplotlib import pyplot as plt
# Call the plot function if you want to plot the data
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot(X, Y, Z, lw=0.5)
ax.set_xlabel("X Axis")
ax.set_ylabel("Y Axis")
ax.set_zlabel("Z Axis")
ax.set_title("Lorenz Attractor")

plt.show()

and while the output I see at the command-line:

[<mpl_toolkits.mplot3d.art3d.Line3D object at 0x7fe017d18e90>]
Text(0.5,0,'X Axis')
Text(0.5,0,'Y Axis')
Text(0.5,0,'Z Axis')
Text(0.5,0.92,'Lorenz Attractor')

matches what I see when SageMath is running properly, no plot window appears. When I run SageMath 8.4 from its official Ubuntu 18.04 tarball, I do not experience this issue on openSUSE Tumbleweed (oddly this tarball works fine on Tumbleweed). The plot window appears, and shows the expected Lorenz attractor in 3D wireframe format:

screenshot_20181124_022704

Steps to reproduce

On openSUSE Tumbleweed install Matplotlib 2.2.3 on Python 2.7 and SageMath 8.4 with:

nix-env -i python2.7-matplotlib-2.2.3 sage-8.4

(after installing Nix with the installer script at https://nixos.org/nix/install and setting up PATH to point to $HOME/.nix-profile/bin), then run:

sage

and under SageMath, either its notebook interface (started by running notebook() in the prompt that starts after running sage) or right at the sage: prompt, run:

x,y,z=var('x,y,z')

# Next we define the parameters
sigma=10
rho=28
beta=8/3

# The Lorenz equations
lorenz=[sigma*(y-x),x*(rho-z)-y,x*y-beta*z]

# Time and initial conditions
N=250000
tmax=250
h=tmax/N
t=srange(0,tmax+h,h)
ics=[0,1,1]
sol=desolve_odeint(lorenz,ics,t,[x,y,z],rtol=1e-13,atol=1e-14)
X=sol[:,0]
Y=sol[:,1]
Z=sol[:,2]

# Plot the result
from mpl_toolkits.mplot3d import axes3d
from matplotlib import pyplot as plt
# Call the plot function if you want to plot the data
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot(X, Y, Z, lw=0.5)
ax.set_xlabel("X Axis")
ax.set_ylabel("Y Axis")
ax.set_zlabel("Z Axis")
ax.set_title("Lorenz Attractor")

plt.show()

Technical details

$ nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.51 MiB download, 2.48 MiB unpacked):
  /nix/store/06ix6l8hb9962x5y90hv9lhvd22v0ra2-nix-info
  /nix/store/1hmvnxp69a6x27cw0c9c8jnhjfcssbxh-bash-interactive-4.4-p23-dev
  /nix/store/9qh104j8iwlq7gm2g8gybmrg04ym3770-bash-interactive-4.4-p23-doc
  /nix/store/b8avbljnbsdwpxb3vrya9gflbrsyjjf5-bash-interactive-4.4-p23-info
  /nix/store/v0mj4rw355qvih4pxp6nihhz7g4w0k92-bash-interactive-4.4-p23-man
copying path '/nix/store/9qh104j8iwlq7gm2g8gybmrg04ym3770-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/1hmvnxp69a6x27cw0c9c8jnhjfcssbxh-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/b8avbljnbsdwpxb3vrya9gflbrsyjjf5-bash-interactive-4.4-p23-info' from 'https://cache.nixos.org'...
copying path '/nix/store/v0mj4rw355qvih4pxp6nihhz7g4w0k92-bash-interactive-4.4-p23-man' from 'https://cache.nixos.org'...
copying path '/nix/store/06ix6l8hb9962x5y90hv9lhvd22v0ra2-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.2-1-default, openSUSE Tumbleweed, noversion`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.1.3`
 - channels(fusion809): `"nixpkgs-19.03pre159739.3ae0407d3c9"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/fusion809/channels/nixpkgs`

Thanks for your time, Brenton

FRidh commented 5 years ago

nix-env -i python2.7-matplotlib-2.2.3 sage-8.4

Installing a Python library like this is not supported. I can't tell you how it is supposed to work with sage though.

fusion809 commented 5 years ago

I didn't actually run that command, I only ran nix-env -i sage, I included the python2.7-matplotlib-2.2.3 (which FYI was already installed on my system when I decided to start plotting with Matplotlib) in case it wasn't drawn in by SageMath and was only on my system due to the packages I have installed in the past. I think this is, in fact, a Matplotlib issue and not a SageMath one as I have also not been able to get Matplotlib to plot plots handed to it by Julia, per #50415.

symphorien commented 5 years ago

if the default backend is gtk based: see this article in the wiki: https://nixos.wiki/wiki/Python#Gnome You probably need to run sage inside nix-shell -p gobjectIntrospection gtk3 For other backends, I don't know.

fusion809 commented 5 years ago

Well I'll be ....ed, @symphorien your comment has saved my Nix-based math experience! Matplotlib is now working, I merely needed to run:

nix-shell -p gobjectIntrospection gtk3 'python36.withPackages(ps : with ps; [ matplotlib pygobject3 ipython ])'

and then run Julia in the resulting prompt to get its PyPlot module to work (which relies on Matplotlib), and for SageMath I'm not sure what fixed it, but I think it was:

nix-shell -p gobjectIntrospection gtk3 'python2.withPackages (ps: with ps; [ pygobject3 ])' --run "python -c \"import pygtkcompat; pygtkcompat.enable_gtk(version='3.0')\""

timokau commented 5 years ago

sage does have matplotlib as a dependency, so you don't need to pull it in explicitly.

Thank you for reporting this. Relying on the user to somehow find out they have to nix-shell -p gobjectIntrospection gtk3 is very bad UX, so I do consider this a bug. @symphorien do you know if there is an easy way to fix that in the matplotlib package without user interaction?

@fusion809 just FYI in case you planned to invest much time in it, the sage notebook is deprecated.

symphorien commented 5 years ago

My understanding is that there are two problems here:

matplotlib is a lib but requires a specific runtime environment (environment variables for the chosen backend), a bit like libsane, which depends on LD_LIBRARY_PATH. The naive solution would be to wrap all dependent packages, with wrapGappsHook for example.
The required environment variables depend on the backend chosen by the user. My impression is that most packages in nixpkgs come with all their optional dependencies included by default, but here I am afraid having matplotlib depend on qt, gtk, tk and others at the same time might be too much.

fusion809 commented 5 years ago

@timokau Yeah I know it is deprecated, but from what I've read it is not going to be removed, simply not further developed. Thanks for the FYI in case I didn't know though.

timokau commented 5 years ago

Shouldn't it be possible to patch matplotlib to set those environment variables itself on import? Not sure if there is some code that always runs on import though, does python always run the top-level python file even if I do from matplotlib import y?

@fusion809 It will be removed. See https://groups.google.com/forum/#!topic/sage-devel/UgVUEFwL_D4 for details.

fusion809 commented 5 years ago

@timokau Thanks I think I read: https://trac.sagemath.org/ticket/25837 and focused on the parts where they said until the Jupyter notebook finally obtains all the features of the old SageMath notebook, the traditional notebook will not be removed. Must have misread that originally, thanks for bringing that sad fact to my attention. I guess it is time for me to develop Jupyter knowledge.

Guessing that question of yours isn't directed at me? Nix is a programming knowledge I know very little about, and the Nix package manager I only know the basics of how to use it.

symphorien commented 5 years ago

Yes importing a python file will execute it. In libraries they usually just define stuff, but they can set the environment. cc @jtojnar who wrote wrapGappHook: would it be sensible to "inline" the wrapper inside matplotlib to ensure gtk can be used by all dependent packages ?

timokau commented 5 years ago

@fusion809 that is right, it won't be removed until its a complete replacement. There is an increasing push in that direction currently though. One of the reasons is that nobody wants to port the sage notebook to python3 and python3 compatibility is planned soon-ish (given the deadline).

Reopening because the matplotlib issue is still around, even though there is a workaround.

jtojnar commented 5 years ago

If you want to get rid of the wrapGAppsHook requirement, you will need to do the same thing GNOME extensions do: prepend Gtk and the typelibs it depends on (Cairo, Pango…) to GIRepository search path: https://github.com/NixOS/nixpkgs/blob/master/pkgs/desktops/gnome-3/core/gnome-shell-extensions/fix_gmenu.patch It needs to be done in this file: https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/backends/_gtk3_compat.py

If the libraries also call file chooser dialogue, it will be even harder and will require patching GTK with GSettings schema paths.

Like this: https://github.com/NixOS/nixpkgs/blob/984285355a57145561a5a1cb049b3a3170f87532/pkgs/applications/backup/deja-dup/hardcode-gsettings.patch

In https://gitlab.gnome.org/GNOME/gtk/blob/master/gtk/gtkfilechooserutils.c#L458

timokau commented 5 years ago

sage apparently by default does not show any interactive plots. It instead expects you to write plots into a file. You can however select a different backend using %matplotlib <backend>.

I've just pushed #51032 to add tkinter (%matplotlib tk). Within the now working jupyter notebook %matplotlib inline works without issues. Not sure why it displayed plots on the "default ubuntu tarball", but I tried it in a sage build from source and it doesn't display anything by default.

NixOS / nixpkgs