Hy and matplotlib

[atisharma]

When using Hy, it is not possible to display figures with matplotlib.

Expected behaviour:

1. On wayland, no figure is shown.
2. On X11, a figure outline is shown but not drawn.

To reproduce:

(import matplotlib)
(.use matplotlib "Qt5Agg")
(.rcdefaults matplotlib)
(.use matplotlib "Qt5Agg")

(import [matplotlib [pyplot]])

(.ion pyplot)
(pyplot.style.use "default")
(pyplot.style.use "fivethirtyeight")

(defn test []
 "Plot a list."
 (setv fig (.figure pyplot))
 (setv axes (.add_subplot fig))
 (.plot axes (list (range 20)))
 {:figure fig :axes axes})

(test)

[Olical]

Does this work if you just run a Hy REPL outside of the editor?

I can think of no reason why Conjure would be any different, I'm just running a REPL and sending code to it, so if it works in one and not the other I won't really know where to start. I'll still try to work it out, but I guess it'll be some env var thing, maybe it knows it's being run in a kind of headless way and it's behaving differently?

[atisharma]

Thanks for the very fast response.

I don't know why either. Yes, it works in a straight hy repl. One possible wrinkle: I'm launching nvim (or nvim-qt) after activating a venv (e.g. . /venv/hyvenv/bin/activate; nvim test.hy).

[Olical]

Pretty sure I've borded down the other side of your profile photo :smile: I'll give this a look as soon as I can, just has to be in my evenings and weekends so might not be for liiiitle while. I'm mostly unfamiliar with Hy (matplotlib even more) so I'm flying a little blind here.

[atisharma]

:) Ping me if you need help.

[taw10]

I took a quick look out of curiosity. I can reproduce the problem without the venv, and with the following minimal example:

(import matplotlib)
(import [matplotlib [pyplot]])
(.ion pyplot)
(setv fig (.figure pyplot))
(setv axes (.add_subplot fig))
(.plot axes (list (range 20)))

The same lines copy/pasted into a Hy REPL in a terminal work fine, observing that an empty and responsive plot window shows up after the fourth line. Interestingly, with the equivalent Python code typed into a command-line REPL, the window only opens up once you add an additional fig.show() at the end. I don't know why it would be any different:

import matplotlib
from matplotlib import pyplot
pyplot.ion
fig = pyplot.figure()
axes = fig.add_subplot()
axes.plot(range(0,20))
fig.show()

The behaviour is similar to what happens if you start a GUI program from a shell and immediately pause it with Ctrl-Z. The REPL and Conjure itself are still responsive, and stopping the REPL with cS works as expected (the un-redrawn window vanishes). It's just that the GUI part (a separate thread started by Python, I think) does not seem to actually run. I suspect it's related to this question - sadly 8 years old with no good solution. Why on earth it would behave differently in this case, I don't know. The only difference when run via Conjure is that Python is not the process group leader?

[Olical]

I had a look at this today and can reproduce it but can't find a way around right now :disappointed: I was trying to set a flag when spawning the process that will make the process a group leader (I think this would fix the issue?) but the libuv "flags" property doesn't seem to do anything.

I wonder if Neovim isn't passing the flags option through to libuv under the hood? If so, I'm not sure how I'd configure this. We want this flag I think:

    /*
    * Spawn the child process in a detached state - this will make it a process
    * group leader, and will effectively enable the child to keep running after
    * the parent exits. Note that the child process will still keep the
    * parent's event loop alive unless the parent process calls uv_unref() on
    * the child's process handle.
    */
    UV_PROCESS_DETACHED = (1 << 3),
    /*

Taken from http://docs.libuv.org/en/v1.x/process.html#c.uv_process_options_t

I tried adding :flags (bit.lshift 1 3) but it just doesn't seem to make a difference. Even like -10 or 500 as a flag is fine and I'd expect it to error if I pass something bad through, so I suspect it's just not threaded through at all? If someone can work this out before me it might just work!

[Olical]

I've tried everything I can to make the process the group leader but still no dice :disappointed: the best I got was adding a show with :block True set:

(defn test []
 "Plot a list."
 (setv fig (.figure pyplot))
 (setv axes (.add_subplot fig))
 (.plot axes (list (range 20)))
 (.show pyplot :block True)
 {:figure fig :axes axes})

This displays the window but it'll block the REPL until you close it. A step in the right direction maybe, but not great. I've actually seen other people complaining about matplotlib+python in REPLs running under VS Code etc with no answers or responses. So this seems like a pretty wide spread general problem!

I wonder if internally inside matplotlib it is looking for an env var or something that says "I'm running under IPython", if so, we could maybe spoof that environment and trick it into working interactively! I have a feeling there's some explicit override inside matplot that's holding this back.

If someone can find this needle in the haystack I'd be super grateful! It'll be hard for me to find since I'm unfamiliar with gestures wildly this whole area of software :sweat_smile:

[Olical]

Maybe something like this: https://github.com/matplotlib/matplotlib/blob/b0a840d50fc20c4385b83b31cffb383172d4be19/lib/matplotlib/pyplot.py#L112

But that doesn't seem to quite do the trick. It does seem like there's a LOT of code in matplotlib that's checking for ipython! So maybe my theory is kinda correct!

[Olical]

Since I can't find a good way to do this and other REPLs seem to have the same issue I'm afraid I'll have to close this unless someone else can think of / find a good solution? I'll leave this open for a while longer, but I don't plan on spending any more time on it right now.

It seems like some assumptions in matplotlib are getting in the way, if someone with more knowledge of it (I have essentially zero) can find a way to trick it I'll be happy to implement it.

[atisharma]

Maybe it would be better as a matplotlib issue. Unfortunately I don't understand the cause well enough to open an issue there without it looking like a conjure problem. Sadly this is a bit outside my skill set.

[Olical]

Hmm perhaps! Although I guess it's more of a question than an issue 🤔 I wonder if they'd be okay with that on their issues, I hope it'll be okay. Could lead to an answer of "it won't work in that situation" or maybe a small change that enables this use case + fixes it for a bunch of other people trying to do matplotlib things in REPLs.