IPython notebook hangs

[dhermes]

Running the provided sample causes my entire machine (not just the example) to hang when using the subplot examples.

Machine info:

$ ipython --version
2.3.0
$ pip show oct2py

---
Name: oct2py
Version: 2.4.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: 
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"
$ octave --version
GNU Octave, version 3.8.1
...

[dhermes]

That was my laptop. Works fine on my desktop (previous stacktrace was erroneous).

It seems causing the entire laptop to hang is related to windowing / UI frameworks but I'm not very familiar with them.

As a (potentially helpful) start, I have the following matplotlib plotting backends:

    OPTIONAL BACKEND EXTENSIONS
                    macosx: no  [Mac OS-X only]
                    qt5agg: no  [PyQt5 not found]
                    qt4agg: yes [installing, Qt: 4.8.6, PyQt: 4.8.6]
                    pyside: yes [installing, Qt: 4.8.6, PySide: 1.2.1]
                   gtk3agg: yes [installing, version 3.8.10]
                 gtk3cairo: yes [installing, version 3.8.10]
                    gtkagg: yes [installing, Gtk: 2.24.23 pygtk: 2.24.0]
                     tkagg: yes [installing, version 81008]
                     wxagg: yes [installing, version 2.8.12.1]
                       gtk: yes [installing, Gtk: 2.24.23 pygtk: 2.24.0]
                       agg: yes [installing]
                     cairo: yes [installing, pycairo version 1.8.8]
                 windowing: no  [Microsoft Windows only]

[blink1073]

Sorry @dhermes, I just saw this Issue report. Do you have gnuplot-x11 installed?

[dhermes]

Yes:

$ dpkg -s gnuplot-x11
Package: gnuplot-x11
Status: install ok installed
Priority: optional
Section: math
...

Should it be un-installed? (i.e. is that a good thing or a bad thing)

[blink1073]

Hmm, can you try firing up just ipython (no notebook) and running %paste with this code on your clipboard:

from oct2py import octave
octave.eval("""
subplot(121);
[x, y] = meshgrid(0:0.1:3);
r = sin(x - 0.5).^2 + cos(y - 0.5).^2;
surf(x, y, r);

subplot(122);
sombrero()
""")

[dhermes]

@blink1073 It works just fine from IPython.

[blink1073]

How about this (from ipython), copy this to clipboard:

%%octave -g
subplot(121);
[x, y] = meshgrid(0:0.1:3);
r = sin(x - 0.5).^2 + cos(y - 0.5).^2;
surf(x, y, r);
subplot(122);
sombrero()

And run this IPython session:

In [1]:  %load_ext oct2py.ipython
In [2]:  %paste

[dhermes]

No issues there either.

I need to re-run the offending notebook (haven't since 11/6). It may be a heisenbug based on some transient kernel / install issue that no longer exists.

I will update after trying (it may take awhile if I have to reboot my machine).

UPDATE: @blink1073 it still brings my machine to its knees.

[blink1073]

Sometimes I have to kill my .ipython folder to clear up notebook issues.

On Fri, Dec 19, 2014 at 8:45 PM, Danny Hermes notifications@github.com wrote:

No issues there either.

I need to re-run the offending notebook (haven't since 11/6). It may be a heisenbug based on some transient kernel / install issue that no longer exists.

I will update after trying (it may take awhile if I have to reboot my machine).

— Reply to this email directly or view it on GitHub https://github.com/blink1073/oct2py/issues/61#issuecomment-67722291.

[dhermes]

I blew up ~/.ipython and it still brought the machine to its knees.

The %%octave -s 500,500 statements are what cause the issue. I isolated this for both of the cells (butterworth filter and sombrero) and got the same behavior.

[blink1073]

Does using the size argument have the same effect in the console (just adding it to my example above)?

[dhermes]

@blink1073 The size argument is the deal breaker. It caused the same issue from the console. I still had to reboot.

[blink1073]

I tried an almost identical setup on my machine with no failures:

$ /usr/bin/ipython --version
2.3.1
silvester: ~ 
$ sudo pip show oct2py
---
Name: oct2py
Version: 2.4.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: 
silvester: ~ 
$ cat /etc/lsb-release
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=17
DISTRIB_CODENAME=qiana
DISTRIB_DESCRIPTION="Linux Mint 17 Qiana"
silvester: ~ 
$ octave --version
GNU Octave, version 3.8.1

Perhaps try uncommenting this line (it should be commented in the released version) and run again from the console?

[dhermes]

Yes I have a Desktop set up identically with no issues. It has to be some low-level library that either isn't installed on one machine or that doesn't work with the drivers on the laptop.

---

I adding the logging and tried a few things.

I ran just from oct2py import octave and the only thing logged was

'NoneType' object has no attribute 'terminate'

on exit. (This was the same if I executed octave.eval(...) after the import.)

Just loading %load_ext oct2py.ipython resulted in

'NoneType' object has no attribute 'write'
'NoneType' object has no attribute 'terminate'

on exit. Executing the sombrero plot with -g (which works fine) I got a large log output but it doesn't seem to be insightful.

Executing with -s (the butterworth snippet from the example notebook) the log was identical except for the code snippet. (I didn't get a chance to capture because the computer froze.)

[blink1073]

Yep, I'm at a loss. The only thing I can think of that might do it is setting the DefaultFigurePosition might not work with your setup. You could try commenting out those lines and see if it works.

[dhermes]

I'm happy to dig into the source to debug, I'm just unfamiliar with how -g and -s pass from IPython to your extension. Can you provide some pointers?

[blink1073]

When you pass -s, the magic function passes the height and width to Oct2Py.eval, which then gets the appropriate plotting commands from Oct2Py._get_plot_commands. This function adds Octave commands to set the figure size (using DefaultFigurePostion), and to optionally save the figure to a file (if the plot dir was set).

If the -g flag is not set, the plot_dir is set to a temp directory, which the magic function reads and passes to IPython, and the Octave plot window is not shown. I don't think the -g flag is your problem, but you could try leaving it out.

[dhermes]

So two interesting things have happened.

1. I've traved the error to _Session.evaluate, particularly self.write when the output is the command wrapped with some other stuff.
2. When Chrome is closed, my computer does not freeze. However, the command still takes about 15 seconds to complete and no plot is ever created. I also tried this with Chrome closed, but Firefox open, and there are no issues there.

Any thoughts?

[blink1073]

I have no idea about your second item, but there's a couple more things we can check. How about:

from oct2py import octave
octave.eval("set(0, 'DefaultFigurePosition', [300, 200, 500, 500]);")
print('one')
octave.eval("__oct2py_figures = [];")
print('two')
octave.eval("close all;")
print('three')
octave.eval("__oct2py_figure_visible = 'off';")
print('four')
octave.eval("plot([1,2,3])")
print('five')

[blink1073]

(preferrably from the IPython prompt using %paste)

[dhermes]

That works just fine and creates a plot. (With Chrome open, whatever issue that might be responsible for.)

[blink1073]

Grasping at straws (trying to mimic the OctaveMagic function call)...

from oct2py import octave
octave.eval("set(0, 'DefaultFigurePosition', [300, 200, 500, 500]);", return_both=True, verbose=False)
print('one')
octave.eval("__oct2py_figures = [];", return_both=True, verbose=False)
print('two')
octave.eval("close all;", return_both=True, verbose=False)
print('three')
octave.eval("__oct2py_figure_visible = 'off';", return_both=True, verbose=False)
print('four')
octave.eval("plot([1,2,3])", return_both=True, verbose=False)
print('five')

[dhermes]

ISTM the issue is process contention and I don't fully understand what is happening when you write to an active process:

    def write(self, message):
        """Write a message to the process using utf-8 encoding"""
        os.write(self.wfid, message.encode('utf-8'))

Also, I captured the output after using -g and it seems that

• __oct2py_figures never gets populated in the -g case or the -s case
• -s doesn't make a plot from the console because there is no drawnow("expose") statement (this is probably not true)

[blink1073]

• The write method is the same one used by pexpect when writing to a child process stdin. I don't think that's the problem, but IPC is a black art.
• You can tell if __oct2py_figures is getting populated by not using -g and adding disp(outfile) after the outfile = sprintf(...) line in core.py.
• drawnow is tied to -g, not -s (it should be there when -g is set, because it forces the Octave plot window to draw).

[blink1073]

Hi @dhermes, would you care to try this with the latest version (3.5) and see if it works for you?

[josephcslater]

Sorry @dhermes didn't reply. I'm in a similar scenario to him and still seeing a hang. I saw your instructions above, and got it unstuck using -g. However, now it matches the failure that I get in my linux installation.

Thoughts? (Please be a bit patient- I'm likely to only be able to test a solution this weekend).

[blink1073]

@josephcslater, I'll look at this after we get https://github.com/Calysto/octave_kernel/issues/50 fixed.

[josephcslater]

Thanks. Let me know if I can test anything.

[blink1073]

@josephcslater, your issue should be fixed by upgrading oct2py to v3.5.10.

[josephcslater]

Unfortunately, it's still hanging.

%load_ext oct2py.ipython
%%octave
a = 1:10
disp(a)

Just hangs.

Likewise

x = %octave [1 2; 3 4];
x

hangs.

How do I verify that the update "stuck" and I am loading the new extension?

[blink1073]

I just tried it again on Ubuntu 14.04, and it is working fine, can you verify the following?

In [1]: import oct2py; oct2py.__version__
Out[1]: '3.5.10'

[josephcslater]

Confirmed.

Works fine on my Ubuntu installation as well. Failing on my Mac.

[blink1073]

Ah, I had to follow the "Using Aquaterm with Gnuplot" from http://wiki.octave.org/Octave_for_MacOS_X to get it to work on my Mac.

[josephcslater]

I'm not even trying to plot. Should this matter?

[blink1073]

Yes because oct2py doesn't know if you are trying to plot until after you run the command, so we treat every command as a possible plotting command.

[josephcslater]

Hmmm, so this may not work for me. What I have working is octave 4.0.3 installed using macports (not brew). It plots via xterm, not aquaterm . Not sure where to go from here.

[josephcslater]

I don't think this is my problem. I just ran gnu plot, changed the terminal to aqua (set term aqua) and plotting followed fine. So, gnu plot will recognize aqua. It may be that I need to set the default to aqua for oct2py interacting with it to work?

[blink1073]

That shouldn't be required, AFAIK. Are you able to plot from octave-cli without changing any settings?

[josephcslater]

Yes.

[josephcslater]

At least this isn't (so far?) the face plant of my issue #50 (I owe you a beer for that level of stupid on my part!).

[blink1073]

I'm not entirely sure where to go from here. I am not keen to install Macports and spending another round of debugging the inner loop at the moment.

[josephcslater]

I get it. Have an idea how I could trace it?

[josephcslater]

It's not a strong need of mine right now, but more of a "this would be nice to have", so I appreciate how much you've already put into it.

[blink1073]

You could try putting print statements in the inner evaluate function, that is essentially what I ended up doing this morning to debug the Homebrew plotting. I'll be afk for the rest of the day.

[josephcslater]

Thank you, and enjoy the time away.

Sent from my iPhone

On Nov 19, 2016, at 12:14 PM, Steven Silvester notifications@github.com wrote:

You could try putting print statements in the inner evaluate function, that is essentially what I ended up doing this morning to debug the Homebrew plotting. I'll be afk for the rest of the day.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[blink1073]

Okay, I think the problem is trying to do control flow logic (for loops in this case) through the pipe, which is too brittle. v3.6.0 adds a separate _make_figs.m function that encapsulates the logic. Do you mind testing that release?

[josephcslater]

Works! Thanks!

That you would ask if I mind... I owe it to you (and other freeware authors (which I am as well) to do so. Thank you for coming back to this.

FWIW: The failure happened even when no figures were involved. Further, figures are indeed working.

In light of this: I thought I should test your demo ipynb, it unfortunately failed on the cell

%%octave -s 500,500

# butterworth filter, order 2, cutoff pi/2 radians
b = [0.292893218813452  0.585786437626905  0.292893218813452];
a = [1  0  0.171572875253810];
freqz(b, a, 32);

the fail results being:

---------------------------------------------------------------------------
ExpatError                                Traceback (most recent call last)
<ipython-input-10-bca0f2444242> in <module>()
----> 1 get_ipython().run_cell_magic('octave', '-s 500,500', '\n# butterworth filter, order 2, cutoff pi/2 radians\nb = [0.292893218813452  0.585786437626905  0.292893218813452];\na = [1  0  0.171572875253810];\nfreqz(b, a, 32);')

/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2113             magic_arg_s = self.var_expand(line, stack_depth)
   2114             with self.builtin_trap:
-> 2115                 result = fn(magic_arg_s, cell)
   2116             return result
   2117 

<decorator-gen-124> in octave(self, line, cell, local_ns)

/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/Users/jslater/Library/Python/3.5/lib/python/site-packages/oct2py/ipython/octavemagic.py in octave(self, line, cell, local_ns)
    337             if plot_format == 'svg':
    338                 image = self._fix_gnuplot_svg_size(image, size=(plot_width,
--> 339                                                                 plot_height))
    340             display_data.append((key, {plot_mime_type: image}))
    341 

/Users/jslater/Library/Python/3.5/lib/python/site-packages/oct2py/ipython/octavemagic.py in _fix_gnuplot_svg_size(self, image, size)
    109 
    110         """
--> 111         (svg,) = minidom.parseString(image).getElementsByTagName('svg')
    112         viewbox = svg.getAttribute('viewBox').split(' ')
    113 

/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/dom/minidom.py in parseString(string, parser)
   1966     if parser is None:
   1967         from xml.dom import expatbuilder
-> 1968         return expatbuilder.parseString(string)
   1969     else:
   1970         from xml.dom import pulldom

/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/dom/expatbuilder.py in parseString(string, namespaces)
    923     else:
    924         builder = ExpatBuilder()
--> 925     return builder.parseString(string)
    926 
    927 

/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/dom/expatbuilder.py in parseString(self, string)
    221         parser = self.getParser()
    222         try:
--> 223             parser.Parse(string, True)
    224             self._setup_subset(string)
    225         except ParseEscape:

ExpatError: not well-formed (invalid token): line 189, column 92

[josephcslater]

Now hanging is:

%%octave -s 600,200 -f png
# Note: On Windows, this will not show the plots unless Ghostscript is installed.

subplot(121);
[x, y] = meshgrid(0:0.1:3);
r = sin(x - 0.5).^2 + cos(y - 0.5).^2;
surf(x, y, r);

subplot(122);
sombrero()

[josephcslater]

The final cell (using the octave plotting GUI) works fine.

Of course, order matters. It can't be executed with the others hanging.

[josephcslater]

Also hanging is:

%%octave -f svg
# Note: On Windows, this will not show the plots unless Ghostscript is installed.

subplot(121);
[x, y] = meshgrid(0:0.1:3);
r = sin(x - 0.5).^2 + cos(y - 0.5).^2;
surf(x, y, r);

subplot(122);
sombrero()

Interrupting yields the following. I have no idea of the value:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-7e66b99b039b> in <module>()
----> 1 get_ipython().run_cell_magic('octave', '-f svg', '# Note: On Windows, this will not show the plots unless Ghostscript is installed.\n\nfigure(1)\nsubplot(121);\n[x, y] = meshgrid(0:0.1:3);\nr = sin(x - 0.5).^2 + cos(y - 0.5).^2;\nsurf(x, y, r);\n\nsubplot(122);\nsombrero()')

/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2113             magic_arg_s = self.var_expand(line, stack_depth)
   2114             with self.builtin_trap:
-> 2115                 result = fn(magic_arg_s, cell)
   2116             return result
   2117 

<decorator-gen-124> in octave(self, line, cell, local_ns)

/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/Users/jslater/Library/Python/3.5/lib/python/site-packages/oct2py/ipython/octavemagic.py in octave(self, line, cell, local_ns)
    305                                                 plot_name=plot_name,
    306                                                 verbose=False,
--> 307                                                 return_both=True)
    308         except oct2py.Oct2PyError as exception:
    309             msg = str(exception)

ValueError: too many values to unpack (expected 2)

However, it works fine with the -g option.

[blink1073]

Thanks again for the reports @josephcslater, I'll try to sneak in some time to work on this over the holiday.