BUG: Overflow error with raw.plot()

[cbrnr]

I have a specific BrainVision file that I cannot plot:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/matplotlib/backends/backend_agg.py", line 152, in draw_path
    self._renderer.draw_path(gc, path, transform, rgbFace)
OverflowError: In draw_path: Exceeded cell block limit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/matplotlib/backends/backend_macosx.py", line 74, in _draw
    self.figure.draw(renderer)
  File "/usr/local/lib/python3.8/site-packages/matplotlib/artist.py", line 38, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/matplotlib/figure.py", line 1735, in draw
    mimage._draw_list_compositing_images(
  File "/usr/local/lib/python3.8/site-packages/matplotlib/image.py", line 137, in _draw_list_compositing_images
    a.draw(renderer)
  File "/usr/local/lib/python3.8/site-packages/matplotlib/artist.py", line 38, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 2630, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)
  File "/usr/local/lib/python3.8/site-packages/matplotlib/image.py", line 137, in _draw_list_compositing_images
    a.draw(renderer)
  File "/usr/local/lib/python3.8/site-packages/matplotlib/artist.py", line 38, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/matplotlib/lines.py", line 802, in draw
    renderer.draw_path(gc, tpath, affine.frozen())
  File "/usr/local/lib/python3.8/site-packages/matplotlib/backends/backend_agg.py", line 154, in draw_path
    raise OverflowError("Exceeded cell block limit (set "
OverflowError: Exceeded cell block limit (set 'agg.path.chunksize' rcparam)

If you suspect this is an IPython 7.13.0 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

I think this is because there might be too many data points to draw in this data set. Other data sets work, and this is also unrelated to matplotlib (I tried older versions, same problem). Not sure what's so special about this file (maybe the sampling frequency of 5kHz?):

>>> raw
<RawBrainVision | brainvision.eeg, 16 x 815200 (163.0 s), ~99.5 MB, data loaded>
>>> raw.info
<Info | 7 non-empty values
 bads: []
 ch_names: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
 chs: 16 EEG
 custom_ref_applied: False
 highpass: 0.0 Hz
 lowpass: 2500.0 Hz
 meas_date: unspecified
 nchan: 16
 projs: []
 sfreq: 5000.0 Hz
>

I can share the data if anyone wants to try to replicate.

[larsoner]

I cannot replicate with a simple example that might test the sampling rate as being the only cause:

import numpy as np
import mne

raw = mne.io.RawArray(np.random.RandomState(0).randn(16, 100000) * 1e-6,
                      mne.create_info(16, 10000., 'eeg'))
raw.plot()

Does it cause an issue for you?

[cbrnr]

Nope, your example works for me too...

[larsoner]

Feel free to either work out what the difference is between your data and the simpler example, or share the data and someone else can try

[cbrnr]

So far I haven't found a possible reason for this strange behavior. If you have time it'd be great if you could take a look (here's the data). I'll try this on Windows to see if it only happens on macOS.

import mne
raw = mne.io.read_raw_brainvision('data.vhdr', preload=True)
raw.plot()

[cbrnr]

It works on Windows, but the plot is extremely slow (because there seem to be a lot of data points to plot):

[larsoner]

Can't reproduce on Linux. I would try clipping='clamp' or clipping='transparent'.

Sometimes I wonder if our default should actually be clipping='clamp', or maybe something likeclipping=2. which clipped channels once they went 2x (or whatever the float is) beyond their isolated range (hence clipping='clamp' would be an alias for clipping=1.). Something like this would really help when scales are way off -- it makes things much more readable and plots faster --but could still work pretty well for normal data where some overlap is okay (e.g., highly correlated EEG data like blinks) I think.

[cbrnr]

Alright, clipping='clamp' works, but the plot is still super slow; clipping='transparent' works and the plot is super fast (because the signals are not plotted apparently).

I'm :+1: for changing to default to avoid cases where the plot doesn't work at all.

[larsoner]

I quickly implemented the clipping=1.5 as a default and this is what it looks like for sample:

I actually prefer clipping='transparent':

But for data like yours clipping='transparent' makes the traces disappear completely which is not great (this is because of how np.nan/masking values are used):

Another option would be to do the clipping properly with matplotlib using a mask, which wouldn't make the data disappear. This would have the nicest appearance, but I strongly suspect it wouldn't help this particular problematic use case (though really this seems like quite a corner case, with workarounds, so I'm hesitant to base our global defaults on this case).

[cbrnr]

Disappearing traces are better than the plot giving an error IMO so I'd prefer anything over the current behavior.

[larsoner]

Okay here it is using matplotlib clipping to achieve 1.5, which seems to work quite nicely for sample:

I doubt it will directly solve your problem, but I think we might want to add some special-case code for it specifically. The simplest would probably be a try/except in whatever draw call we make in MNE that leads to the error at your end, check the traceback, and make a suggestion to use clipping='clamp' before reraising.

[cbrnr]

Good idea, this example doesn't work with any float value. The last line in MNE is:

https://github.com/mne-tools/mne-python/blob/master/mne/viz/raw.py#L908

I'll try to see if catching the OverflowError works, and if so I'll submit a PR.

[cbrnr]

It doesn't work, I can't catch the error. It's probably because there is more than one exception inside matplotlib. Any suggestions?

[larsoner]

@cbrnr I cannot replicate on my macOS machine, maybe you need to update matplotlib?

Also doing:

try:
    params['fig'].canvas.draw()
except Exception as exp:
    warn(...)
    raise

should really catch any exception that derives from Exception (which should reasonably include anything that matplotlib would raise). Did you try this? If this fails you can try:

try:
    ...
except:
    ...

but this is rightfully frowned upon so better to avoid it if possible

[larsoner]

... also it's probably worth seeing if this is also a problem when you run with MPLBACKEND=Qt5Agg python .... I suspect it will be since it looks like an agg drawing problem, but who knows

[cbrnr]

Yes, it also fails with Qt5Agg. What exactly can't you reproduce? I'm already using the latest matplotlib 3.2.1...

[larsoner]

I don't get a failure with raw.plot() for that dataset

[cbrnr]

Could it be related to retina?

[larsoner]

My macOS machine has a retina display, if it matters. You should be able to test this regardless of your actual monitory by using something like QT_SCALE_FACTOR=2 on the Qt backend

[cbrnr]

@agramfort you have a Mac too - can you replicate with the data and code from https://github.com/mne-tools/mne-python/issues/7521#issuecomment-605013819?

[agramfort]

I get on my mac:

OverflowError: Exceeded cell block limit (set 'agg.path.chunksize' rcparam)

Traceback (most recent call last):
  File "/Users/alex/miniconda3/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 146, in draw_path
    self._renderer.draw_path(gc, path, transform, rgbFace)
OverflowError: In draw_path: Exceeded cell block limit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alex/miniconda3/lib/python3.7/site-packages/matplotlib/backends/backend_qt5.py", line 505, in _draw_idle
    self.draw()
  File "/Users/alex/miniconda3/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 388, in draw
    self.figure.draw(self.renderer)
  File "/Users/alex/miniconda3/lib/python3.7/site-packages/matplotlib/artist.py", line 38, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/alex/miniconda3/lib/python3.7/site-packages/matplotlib/figure.py", line 1707, in draw
    self.patch.draw(renderer)
  File "/Users/alex/miniconda3/lib/python3.7/site-packages/matplotlib/artist.py", line 38, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/alex/miniconda3/lib/python3.7/site-packages/matplotlib/patches.py", line 580, in draw
    self._facecolor if self._facecolor[3] else None)
  File "/Users/alex/miniconda3/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 148, in draw_path
    raise OverflowError("Exceeded cell block limit (set "
OverflowError: Exceeded cell block limit (set 'agg.path.chunksize' rcparam)

[cbrnr]

Thanks @agramfort, that's what I get too. Now it would be interesting to find out why this file works for @larsoner...

[larsoner]

I tried on:

matplotlib:    3.1.3 {backend=Qt5Agg}

And can't get it to bump higher. Maybe it's because you're on 3.2.3 (presumably conda-forge)?

[agramfort]

Platform: Darwin-19.0.0-x86_64-i386-64bit Python: 3.7.5 (default, Oct 25 2019, 10:52:18) [Clang 4.0.1 (tags/RELEASE_401/final)] Executable: /Users/alex/miniconda3/bin/python CPU: i386: 16 cores Memory: 32.0 GB

mne: 0.21.dev0 numpy: 1.17.4 {blas=mkl_rt, lapack=mkl_rt} scipy: 1.3.1 matplotlib: 3.1.1 {backend=TkAgg}

sklearn: 0.23.dev0 numba: 0.46.0 nibabel: 3.0.1 cupy: Not found pandas: 0.25.3 dipy: 1.0.0 mayavi: 4.7.1 {qt_api=pyqt5, PyQt5=5.13.2} pyvista: 0.24.0 vtk: 8.2.0

[cbrnr]

No, I use pip packages. I'll try to downgrade tomorrow and see if this makes it work (but I doubt it because @agramfort has an even older version).

[larsoner]

Another user on Gitter has had a problem solved by switching to Qt5Agg. I say we officially recommend it to macOS users. There continue to be bugs and they need it anyway for the new 3D functionality to work, so it seems easier for us as a package to support Qt5Agg across all three platforms.

[agramfort]

Ok for me

[cbrnr]

First, I don't think this issue is related to the backend at all.

I'm OK with recommending Qt5Agg also on macOS, but I think it is important that we still try to fix issues related to the macosx backend. This is still the default MPL backend, so we need to make sure that basic plotting works. Switching to Qt5Agg on a Mac requires users to include one or two additional lines in their scripts/interactive sessions, or change the MPL default backend (not obvious, especially for new Pythonistas).

[kylemath]

Just running into a problem where tkAgg crashes my macbook during any plotting (which I have had before in other software). Came to search for other issues reporting this, and how to avoid having to add this backend change to tutorials and test scripts, etc:

import matplotlib
matplotlib.use('Qt5Agg')

I fixed this by changing MPL backend on my machine like this: https://stackoverflow.com/a/57090889

[cbrnr]

Coming back to the original issue, I tried the MWE and for some reason it doesn't crash anymore. It's still super super slow though (scrolling by one unit takes about 30 seconds, but part of that might be due to my MacBook being super slow on scaled resolutions).

I think the issue is not a high sampling frequency, but some channels are scaled incorrectly. If I modify these channels as follows, plotting works as expected:

import mne
raw = mne.io.read_raw_brainvision('data.vhdr', preload=True)
raw._data[8] *= 1e-5
raw._data[9:] *= 1e-2
raw.plot(block=True)

Therefore, I'm closing this issue because at least it doesn't crash anymore.