Lost interactive features during MNE Matplotlib integration in to Kivy application

[ghost]

Hi all,

I'm currently trying to integrate MNE functionality in to a Kivy application, the plots display, however, i lose all interactive functionality, i suspect from the Kivy garden matplotlib backend (https://github.com/kivy-garden/garden.matplotlib/blob/master/backend_kivy.py) being called after any mpl_connect requests are made in the creation of the MNE returned figure from the command fig = raw.plot() or fig = mne.viz.raw_plot(raw)

I don't suppose anyone has any recommendations or ideas to reconnect the functionality? My guess is i need to find and import all required functions called by mpl_connect() in the MNE distribution files and call them again in my Kivy application fig.canvas.mpl_connect().

Any help or comments would be greatly appreciated.

Regards,

Liam

[ghost]

Well i appear to have found a return to some interactive features, in mne\viz\raw.py plot_raw i import the kivy backend after matplotlib is imported:

import matplotlib as mpl mpl.use('module://kivy.garden.matplotlib.backend_kivy')

Then later in the same function, after the comment # set up callbacks i call: from kivy.garden.matplotlib.backend_kivyagg import FigureCanvas livecanvas = FigureCanvas(params['fig']) (Note: i chose to do this here as it is before any of the mpl_connect functions for params['fig'] are called)

Toward the end of the function i comment out: #try: # plt_show(show, block=block) #except TypeError: # not all versions have this # plt_show(show)

Then i return livecanvas with params['fig']: return params['fig'], livecanvas

Now i call livecanvas.draw() back in my kivy application and i have an interactive plot. Though the help box won't open quite yet but i'm sure a bit more poking around will fix this.

Is there a slicker fix for me to do this that won't require me altering the core distributed mne files?

All comments and suggestions are welcome.

Regards,

Liam

[drammock]

I don't have any experience with kivy, but before you spend too much time customizing a workaround I wanted to give you a heads-up that a lot will be changing soon in our 2D plotting code. See #7955 for the WIP implementation, and #7751 for the roadmap. I'm about to be on leave for the next week, but when I'm back in happy to answer any questions you have about the new code.

[ghost]

That's great to see there is a bunch of improvements and optimisation planned for the future, enjoy your week off and i look forward to hearing from you.

With creating the new MNEFigParams, MNEFigure, and MNEBrowseFigure classes in the new update would it be at all possible to allow the user to optionally pass in their own created matplotlib figure to the 2D plotting functions (it may be for a niche audience of one) as the Kivy backend needs FigureCanvas(fig) to be called before any mpl_connect requests are made to maintain interactive functionality, or is this likely to cause issues beyond my scope? It could help anyone else in the future using more peculiar matplotlib backends.

[drammock]

would it be at all possible to allow the user to optionally pass in their own created matplotlib figure to the 2D plotting functions

Just thinking about the code off the top of my head I think that would be possible and not too difficult. But we will have to wait a week to find out if I was right.

[ghost]

Thread Bump

I'm slowly editing my way through the MNE plotting functions to work with Kivy, is it known when rev0.21 will be released?

[larsoner]

Currently scheduled for Sept 15th (two weeks-ish)

[drammock]

@LMBooth note that the changes in #7955 will not be part of 0.21 stable release. They are big changes and will probably get merged into master a few weeks after the release so that we have a long period of time to find and fix bugs before they become part of a stable release.

[ghost]

Thank you @drammock and @larsoner for clarifying that, i'll keep my eyes peeled for when i can grab a copy and see if it works within a Kivy environment. I'm currently modifying ERP plots to integrate in to an application and about to have a go with the MNE-realtime LSL example to work within Kivy too.

[drammock]

@LMBooth I played around today with allowing to pass a Figure instance to the new version of raw.plot(). My naive attempt was to change the _figure() function in mne/viz/_figure.py to the following:

def _figure(toolbar=True, FigureClass=MNEFigure, **kwargs):
    """Instantiate a new figure."""
    from matplotlib import rc_context
    from matplotlib.pyplot import figure
    title = kwargs.pop('window_title', None)  # extract title before init
    fig = kwargs.pop('fig', None)
    if fig is None:
        rc = dict() if toolbar else dict(toolbar='none')
        with rc_context(rc=rc):
            fig = figure(FigureClass=FigureClass, **kwargs)
    else:
        fig.__class__ = FigureClass
        fig.__init__(**kwargs)
    if title is not None:
        _set_window_title(fig, title)
    return fig

The only other modifications I made were to mne.viz.raw.py to add a fig=None to the signature of plot_raw_alt, and to add fig=fig in the params dictionary in that same function. All of this "worked" in the sense that it did take an existing figure instance and plot data, annotations, scrollbars, etc. into it, and with the correct layout. But the interactivity was gone. I confess that it was a fairly quick-and-dirty attempt and I didn't spend a ton of time trying other things, but realistically I can't prioritize getting this working --- I have other development priorities that are grant-related so they can't be bumped. But you're welcome to pick up where I left off and see if you can get it working. If you succeed, and the changes needed are not too byzantine, I would advocate for getting them merged in after #7955 is merged.

[ghost]

Thank you @drammock for having a go for me, i'll have a play with this to see if i can get it working without needing to alter too much of the code, i've taken a copy of the current #7955.

Instead of me locally calling the kivy backend in after each matplotlib call, i could just change the backend to use in my .matplotlib/rcsetup file, so there's even less i need to do within the MNE functions, I'm new to Github and making commits so i'll make sure to go through the contributions guide before suggesting any mergers.

[ghost]

I've followed your suggestion above and got the figure loading in to kivy with the new classes and by passing in a created figure, however, same as you i don't have any interactive capabilities. I'll spend tomorrow afternoon on it and see if i can solve the problem 'elegantly'.

[ghost]

Comments and feedback are welcome on if this is acceptable and how to move forward.

I've found a return to functionality, i've taken the first half of plot_raw_alt up until the point that the params dict is created and made a new function, return_params to return the params dict, i needed to do this as i believe the issue is when the _figure function is called in the else: when fig is renitialised, it overwrote... something i'm not too sure of yet, so i changed that to the following:

def _figure(toolbar=True, FigureClass=MNEFigure, **kwargs):
    """Instantiate a new figure."""
    from matplotlib import rc_context
    from matplotlib.pyplot import figure
    title = kwargs.pop('window_title', None)  # extract title before init
    fig = kwargs.pop('figs', None)
    if fig is None:
        rc = dict() if toolbar else dict(toolbar='none')
        with rc_context(rc=rc):
            fig = figure(FigureClass=FigureClass, **kwargs)
    #else:
        #fig.__class__ = FigureClass
        #fig.__init__(**kwargs)
    #if title is not None:
    #    _set_window_title(fig, title)
    return fig

Then in the creation of the figure i import the MNEBrowseFigure with the params dict (**kwargs), now i have an interactive plot within a Kivy application. To tidy up, it would be nice to get an empty params dict to pass in the figure creation so i don't have to pass a copy of raw beforehand, though i'm not entirely sure if this is possible yet.

class Test(BoxLayout):
    def __init__(self, *args, **kwargs):
        super(Test, self).__init__(*args, **kwargs)
        sample_data_folder = mne.datasets.sample.data_path()
        sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                            'sample_audvis_raw.fif')
        self.raw = mne.io.read_raw_fif(sample_data_raw_file)
        params = return_params(self.raw)
        self.fig = plt.figure(FigureClass=MNEBrowseFigure, **params)
        canvas = FigureCanvas(self.fig)
        plot_raw_alt(self.raw,  butterfly=False, figs = self.fig)
        self.add_widget(canvas)

I also changed the fig=None to figs=None as i didn't want to confuse myself with the created fig object in the raw_plot_alt function.

The function for return_params was placed in viz.raw.py has not been refined in any form, found below:

_RAW_CLIP_DEF = 1.5
def return_params(raw, events=None, duration=10.0, start=0.0, n_channels=20,
                 bgcolor='w', color=None, bad_color=(0.8, 0.8, 0.8),
                 event_color='cyan', scalings=None, remove_dc=True, order=None,
                 show_options=False, title=None, show=True, block=False,
                 highpass=None, lowpass=None, filtorder=4,
                 clipping=_RAW_CLIP_DEF,
                 show_first_samp=False, proj=True, group_by='type',
                 butterfly=False, decim='auto', noise_cov=None, event_id=None,
                 show_scrollbars=True, show_scalebars=True, verbose=None,figs=None):
    """."""
    from ..io.base import BaseRaw
    from . import browse_figure

    info = raw.info.copy()
    sfreq = info['sfreq']
    projs = info['projs']
    # this will be an attr for which projectors are currently "on" in the plot
    projs_on = np.full_like(projs, proj, dtype=bool)
    # disable projs in info if user doesn't want to see them right away
    if not proj:
        info['projs'] = list()

    # handle defaults / check arg validity
    color = _handle_default('color', color)
    scalings = _compute_scalings(scalings, raw, remove_dc=remove_dc,
                                 duration=duration)
    _validate_type(raw, BaseRaw, 'raw', 'Raw')
    decim, picks_data = _handle_decim(info, decim, lowpass)
    noise_cov = _check_cov(noise_cov, info)
    units = _handle_default('units', None)
    unit_scalings = _handle_default('scalings', None)
    _check_option('group_by', group_by,
                  ('selection', 'position', 'original', 'type'))

    # clipping
    _validate_type(clipping, (None, 'numeric', str), 'clipping')
    if isinstance(clipping, str):
        _check_option('clipping', clipping, ('clamp', 'transparent'),
                      extra='when a string')
        clipping = 1. if clipping == 'transparent' else clipping
    elif clipping is not None:
        clipping = float(clipping)

    # be forgiving if user asks for too many channels / too much time
    n_channels = min(info['nchan'], n_channels)
    duration = min(raw.times[-1], float(duration))

    # determine IIR filtering parameters
    if highpass is not None and highpass <= 0:
        raise ValueError(f'highpass must be > 0, got {highpass}')
    if highpass is None and lowpass is None:
        ba = filt_bounds = None
    else:
        filtorder = int(filtorder)
        if filtorder == 0:
            method = 'fir'
            iir_params = None
        else:
            method = 'iir'
            iir_params = dict(order=filtorder, output='sos', ftype='butter')
        ba = create_filter(np.zeros((1, int(round(duration * sfreq)))),
                           sfreq, highpass, lowpass, method=method,
                           iir_params=iir_params)
        filt_bounds = _annotations_starts_stops(
            raw, ('edge', 'bad_acq_skip'), invert=True)

    # compute event times in seconds
    if events is not None:
        event_times = (events[:, 0] - raw.first_samp).astype(float)
        event_times /= sfreq
        event_nums = events[:, 2]
    else:
        event_times = event_nums = None

    # determine trace order
    ch_names = np.array(raw.ch_names)
    ch_types = np.array(raw.get_channel_types())
    if order is None:
        ch_type_order = _DATA_CH_TYPES_ORDER_DEFAULT
        order = [pick_idx for order_type in ch_type_order
                 for pick_idx, pick_type in enumerate(ch_types)
                 if order_type == pick_type]
    elif not isinstance(order, (np.ndarray, list, tuple)):
        raise ValueError('order should be array-like; got '
                         f'"{order}" ({type(order)}).')
    order = (np.arange(len(order)) if group_by == 'original' else
             np.asarray(order))
    # adjust order based on channel selection, if needed
    selections = None
    if group_by in ('selection', 'position'):
        selections = _setup_channel_selections(raw, group_by)
        order = np.concatenate(list(selections.values()))
        default_selection = list(selections)[0]
        n_channels = len(selections[default_selection])

    # if event_color is a dict
    if isinstance(event_color, dict):
        event_color = {_ensure_int(key, 'event_color key'): value
                       for key, value in event_color.items()}
        default = event_color.pop(-1, None)
        default_factory = None if default is None else lambda: default
        event_color_dict = defaultdict(default_factory)
        for key, value in event_color.items():
            if key < 1:
                raise KeyError('event_color keys must be strictly positive, '
                               f'or -1 (cannot use {key})')
            event_color_dict[key] = value
    # if event_color is a string or other MPL color-like thing
    else:
        event_color_dict = defaultdict(lambda: event_color)

    # handle first_samp
    first_time = raw._first_time if show_first_samp else 0
    start += first_time
    event_id_rev = {v: k for k, v in (event_id or {}).items()}

    # generate window title; allow instances without a filename (e.g., ICA)
    if title is None:
        title = '<unknown>'
        fnames = raw._filenames.copy()
        if len(fnames):
            title = fnames.pop(0)
            extra = f' ... (+ {len(fnames)} more)' if len(fnames) else ''
            title = f'{title}{extra}'
            prefix = '...' if len(title) > 60 else ''
            title = f'{prefix}{title[-60:]}'
    elif not isinstance(title, str):
        raise TypeError(f'title must be None or a string, got a {type(title)}')

    # gather parameters and initialize figure
    params = dict(inst=raw,
                  info=info,
                  # channels and channel order
                  ch_names=ch_names,
                  ch_types=ch_types,
                  ch_order=order,
                  picks=order[:n_channels],
                  n_channels=n_channels,
                  picks_data=picks_data,
                  group_by=group_by,
                  ch_selections=selections,
                  # time
                  t_start=start,
                  duration=duration,
                  n_times=raw.n_times,
                  first_time=first_time,
                  decim=decim,
                  # events
                  event_color_dict=event_color_dict,
                  event_times=event_times,
                  event_nums=event_nums,
                  event_id_rev=event_id_rev,
                  # preprocessing
                  projs=projs,
                  projs_on=projs_on,
                  apply_proj=proj,
                  remove_dc=remove_dc,
                  filter_coefs=ba,
                  filter_bounds=filt_bounds,
                  noise_cov=noise_cov,
                  # scalings
                  scalings=scalings,
                  units=units,
                  unit_scalings=unit_scalings,
                  # colors
                  ch_color_bad=bad_color,
                  ch_color_dict=color,
                  # display
                  butterfly=butterfly,
                  clipping=clipping,
                  scrollbars_visible=show_scrollbars,
                  scalebars_visible=show_scalebars,
                  window_title=title,
                  figs=figs)

    return params

For anyone else interested here is the base main Kivy file (it does require installing Kivy, Kivy-garden and the garden matplotlib extension too):

from kivy.app import App
from kivy.lang import Builder
from kivy.uix.boxlayout import BoxLayout
import os
import matplotlib
matplotlib.use("module://kivy.garden.matplotlib.backend_kivy")
from kivy.garden.matplotlib.backend_kivyagg import FigureCanvas
import matplotlib.pyplot as plt
import mne
from mne.viz._figure import MNEBrowseFigure
from mne.viz.raw import return_params, plot_raw_alt
kv = """
<Test>:
    orientation: 'vertical'
    Button:
        text: "MNE Kivy Test Button"
        size_hint_y: None
        height: 40
        on_press: print("testing")
"""
Builder.load_string(kv)

class Test(BoxLayout):
    def __init__(self, *args, **kwargs):
        super(Test, self).__init__(*args, **kwargs)
        sample_data_folder = mne.datasets.sample.data_path()
        sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                            'sample_audvis_raw.fif')
        self.raw = mne.io.read_raw_fif(sample_data_raw_file)
        params = return_params(self.raw)
        self.fig = plt.figure(FigureClass=MNEBrowseFigure, **params)
        canvas = FigureCanvas(self.fig)
        plot_raw_alt(self.raw,  butterfly=False, figs = self.fig)
        self.add_widget(canvas)

class TestApp(App):
    def build(self):
        return Test()
if __name__ == '__main__':
    TestApp().run()```

[ghost]

Slight other snag, so it seems to be an interactive plot with the raw object returned in raw = mne.io.read_raw_fif(sample_data_raw_file), however, i get an error returned with my raw object from raw = mne.io.read_raw_bdf('test_bdf.bdf') where local variable' ax_proj' referenced before assignment in line 391 in init of _figure.py, i assume some functions for creating raw are still in need of updating?

[drammock]

Can you post the error traceback, and a link to the problematic file?

[ghost]

The traceback is as follows:

 Traceback (most recent call last):
   File "C:\Users\Liam\Documents\python\kivy\screen.py", line 41, in <module>
     TestApp().run()
   File "C:\python37\lib\site-packages\kivy\app.py", line 829, in run
     root = self.build()
   File "C:\Users\Liam\Documents\python\kivy\screen.py", line 39, in build
     return Test()
   File "C:\Users\Liam\Documents\python\kivy\screen.py", line 32, in __init__
     self.fig = plt.figure(FigureClass=MNEBrowseFigure, **params)
   File "C:\python37\lib\site-packages\matplotlib\pyplot.py", line 545, in figur
e
     **kwargs)
   File "C:\python37\lib\site-packages\kivy\garden\matplotlib\backend_kivy.py",
line 374, in new_figure_manager
     thisFig = FigureClass(*args, **kwargs)
   File "C:\python37\lib\site-packages\mne\viz\_figure.py", line 391, in __init_
_
     ax_main=ax, ax_help=ax_help, ax_proj=ax_proj,
 UnboundLocalError: local variable 'ax_proj' referenced before assignment

C:\Users\Liam\Documents\python\kivy>

Here's a link to the bdf and py files i'm using, i did create the bdf file myself with my own built converter from a separate in house eeg file format we were using, though it can be loaded in to EEGLab with no problems and it works in 0.20 of MNE in and out of Kivy.

[drammock]

That is a bug in init. I'll push a commit today to fix it.

[drammock]

OK, I've just pushed 52be66c that should fix the error.

[ghost]

Seems to be working great now, i'll keep you posted in case i come across any other issues.

[ghost]

Hi @drammock , i'd like to contribute toward your work on #7955, or build on #7955 when it's complete, to allow the passing of figures to plotting functions, would you be okay with me creating a fork and submitting proposed changes, or would you rather i create my own branch after these changes are eventually merged?

Out of curiosity, is it only the standard plotting function (plot_raw_alt) which has had the new figure classes integrated for now with psd and epoch plots still in need of updating?

[drammock]

@LMBooth the best practice would be to fork mne-tools/mne-python (rather than forking drammock/mne-python) and then add my fork as an additional remote. That will make it easier in the long run to (1) keep mne-python current, and (2) submit your changes as a separate PR (if that turns out to make more sense than rolling it into #7955). After forking the main repository to your user account via the GitHub website, I would do something like:

$ git clone https://github.com/LMBooth/mne-python.git
$ cd mne-python
$ git remote add upstream https://github.com/mne-tools/mne-python.git
$ git remote add drammock https://github.com/drammock/mne-python.git
$ git fetch --all

Then you would have 3 remotes (origin=your fork, drammock=my fork, and upstream=the main shared codebase). Then, to work off of my changes, do

$ git checkout -b my_version drammock/mpl-refactor  # you could call the local branch whatever, doesn't have to be "my_version"

That gives you a local branch mpl-refactor that is up-to-date with #7955 at the time you performed the git fetch. This setup makes it fairly easy to get new changes I've done without immediately sqashing changes that you've made, e.g.:

$ git fetch drammock
$ git checkout -b drammocks_version drammock/mpl-refactor

it also makes rebasing easy, if needed:

$ git fetch drammock
$ git checkout my_version
$ git rebase drammock/mpl-refactor

Finally, you can set it up as a PR-into-my-PR:

$ git push -u origin my_version
$ # then create a PR from LMBooth/mne-python:my_version into drammock/mne_python:mpl-refactor

Let me know if any of that is confusing or if you hit any roadblocks. I can't guarantee I'll be willing to merge your changes into #7955 until I've seen the diff --- It's already a monster PR and I want to be kind to the other devs who have to review it. It may end up making more sense to wait until my PR is merged and do yours as a separate PR into upstream after that, just so the diffs are easier to review.

[ghost]

Thank you so much for explaining that to me Daniel (I'm a total novice to github). I can completely understand not wanting to merge with #7955 due to it complicating the review process. I'll do as you've suggested with the 3 remotes between my own and your forks and the mne main and like you said, save for a separate PR.

I've got the plot_raw_alt function working, when do you expect the psd or epoch functions to have the new figure classes integrated?

[drammock]

when do you expect the psd or epoch functions to have the new figure classes integrated?

hard to say. In principle #7955 could be merged without adding Epoch or ICA support, and those could be separate PRs. If my colleagues agree we should go that route, it might be merged as soon as a week from now. If we wait until Epochs and ICA support are finished, and/or the addition of proposed new functionality (click-drag to show a spectrum plot of the selected time span; right-click channel names to show sensor location) then it's harder to predict... 4-6 weeks maybe?

[drammock]

@LMBooth just a note to say that #7955 has been merged into master for a couple weeks now, and appears to be working reasonably well (only a couple minor bugs have had to be addressed so far). Note that only Raw.plot() is using the new figure class so far; #8381 has Epochs and ICA support mostly working, but will probably be another couple weeks before that's wrapped up and merged.

[zhengliuer]

Comments and feedback are welcome on if this is acceptable and how to move forward.

I've found a return to functionality, i've taken the first half of plot_raw_alt up until the point that the params dict is created and made a new function, return_params to return the params dict, i needed to do this as i believe the issue is when the _figure function is called in the else: when fig is renitialised, it overwrote... something i'm not too sure of yet, so i changed that to the following:

def _figure(toolbar=True, FigureClass=MNEFigure, **kwargs):
    """Instantiate a new figure."""
    from matplotlib import rc_context
    from matplotlib.pyplot import figure
    title = kwargs.pop('window_title', None)  # extract title before init
    fig = kwargs.pop('figs', None)
    if fig is None:
        rc = dict() if toolbar else dict(toolbar='none')
        with rc_context(rc=rc):
            fig = figure(FigureClass=FigureClass, **kwargs)
    #else:
        #fig.__class__ = FigureClass
        #fig.__init__(**kwargs)
    #if title is not None:
    #    _set_window_title(fig, title)
    return fig

Then in the creation of the figure i import the MNEBrowseFigure with the params dict (**kwargs), now i have an interactive plot within a Kivy application. To tidy up, it would be nice to get an empty params dict to pass in the figure creation so i don't have to pass a copy of raw beforehand, though i'm not entirely sure if this is possible yet.

class Test(BoxLayout):
  def __init__(self, *args, **kwargs):
      super(Test, self).__init__(*args, **kwargs)
      sample_data_folder = mne.datasets.sample.data_path()
      sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                          'sample_audvis_raw.fif')
      self.raw = mne.io.read_raw_fif(sample_data_raw_file)
      params = return_params(self.raw)
      self.fig = plt.figure(FigureClass=MNEBrowseFigure, **params)
      canvas = FigureCanvas(self.fig)
      plot_raw_alt(self.raw,  butterfly=False, figs = self.fig)
      self.add_widget(canvas)

I also changed the fig=None to figs=None as i didn't want to confuse myself with the created fig object in the raw_plot_alt function.

The function for return_params was placed in viz.raw.py has not been refined in any form, found below:

_RAW_CLIP_DEF = 1.5
def return_params(raw, events=None, duration=10.0, start=0.0, n_channels=20,
                 bgcolor='w', color=None, bad_color=(0.8, 0.8, 0.8),
                 event_color='cyan', scalings=None, remove_dc=True, order=None,
                 show_options=False, title=None, show=True, block=False,
                 highpass=None, lowpass=None, filtorder=4,
                 clipping=_RAW_CLIP_DEF,
                 show_first_samp=False, proj=True, group_by='type',
                 butterfly=False, decim='auto', noise_cov=None, event_id=None,
                 show_scrollbars=True, show_scalebars=True, verbose=None,figs=None):
    """."""
    from ..io.base import BaseRaw
    from . import browse_figure

    info = raw.info.copy()
    sfreq = info['sfreq']
    projs = info['projs']
    # this will be an attr for which projectors are currently "on" in the plot
    projs_on = np.full_like(projs, proj, dtype=bool)
    # disable projs in info if user doesn't want to see them right away
    if not proj:
        info['projs'] = list()

    # handle defaults / check arg validity
    color = _handle_default('color', color)
    scalings = _compute_scalings(scalings, raw, remove_dc=remove_dc,
                                 duration=duration)
    _validate_type(raw, BaseRaw, 'raw', 'Raw')
    decim, picks_data = _handle_decim(info, decim, lowpass)
    noise_cov = _check_cov(noise_cov, info)
    units = _handle_default('units', None)
    unit_scalings = _handle_default('scalings', None)
    _check_option('group_by', group_by,
                  ('selection', 'position', 'original', 'type'))

    # clipping
    _validate_type(clipping, (None, 'numeric', str), 'clipping')
    if isinstance(clipping, str):
        _check_option('clipping', clipping, ('clamp', 'transparent'),
                      extra='when a string')
        clipping = 1. if clipping == 'transparent' else clipping
    elif clipping is not None:
        clipping = float(clipping)

    # be forgiving if user asks for too many channels / too much time
    n_channels = min(info['nchan'], n_channels)
    duration = min(raw.times[-1], float(duration))

    # determine IIR filtering parameters
    if highpass is not None and highpass <= 0:
        raise ValueError(f'highpass must be > 0, got {highpass}')
    if highpass is None and lowpass is None:
        ba = filt_bounds = None
    else:
        filtorder = int(filtorder)
        if filtorder == 0:
            method = 'fir'
            iir_params = None
        else:
            method = 'iir'
            iir_params = dict(order=filtorder, output='sos', ftype='butter')
        ba = create_filter(np.zeros((1, int(round(duration * sfreq)))),
                           sfreq, highpass, lowpass, method=method,
                           iir_params=iir_params)
        filt_bounds = _annotations_starts_stops(
            raw, ('edge', 'bad_acq_skip'), invert=True)

    # compute event times in seconds
    if events is not None:
        event_times = (events[:, 0] - raw.first_samp).astype(float)
        event_times /= sfreq
        event_nums = events[:, 2]
    else:
        event_times = event_nums = None

    # determine trace order
    ch_names = np.array(raw.ch_names)
    ch_types = np.array(raw.get_channel_types())
    if order is None:
        ch_type_order = _DATA_CH_TYPES_ORDER_DEFAULT
        order = [pick_idx for order_type in ch_type_order
                 for pick_idx, pick_type in enumerate(ch_types)
                 if order_type == pick_type]
    elif not isinstance(order, (np.ndarray, list, tuple)):
        raise ValueError('order should be array-like; got '
                         f'"{order}" ({type(order)}).')
    order = (np.arange(len(order)) if group_by == 'original' else
             np.asarray(order))
    # adjust order based on channel selection, if needed
    selections = None
    if group_by in ('selection', 'position'):
        selections = _setup_channel_selections(raw, group_by)
        order = np.concatenate(list(selections.values()))
        default_selection = list(selections)[0]
        n_channels = len(selections[default_selection])

    # if event_color is a dict
    if isinstance(event_color, dict):
        event_color = {_ensure_int(key, 'event_color key'): value
                       for key, value in event_color.items()}
        default = event_color.pop(-1, None)
        default_factory = None if default is None else lambda: default
        event_color_dict = defaultdict(default_factory)
        for key, value in event_color.items():
            if key < 1:
                raise KeyError('event_color keys must be strictly positive, '
                               f'or -1 (cannot use {key})')
            event_color_dict[key] = value
    # if event_color is a string or other MPL color-like thing
    else:
        event_color_dict = defaultdict(lambda: event_color)

    # handle first_samp
    first_time = raw._first_time if show_first_samp else 0
    start += first_time
    event_id_rev = {v: k for k, v in (event_id or {}).items()}

    # generate window title; allow instances without a filename (e.g., ICA)
    if title is None:
        title = '<unknown>'
        fnames = raw._filenames.copy()
        if len(fnames):
            title = fnames.pop(0)
            extra = f' ... (+ {len(fnames)} more)' if len(fnames) else ''
            title = f'{title}{extra}'
            prefix = '...' if len(title) > 60 else ''
            title = f'{prefix}{title[-60:]}'
    elif not isinstance(title, str):
        raise TypeError(f'title must be None or a string, got a {type(title)}')

    # gather parameters and initialize figure
    params = dict(inst=raw,
                  info=info,
                  # channels and channel order
                  ch_names=ch_names,
                  ch_types=ch_types,
                  ch_order=order,
                  picks=order[:n_channels],
                  n_channels=n_channels,
                  picks_data=picks_data,
                  group_by=group_by,
                  ch_selections=selections,
                  # time
                  t_start=start,
                  duration=duration,
                  n_times=raw.n_times,
                  first_time=first_time,
                  decim=decim,
                  # events
                  event_color_dict=event_color_dict,
                  event_times=event_times,
                  event_nums=event_nums,
                  event_id_rev=event_id_rev,
                  # preprocessing
                  projs=projs,
                  projs_on=projs_on,
                  apply_proj=proj,
                  remove_dc=remove_dc,
                  filter_coefs=ba,
                  filter_bounds=filt_bounds,
                  noise_cov=noise_cov,
                  # scalings
                  scalings=scalings,
                  units=units,
                  unit_scalings=unit_scalings,
                  # colors
                  ch_color_bad=bad_color,
                  ch_color_dict=color,
                  # display
                  butterfly=butterfly,
                  clipping=clipping,
                  scrollbars_visible=show_scrollbars,
                  scalebars_visible=show_scalebars,
                  window_title=title,
                figs=figs)

    return params

For anyone else interested here is the base main Kivy file (it does require installing Kivy, Kivy-garden and the garden matplotlib extension too):

from kivy.app import App
from kivy.lang import Builder
from kivy.uix.boxlayout import BoxLayout
import os
import matplotlib
matplotlib.use("module://kivy.garden.matplotlib.backend_kivy")
from kivy.garden.matplotlib.backend_kivyagg import FigureCanvas
import matplotlib.pyplot as plt
import mne
from mne.viz._figure import MNEBrowseFigure
from mne.viz.raw import return_params, plot_raw_alt
kv = """
<Test>:
  orientation: 'vertical'
  Button:
      text: "MNE Kivy Test Button"
      size_hint_y: None
      height: 40
      on_press: print("testing")
"""
Builder.load_string(kv)

class Test(BoxLayout):
  def __init__(self, *args, **kwargs):
      super(Test, self).__init__(*args, **kwargs)
      sample_data_folder = mne.datasets.sample.data_path()
      sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                          'sample_audvis_raw.fif')
      self.raw = mne.io.read_raw_fif(sample_data_raw_file)
      params = return_params(self.raw)
      self.fig = plt.figure(FigureClass=MNEBrowseFigure, **params)
      canvas = FigureCanvas(self.fig)
      plot_raw_alt(self.raw,  butterfly=False, figs = self.fig)
      self.add_widget(canvas)

class TestApp(App):
  def build(self):
      return Test()
if __name__ == '__main__':
  TestApp().run()```

Hi, now I am using PyQT5 to embed the interactive figure from plot_raw inside a GUI, and I have encountered the same question. I tried to reproduce your work but it seems that mne has changed a lot. For example, there is no return_params and plt_raw_alt in mne.viz.raw. Which version of mne did you use? And any methods to solve the same problem using the latest version of mne?

[zhengliuer]

And I think I get your main idea to solve this problem, which is get the params of the raw data and set up a Figure instance.

[zhengliuer]

No plot_raw_alt now? What's the alternative?

[drammock]

@BarryLiu-97 plot_raw_alt was a temporary function name used during development of the new Figure class in #7955. Once it was working properly, its name changed to plot_raw and it replaced the old version of that function. If you want to try out @LMBooth's approach, you can do so in your own fork of current master version, by modifying mne.viz.plot_raw and some of the functions in mne/viz/_figure.py, as described above.

If you find a way of making interactivity work when embedding the result raw.plot() into an existing canvas/GUI, please open a PR to discuss whether the solution is one that is general enough (and does not create too much maintenance burden) to be included in MNE-Python.

[zhengliuer]

Hi, recently I was trying to embed an interactive matplotlib figure into a PyQt5 GUI. I think I have some progress. While here is one sample I think is relevent.

import matplotlib
from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib.backends.backend_qt5agg import NavigationToolbar2QT as NavigationToolbar
# from matplotlib.backends.qt_compat import QtCore
from PyQt5 import QtCore
from matplotlib.figure import Figure
from PyQt5 import QtWidgets
import sys
import numpy as np

class My_Main_window(QtWidgets.QMainWindow):
    def __init__(self, parent=None):
        super(My_Main_window, self).__init__(parent)

        self.canvas = FigureCanvas(Figure(constrained_layout=True))
        self.canvas.draw()
        self.ax = self.canvas.figure.add_subplot()

        self.con_1 = np.random.random((30, 40, 10))
        self.num = 0
        con_plot = self.con_1[:, :, self.num]
        self.im = self.ax.matshow(con_plot)
        self.ax.set_title(self.num)
        self.canvas.figure.colorbar(self.im)
        self.canvas.mpl_connect('key_press_event', self.on_move)
        self.canvas.mpl_connect(
            "button_press_event", lambda *args, **kwargs: print(args, kwargs)
        )
        self.canvas.mpl_connect(
            "key_press_event", lambda *args, **kwargs: print(args, kwargs)
        )

        self.setCentralWidget(self.canvas)

        # Give the keyboard focus to the figure instead of the manager:
        # StrongFocus accepts both tab and click to focus and will enable the
        # canvas to process event without clicking.
        # https://doc.qt.io/qt-5/qt.html#FocusPolicy-enum
        self.canvas.setFocusPolicy(QtCore.Qt.StrongFocus)
        self.canvas.setFocus()
        print(type(self.canvas))

    def on_move(self, event):
        print(f"activate this {event.key}, {self.num}")
        if event.key == "left":
            if self.num == 0:
                self.num = 0
                con_plot = self.con_1[:, :, self.num]
            else:
                self.num -= 1
                con_plot = self.con_1[:, :, self.num]
        elif event.key == "right":
            if self.num == self.con_1.shape[2] - 1:
                self.num = 0
                con_plot = self.con_1[:, :, self.num]
            else:
                self.num += 1
                con_plot = self.con_1[:, :, self.num]
        else:
            con_plot = self.con_1[:, :, self.num]
        # update the image in place
        self.im.set_data(con_plot)
        self.ax.set_title(self.num)
        # tell Qt that the widget needs to be re-repainted
        self.canvas.draw_idle()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    app.setApplicationName("Plot")
    main_win = My_Main_window()
    main_win.show()
    app.exec()

The key part is

self.canvas.setFocusPolicy(QtCore.Qt.StrongFocus)
        self.canvas.setFocus()

so the gui will focus on the events aiming at the canvas. While it didn't work when I used it in mne.viz.plot() Here is the code:

import matplotlib
from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib.backends.qt_compat import QtCore
from PyQt5 import QtWidgets
import sys
from mne.datasets import sample
import mne

class My_Main_window(QtWidgets.QMainWindow):
    def __init__(self, parent=None):
        super(My_Main_window, self).__init__(parent)

        data_path = sample.data_path()
        fpath= data_path + '/MEG/sample/sample_audvis_raw.fif'
        self.data = mne.read_epochs(fpath, preload=True)
        self.fig = mne.viz.plot_epochs(self.data, title='1', show=False)
        self.canvas = FigureCanvas(self.fig)
        self.canvas.draw()
        self.stack = QtWidgets.QStackedWidget()
        self.stack.addWidget(self.canvas)
        self.setCentralWidget(self.stack)

        # Give the keyboard focus to the figure instead of the manager:
        # StrongFocus accepts both tab and click to focus and will enable the
        # canvas to process event without clicking.
        # https://doc.qt.io/qt-5/qt.html#FocusPolicy-enum
        self.canvas.setFocusPolicy(QtCore.Qt.StrongFocus | QtCore.Qt.WheelFocus)
        self.canvas.setFocus()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    app.setApplicationName("Plot")
    main_win = My_Main_window()
    main_win.show()
    app.exec()

@drammock

[larsoner]

I think there is nothing to do for this issue at this point so I'll close, but feel free to comment/reopen if I'm mistaken

[drammock]

@LMBooth I don't know if you're still working on this, but I wonder if the new-ish Matplotlib subfigure feature might be helpful?

[LMBooth]

Hi again @drammock, i switched my github account so missed your previous tags, i've actually just come back round to doing post analysis again using mne, i've moved away from Kivy and like @BarryLiu97 am looking at integrating the MNE viewer in to a custom PyQt5 data plotter and xdf logger for some custom in-house EEG and ECG devices, i'll have a look at the subfigure feature and see if it's what i'm after.

I'm pretty sure the 0.23 build of MNE should be fine without any modification if the layout you want the canvas to sit in is created after getting the figure from calling raw.plot().

If i get it working I'll post a short example.

[LMBooth]

I've just implemented focus settings similar to that discussed by @BarryLiu97 above and i have full functionality of my plots, great stuff!

[LMBooth]

One issue i am having actually is getting the selected channels from the plot. According to the handling bad channels tutorial, around the section discussing 'Blocking Execution' it states how block should be set to True if you want to mark selected channels as bad on the fly - that is if im reading it right.

I've attached a modified example of @BarryLiu97 code to illustrate the problem i'm having:

from PyQt5 import QtWidgets, QtGui
import matplotlib
from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib.backends.qt_compat import QtCore
import sys
from mne.datasets import sample
import mne
import os 
class My_Main_window(QtWidgets.QMainWindow):
    def __init__(self, parent=None):
        super(My_Main_window, self).__init__(parent)

        sample_data_folder = sample.data_path()
        sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                    'sample_audvis_filt-0-40_raw.fif')
        self.raw = mne.io.read_raw_fif(sample_data_raw_file)
        self.fig = self.raw.plot(show=False, block=True)

        self.canvas = FigureCanvas(self.fig)
        self.canvas.draw()
        layout = QtWidgets.QVBoxLayout()
        layout.addWidget(self.canvas)
        self.button = QtWidgets.QPushButton('Print Bad', self)
        self.button.clicked.connect(self.BadButton)
        layout.addWidget(self.button)

        self.stack = QtWidgets.QTabWidget()
        self.stack.setLayout(layout)
        self.setCentralWidget(self.stack)

        # Give the keyboard focus to the figure instead of the manager:
        # StrongFocus accepts both tab and click to focus and will enable the
        # canvas to process event without clicking.
        # https://doc.qt.io/qt-5/qt.html#FocusPolicy-enum
        self.canvas.setFocusPolicy(QtCore.Qt.StrongFocus | QtCore.Qt.WheelFocus)
        self.canvas.setFocus()

    def BadButton(self):
        print(self.raw.info['bads'])

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    app.setApplicationName("Plot")
    main_win = My_Main_window()
    main_win.show()
    app.exec()

After selecting multiple extra plots to be marked as bad the raw.info["bads"] field isn't updated, am i overlooking something here or is this perhaps a bug? @drammock