Fixes #21. Plot source estimates using ipyvolume.

[OlehKSS]

Function for plotting source estimates using ipyvolume. In this version it does everything previously created functions for visualizing raw data do.

So far, most of the features of original function plot_source_estimates have been implemented.

All implemented:

• [x] time_label,
• [x] time_viewer,
• [x] initial_time,
• [x] time_unit.
• [x] cortex, it not possible to provide any options different from the default one
• [x] foreground: used for text color (so far), implemented
• [x] views as a list added
• [x] colorbar, done with bqplot
• [x] title of the figure, added.
• [x] the figure parameter was added, so far it will raise NotImplementedError.

Not added on purpose:

• [x] spacing(can be applied to the matplotlib-based solution only), omitted.

[larsoner]

I have a question regarding SourceEstimates.data property, the shape of that numpy array is around 30000 by 25, now I am doing stc.crop(0.09, 0.09) and select 0th column stc.data[:, 0]. Is it correct approach?

You shouldn't need to do any .crop operation. Based on the time index the user has selected, you display the correct time index with something like stc.data[:, time_idx].

Another question regarding the SourceEstimate data type, should the number of vertices of the mesh be equal to the number of data points of stc object?

In general no, there should be an order of magnitude fewer in stc than in the source space mesh. This is what the smoothing_steps parameter does in PySurfer. We have code to compute the upsampling matrix in MNE using http://www.martinos.org/mne/stable/generated/mne.compute_morph_matrix.html with something like:

mat = compute_morph_matrix(subject, subject, stc.vertices, [np.arange(len(lh_surf['rr'])), np.arange(len(rh_surf['rr']))], smoothing_steps, subjects_dir)

then to upsample you do mat.dot(stc.data[:, time_idx]) because mat is a SciPy sparse matrix.

For the other params, play with PySurfer until you are comfortable. But here is a short version:

• time_label : text in the lower right of the figure that is updated to show the time
• time_viewer : in PySurfer this is a new window, for you it should be an IPython Widget slider
• initial_time : time in time_unit (s or ms) to show (can figure out the index using stc.times
• time_unit : 's' or 'ms', how to convert to times
• cortex : the MNE docstring is outdated, see http://pysurfer.github.io/generated/surfer.Brain.html#surfer.Brain and adapt the code from it
• foreground : color used for things in the foreground, usually just used for text objects (if you set background='white' then foreground='auto' will be foreground='k', and if background='k' then it will use foreground='w', etc. -- the logic for this lives in PySurfer, you can take it
• spacing : only used for matplotlib backend, don't bother with this
• colorbar : for now just omit it, please open an issue to add it, it will probably take a bit of work / potential collaboration with IPyvolume folks
• title of the figure : we don't use this in stc.plot
• figure : omit this, I think we always want to create a new one for now

Also, please add an issue for how we want to deal with multiple views and hemi. In PySurfer you can do hemi='lh' | 'rh' | 'both' | 'split', as well as views='lat' or views=['lat', 'med']. Play with it until you get a sense of this. Hopefully IPyvolume gives you some way of having multiple sub-panels/sub-scenes in a single rendering context (i.e., a 2x2 grid where each panel in the grid has its own objects and camera controls). If it already has these mechanisms the translation should be straightforward. If it does not, then please open an IPyvolume issue discussing this use case so we can discuss potential solutions.

[OlehKSS]

I have added two issues according to the comments above: #27, #26. By looking at ipyvolume examples, I am not sure whether there will be a need to use a foreground color for ipyvolume backend. Well, it doesn't seem to be of the biggest priority so far.

As far as I understand from stc.data[:, time_idx] , time_idx is an epoch, right? So, I will need to add selection of proper time_idx and timing functions, options for the cortex (seems like a lot of them).

[larsoner]

cortex is a very low priority.

time_idx is the index corresponding to a particular time point in the sources x time array.

In MNE "epoch" refers to a channels x time array (basically a trial), typically one of set of epochs in an array of shape n_epochs x channels x time

[OlehKSS]

Ok, I have crossed cortex out of my list, and I have added an issue for it #28.

[OlehKSS]

I have added initial_time parameter support for plot_source_estimates function. I have also support for time_viewer=True, so this pull request will also fix #22. Am I missing anything else? Can we do a code review in case everything looks good?

Example of cortex graph with time scale enalbled

[agramfort]

nice !

how do I test this? is it fast to interact?

[OlehKSS]

@agramfort Use this notebook for testing.

I am using ipyvolume package installation from source code. Moreover, I am awaiting my pull request to ipyvolume with WebGL configuration changes to be merged. So, I suggest doing installation from that branch, since otherwise you will have a plot slightly different from what I have.

[OlehKSS]

And it quite fast to interact

[OlehKSS]

I have changed slider label to show correct time instance

[larsoner]

Great. Eventually we want this time_label to be used the same way that it is in stc.plot, which IIRC means it will respect the time_unit parameter, too

[OlehKSS]

I have fixed time label and added time_label parameter, so now this behavior corresponds to the behavior of plot_source_estimate function

[larsoner]

Works great, and is smooth on my workstation.

@OlehKSS maybe the next thing to do is to complete the set of arguments for the plot_source_estimates API. For arguments that are not supported with this backend yet, raise NotImplementedError (e.g., if cortex is not True or hemi == 'split'). You'll probably need to add support for hemi='lh' and hemi='rh' (but not yet hemi='split'), as well as the set of views that PySurfer supports, which should be easy enough.

Once we have that, perhaps it is time to actually move the code over to mne-python as an experimental feature?

[agramfort]

I see this on my system:

any idea what's going on? I am using PR/145 from ipyvolume as you suggested.

[larsoner]

Do the main ipyvolume examples work? I'm guessing not.

What browser? IIRC Chrome tends to work best.

[agramfort]

indeed they were not. I managed to fix it in another conda env. Really nice !

I would be great to expose sliders to set the colorbar fmin, fmid and fmax. And I also agree with @larsoner that using bqplot for the colorbar would really bring us close to a fully functional viewer in the notebook.

great job.

[OlehKSS]

I have added NotImplementedError for cortex, hemi="split", foreground, colorbar=True, and for unsupported values (np.array) of colormap. I have also added support for time_label of callable type. hemi="lh" | "rh" was implemented before, as well as views. I experience problems with changing views using ipyvolume, it seems like a bug to me, because ipyvolume graph does not change its view at all, no matter which arguments are provided. I have tested on my old code which was working before, I do not obtain the same results now.

[OlehKSS]

As for adding a new backend to MNE:plot_source_estimates, I think it can be added in the same way as _plot_mpl_stc function. E. g. I can add a _plot_ipv_stc function, which will be similar to what I have now implemented in this pull request. Should I make a pull request to MNE repository afterwards?

[larsoner]

Let's see if we can get the bqplot colorbar interaction working, then yeah a PR to mne-python adding a private function would work. Then future PRs would probably be made directly to mne-python

[OlehKSS]

It seems like bqplot has no support for custom color maps and transparent colors. I was trying to create a color bar using heatmap and found this problem. It is possible to provide a list of several colors (seems like no more than three entries). There is no 'hot' color scheme available in bqplot. Well, it seems like bqplot support only color maps available in d3.js and several more, list of color maps. I asked bqplot guys for clarification. Nevertheless, it seems like matplotlib suits more to what we need.

[choldgraf]

One thing you could probably do is to just use the numpy array color outputs from the colormaps in matplotlib and build a bqplot colorscale out of it: http://bqplot.readthedocs.io/en/stable/_generate/bqplot.scales.ColorScale.html

[OlehKSS]

ColorScale constructor of bqplot accepts only a list of strings of colors specified in hexidecimal format. Output values from colormaps in matplotlib can be converted to comply with that type of format. Nevertheless, I see several problems so far. First of all, ColorScale doesn't support transparency channel, what to do with transparent colors in our case? Second, it doesn't seem to support centered colormaps and it takes into account only first three colors while creating an instance of ColorScale class.

[OlehKSS]

I have been also trying to implement colormap with matplotlib, the problem I am facing now is that a plot done with matplotlib cannot be easily put together with existing ipywidgets. One way to overcome it is to create a png-image out of the plot and show it using Ipywidgets:Image widget. Another way to visualize data is using d3js in Jupyter and write code for visualization in HTML/CSS/JavaScript. I haven't tried this approach yet. Is there any other library for interactive data visualization similar to bqplot I can check?

[agramfort]

what do ipyvolume / bqplot folks suggest?

[OlehKSS]

After almost a week I got no answer from bqplot folks. I have sent them a message via gitter. Should I open an issue in bqplot repository instead?

[larsoner]

Transparency isn't a problem, you can use the underlying cortex average color (usually gray) and do the "transparency" mixing yourself. This is what we do already in PySurfer.

As for bqplot etc. I thought it was possible to have matplotlib interactive plots with widgets? If so then a colorbar should work.

As for Gitter, many times it is not actively monitored so an issue would be better

[OlehKSS]

It is possible to create interactive plots with matplotlib itself, by default they have several control buttons unnecessary for a colorbar. I will check how to customize it and whether it is possible at all. I have added an issue in bqplot repository similar to my question on gitter.

[OlehKSS]

I have added a color bar with bqplot, using custom library build from the repository master branch, since the changes that I need have not been released yet. I have pushed my code to a separate branch.

Color bar

[OlehKSS]

Should I add the code for color bar creation here or should we merge this pull request and continue to work on color bar addition in a different pull request?

[larsoner]

Adding it here is okay with me. But is the colorbar interactive, i.e., can you set the fmin/fmid/fmax for both one-sided (hot) and two-sided (mne) colormaps?

[OlehKSS]

@larsoner the color bar is not interactive, for now it displays standard settings. It works in this way for mne and hot color maps. I am not sure how interactivity should be implemented like, do you have any suggestions? I have added the code for the color bar creation to this pull request.

[agramfort]

@OlehKSS I discussed briefly with @larsoner and we agree that this code should end up in a pysurfer-like package and not directly in MNE. It will help spreading the tool at there is a lot more pysurfer users than MNE users.

We would shoot for a package called "ipysurfer" that mimics the API of PySurfer.

Then we'll integrate it into MNE.

what do you think?

[OlehKSS]

Yeah, sure. In order to implement ipysurfer, should we create a different repository or continue to work in this one?

[agramfort]

we'll rename this one but you should converge to a pysurfer like project structure with compatible API ASAP.

[OlehKSS]

I have added an issue for creating ipysurfer package, see #29.

[larsoner]

For the colorbar you should be able to use widgets to set the min/mid/max float values, and the colorbar and brain plot should both update

[larsoner]

IIRC in some topomap viz we actually make it so you can click and drag in the colorbar to change limits but we can always add this later (and we want the float slider/entry boxes either way)

[larsoner]

... you can get an idea for this already based on how PySurfer does it (like with most things). In this case, time_viewer=True shows float sliders.

[OlehKSS]

I have changed upsampling, so now it uses sparse matrix and does upsampling for a selected time instance. Animation controls provided by ipyvolume also work, but now it is not that quick. I have added sliders for changing control points of the color map. I could not find a PySurfer example of sliders for color map control, could you send a link to one? Here is a screenshot of what I have now:

Do you have any naming suggestions for the sliders instead of fmin/fmid/fmax? About changing API to PySurfer-like, do you have anything like UML table or something similar with information about high-level description of PySurfer project, so it would take less time for me to understand everything that should be done?

[agramfort]

fmin/fmid/fmax is MNE standard naming convention.

regarding pysurfer I would try to replicate the examples:

https://pysurfer.github.io/auto_examples/index.html

especially

https://pysurfer.github.io/auto_examples/plot_basics.html#sphx-glr-auto-examples-plot-basics-py https://pysurfer.github.io/auto_examples/plot_meg_inverse_solution.html#sphx-glr-auto-examples-plot-meg-inverse-solution-py https://pysurfer.github.io/auto_examples/plot_fmri_activation.html#sphx-glr-auto-examples-plot-fmri-activation-py

[OlehKSS]

Ok, so I will keep naming for sliders like it is now. Is there any way to control color map parameters with PySurfer? I was looking for something similar to the sliders I've done for fmin/fmid/fmax.

[larsoner]

See my comment from a few days ago, use time_viewer=True it will show:

[larsoner]

(turns out they are not sliders but rather text entry boxes)

[larsoner]

and as far as API is concerned, the API page for Brain should hopefully tell you what you need to know:

http://pysurfer.github.io/generated/surfer.Brain.html#surfer.Brain

[OlehKSS]

I have changed sliders to input boxes and changed color map calculation slightly, so it will be more consistent with PySurfer-based plots.

[agramfort]

looks good !

[larsoner]

Does it work for mne / the two-sided colormap, too?

[larsoner]

(If not, properly supporting two-sided data would be the next step)

[larsoner]

FYI PySurfer does not support two-sided data so nicely, so what you create could in principle do a better job

[OlehKSS]

@larsoner I haven't tried two-sided data so far, could you share any examples of it? I guess it should work with some small changes , since I use double-sided material for the overlay, look here.

[larsoner]

Running this with pick_ori='normal' in the apply_inverse call will get you signed data:

https://github.com/mne-tools/mne-python/blob/master/tutorials/plot_mne_dspm_source_localization.py