Mesh overlay with activation data

[OlehKSS]

@larsoner I am not sure which color map to use. In your gist you were using RdYlBuCy, but I have found the same in matplotlib, the most similar is RdYlBu. In the gist from GSOC wiki hot color map was used. I have also tried RdBu_r, and it makes image blue, in contrast to previous two. Which should I stick to?

RdYlBuCy color map

Hot color map

[larsoner]

There are some somewhat annoying considerations about colormaps regarding control points and transparency that are handled already in MNE. These data are all positive, right? If so, for now just use hot, ensuring that the values in the lower half of the colormap go smoothly from transparent (at the lowest value) to opaque (at the middle value, and middle-highest value is also opaque).

If the data are negative and positive, then look at the code for mne_analyze_colormap -- basically assuming you want colormap limits from -1 to 1, values go linearly from opaque cyan (-1) to opaque blue (-0.5) to transparent gray (0) to opaque red (0.5) to opaque yellow (1.0).

[OlehKSS]

I have tried to apply your suggestions to my code, @larsoner , and I have faced several issues.

First of all, mne_analyze_colormap function either behaves odd or I don't understand that type of color map. I was trying to generate a matplotlib color map, with input range [-1, 1]. The problem is that output color map gives different output, when I assumed that output is the same. For example, if I provide [1, 1] list as input and [0, 0] I will receive the same colors as output, i. e. [0, 1, 1, 1]. I thought it would behave similar to the ReBu_r color map from matplotlib. I thought it would be possible to use this function for color map generation.

Second problem, I could not find any good combination of blending for the color map ReBu_r, since in the middle it has white color. From threejs examples page, it seams like much easier will be to have change from blue to black and from black to red. It seams to be slightly different from your suggestion about using transparent grey at 0, and opaque color (red or blue) at the extremes. It seams like I would need to use opaque black, when I have no signal, and transparency will be obtained by the color blending.

And last one, do you have any way of eliminating noise from the source estimates data? I had the minimal signal value of -3^(-8), and maximum around 24. I was trying to check signal signs in order to decide which color map (e. g. "hot" or "RdBu_r").

[larsoner]

First of all, mne_analyze_colormap function either behaves odd or I don't understand that type of color map. I was trying to generate a matplotlib color map, with input range [-1, 1].

matplotlib wants inputs to be normalized to [0, 255] so you can see the bounds / behavior I mention above (opaque cyan -> opaque blue -> transparent gray -> opaque red -> opaque yellow), with the exception that the alpha channel is wrong:

>>> cmap = mne.viz.mne_analyze_colormap([0, 0.5, 1], format='matplotlib')
>>> cmap(255 * 0.)
(0.0, 1.0, 1.0, 1.0)
>>> cmap(255 * 0.25)
(1.0, 1.0, 0.0, 1.0)
>>> cmap(255 * 0.5)
(1.0, 1.0, 0.0, 1.0)
>>> cmap(255 * 0.75)
(1.0, 1.0, 0.0, 1.0)
>>> cmap(255 * 1.0)
(1.0, 1.0, 0.0, 1.0)

The weird thing is that when you tell mne_analyze_colormap to use [0, 0.5, 1], it really means "treat it like [-1, -0.5, 0, 0.5, 1]" because it enforces symmetry. Then it's complicated by the fact that matplotlib wants all colormap inputs to be in the range [0, 255]. But it's possible that ipyvolume uses some other normalization scheme.

Don't agonize too much over these matplotlib details. Instead -- do you understand what I'm saying the mne colormap should do given some data? Can you get these colors (with alpha channel) into whatever format ipyvolume needs them to be for display?

do you have any way of eliminating noise from the source estimates data? I had the minimal signal value of -3^(-8), and maximum around 24. I was trying to check signal signs in order to decide which color map (e. g. "hot" or "RdBu_r").

We already have code for this in MNE that we can use/take/adapt. Let's focus on getting colormaps with alpha channels working now, i.e., getting custom LUTs/colormaps with alpha channels into ipyvolume.

It seams like I would need to use opaque black, when I have no signal, and transparency will be obtained by the color blending.

I am confused -- if the middle black (or gray) is opaque, there will be no alpha blending. The entire brain surface will be only black, red, or blue -- no gray will show. If you are talking about an overall "object alpha" or "object opacity" for the overlay mesh that is less than zero, that is not what we want / are talking about here (though it could be a complementary option). The overlay colormap/lookup table should be completely opaque in some places (the extreme ends for symmetric cmaps) and completely transparent in others (in the middle for symmetric cmaps), at least by some option (probably transparent=True).

For divergent/one-sided colormaps like hot, the transparency would basically be from the [0, 0.5] point. Try running PySurfer examples with the hot colormap for transparent=True and transparent=False to see what I mean.

[OlehKSS]

I haven't seen an explicit mention of transparent color availability for the vertices for the ipyvolume, so I am not sure whether ipyvolume supports it. So I provided colors without alpha channel, rgb only. For that case I was panning to do color blending, not color alpha blending, e. g. additive color blending. That's why black will be helpful, for the additive blending black color regions will have no effect on the grey surface.

So, ipyvolume supports alpha channel after all, doesn't it?

From what I've seen in matplolib documentation, they seem to use [-1, 1] or [0, 1] normalization for the color maps, so I was expecting the same from mne_analyze_colormap.

[larsoner]

Can you investigate if alpha is possible?

[OlehKSS]

Yes

[OlehKSS]

Threejs seems to support Alpha blending, since for the blending factors they have OneMinusSrcAlphaFactor, SrcAlphaFactor, CustomBlendingEquations. I tried to use this blending for the brain mesh, but it fails. For the WebGL Renderer they have an option for enabling transparency buffer, bu default it is disabled. It might be a problem, but I have run out of ideas how to check how ipyvolume handles this configuration.

[larsoner]

Can you open an ipyvolume issue to see if it's something that can be added? This might need to be your first PR to ipyvolume :)

[OlehKSS]

Here is a link to the error

[OlehKSS]

According to this pull request three.js and webgl don't support alpha channel in colors at all, three.js just silently remove alpha component.

[larsoner]

Read through this:

https://github.com/mrdoob/three.js/issues/2730

It sounds like the workaround is to use a custom Vector4 attribute, which you would map to the color component in the shader. This will need to be done at the ipyvolume end

[OlehKSS]

I looked through that issue. As it seams from the discussion vecotor4 is not an ideal solution, but we can try to do it. Well, so far I have no idea how it can be implemented, I will need some time to figure it out.

[larsoner]

I have no idea how it can be implemented, I will need some time to figure it out.

This idea came up here as making some sense:

https://github.com/maartenbreddels/ipyvolume/pull/132#issuecomment-389977350

So the fastest way to figure it out is comment over in https://github.com/maartenbreddels/ipyvolume/issues/138 to explain the situation and solicit suggestions from Maarten

[larsoner]

(he is not in this discussion, for example, so he will not be aware that three.js explicitly does not allow alpha to vary on a per-vertex level and the suggested workaround is to use Vector4)

[OlehKSS]

Yep, thanks. I have explained the situation in that issue

[OlehKSS]

I am working now on transparency for the ipyvolume (with changing their shaders), should I create a new issue for that or should I continue to work in the scope of this?

[agramfort]

pushing a fix upstream in ipyvolume ?

[OlehKSS]

Yes, I plan to do it like that, but it will take some time. So far I am getting around in their code, debugging it, I am studying how WebGL works with shaders.

[larsoner]

Just work there and ping us with @ when you open a PR or need help

[OlehKSS]

I've added a pull request with draft changes in the Ipyvolume repository, please review.

[OlehKSS]

I have updated code and changed color blending to alpha blending, it requires my changes for ipyvolume to be merged.

Inflated surface with activation data

[agramfort]

maybe a bit too much transparency but looks really nice !

[OlehKSS]

I was using linearly spaced transparency values, 256 entries in the range [0, 1). I might shift values toward zero or use some non-linearly spaced range.

[larsoner]

If you want to get into colormap handling, we already have code in MNE to automatically choose limits in mne/viz/_3d.py, and then look at how PySurfer handles the mapping to 256 colors step (it's also done for Mayavi0

[OlehKSS]

Thank you for the reference, I am analyzing the file now. Well, that's quite a large file, do you happen to remember where or in which function you've been doing color map handling?

[larsoner]

Study how _limits_to_control_points is used, and how these values get passed to PySurfer, which converts them to a colormap (with bounds):

https://github.com/mne-tools/mne-python/blob/master/mne/viz/_3d.py#L1659

[OlehKSS]

There is a problem with _limits_to_control_points. It does not seem to give correct alpha values if I set transparent=True, and, as far as I see, it is due to the reliance on the mne_analyze_colormap. I have not found any issue concerning this. I can set alpha values on my own. I assume that the shades of grey on mne color map should be fully transparent, shouldn't they? Should the same be applied to the hot color map using control points that I obtain from _limits_to_control_points.

[larsoner]

Indeed, you should not use transparent=True with the mne colormap. transparent should only be used with one-sided colormaps that do not provide their own alpha like hot, not with two-sided colormaps that have alpha built in like mne.

I assume that the shades of grey on mne color map should be fully transparent, shouldn't they?

Yes it has essentially 6 control points based on what the user passes as the 3-tuple pos_lims that ends up symmetric about a zero: opaque cyan (-pos_lims[2]) -> opaque blue (-pos_lims[1]) -> transparent gray (-pos_lims[0]) -> transparent gray (pos_lims[0]) -> opaque red (pos_lims[1]) -> opaque yellow (pos_lims[2]). You can see if pos_lims[0] == 0 this essentially becomes 5 control points. Play with stc.plot with signed data and mne colormap with pos_lims values and you should be able to confirm this behavior.

[OlehKSS]

@larsoner I have implemented transparency calculations for the mne color map as you suggested. Which subject data can I use to show you the result? The problem with sample subject is that the negative values are similar to noise (3E-8) and as a result I obtain a fully yellow mesh, since positive values, e. g. 20, are getting to the other spectrum extreme (something like saturation). Fig. 1. Saturation due to noisy data, mne color map was used.*

I have also a question regarding 'hot' color map usage. So, as output of _limits_to_control_points from mne I obtain these array of control points [ 8.54465169, 9.98478648, 20.2285404 ]. I set transparency to 0 for all the data values that are below first control point, i. e. 8.545. All the points above the first control point will have transparency set to 1. Is it correct approach? This is an output of what I obtain for sample subject with the algorithm mentioned above. Fig. 2 hot color map, values below first control point (8.545) are fully transparent

[OlehKSS]

By the way, my changes to the ipyvolume have been merged. I have also added a new pull request to that repository with styling changes to several files that I have been working with previously.

[larsoner]

Which subject data can I use to show you the result? The problem with sample subject is that the negative values are similar to noise (3*E-8) and as a result I obtain a fully yellow mesh, since positive values, e. g. 20, are getting to the other spectrum extreme (something like saturation).

It sounds like the data you are working with are not signed. You need signed data. So some output of apply_inverse(..., pick_ori='normal') would work, you could use the values here with that change from None to 'normal':

https://martinos.org/mne/stable/auto_tutorials/plot_mne_dspm_source_localization.html#compute-inverse-solution

I set transparency to 0 for all the data values that are below first control point, i. e. 8.545. All the points above the first control point will have transparency set to 1. Is it correct approach?

Not quite -- below fmin (8.545) are transparent, from fmin to fmid (9.98) they should linearly increase in alpha 0->1, and from fmid to fmax (20.22) they should stay opaque (alpha 1).

[OlehKSS]

I have made changes to the transparency calculation in case of the "hot" color map. Here are several examples of how alpha blending will look like for different color maps and depending on data:

'Hot' color map

'MNE' color map, positive data

'MNE' color map, negative data

[agramfort]

it seems like to sharp color transition no?

[OlehKSS]

@agramfort Are you talking about examples with 'mne' color map? Do you mean that for this color map transition between transparent-opaque colors should be added? For the 'mne' color map I use color map object returned by _limits_to_control_points function and I have implemented transparency as was suggested in the comment above: opaque cyan (-pos_lims[2]) -> opaque blue (-pos_lims[1]) -> transparent gray (-pos_lims[0]) -> transparent gray (pos_lims[0]) -> opaque red (pos_lims[1]) -> opaque yellow (pos_lims[2]). So, between -pos_lims[1] and pos_lims[1] I have transparent gray and colors (red, yellow, blue, cyan) are opaque.

[agramfort]

can you put next to each other the pysurfer/mayavi figure and yours so we can compare?

[larsoner]

For some reason it looks like each "blob" has a white border. It should instead smoothly transition to the curv surface colors.

[OlehKSS]

I had white blobs appear before, while testing alpha blending with ipyvolume. So, I suppose that it might be caused by the way GLSL vertex/texture shaders work.

[larsoner]

So, I suppose that it might be caused by the way GLSL vertex/texture shaders work.

The weird thing is that it's not on every blob, though. The "hot" colormap orange blobs have no white borders, and you can see the curv shading beneath it. So perhaps it's a problem with the colormap definition?

[OlehKSS]

I haven't been using color maps before, and still had the white borders. 'hot' color map has smooth transition of alphas from 0 to 1, so it might be the case. This is an example that I have implemented without any color map, just plain blue color assigned to some of the points. While blob surrounds areas here as well. Example of alpha blending, without color maps

[larsoner]

'hot' color map has smooth transition of alphas from 0 to 1

The MNE colormap should also have smooth (linear) transitions from 0 to 1. You coded it that way, right?

[OlehKSS]

Hm, I'll need to do it like that, smooth transition should be from control point 0 to control point 1, and from control point 1 to control point 2 it should be opaque?

[larsoner]

Yes all transitions (color and alpha) are linear from alpha=1 cyan to alpha=1 blue to alpha=0 gray to alpha=1 red to alpha=1 yellow in the MNE colormap

[OlehKSS]

I have added linear transition of transparency for the 'MNE' color map as well. For fixing white glowing I have also made changes to the WebGL configurations of alpha blending in ipyvolume. This is an outcome:

'MNE' color map positive and negative data visualization

'HOT' color map

[larsoner]

Great, those do indeed look better. (For those looking through GitHub emails, the post has been edited and the images now look correct.)

Perhaps the next step is that we actually review the code and merge this one?

[OlehKSS]

The code is ready to be reviewed.

[OlehKSS]

@larsoner I have made changes according to the code review suggestions, I have also opened a separate pull request to ipyvolume, it will fix problems with white glowing on the borders. I think we need to make updates to the agenda.

[OlehKSS]

@larsoner @choldgraf should I merge this pull request?

[choldgraf]

woo, way to go @OlehKSS !