search

mne-tools / mne-python

MNE: Magnetoencephalography (MEG) and Electroencephalography (EEG) in Python
https://mne.tools
BSD 3-Clause "New" or "Revised" License
2.66k stars 1.31k forks source link

Plotting scatterplots on a 3D surface #2663

Closed choldgraf closed 7 years ago

choldgraf commented 8 years ago

I've been playing around with Freesurfer and the gselu for doing electrode localization in python with ecog grids. It seems to work relatively well, which is really cool.

However, once I've got a FS surface model of a brain, as well as electrode positions, the next step is to use them to make brain plots. The two most common ways the ecog world does this is to

A. Make a scatterplot on top of the brain (where size / color represent your effect and location is the electrode position) B. Create some sort of heatmap on the cortical surface model using the electrode positions

It seems relatively straightforward to figure out B, and if an ecog type is ever added to mne-python I think it'd be a great viz addition. However, I'm not so sure how to do "A" effectively. Right now, I've tried using pysurfer to read my surface model. I've then tried two methods for plotting, which both fail for different reasons:

  1. Using Mayavi to plot a trisurface. This creates the brain beautifully, but the only way to create a scatterplot is to create these "spheres" that are rendered in 3D and look really ugly.
  2. Using matplotlib plot_trisurf to do the same thing. In this case, it creates nice "flat" scatterplots in the 3D axis. However, the call to plot_trisurf takes a really long time, and is unusably slow when it comes to rotating etc.

So I wonder, is there some way to combine the 3D capability of mayavi with the "data plotting" capability of matplotlib? I know that packages like this exist for matlab (e.g., here). These use matlab's version of plot_trisurf, which I guess is more usable. Any thoughts on how to make this work?

agramfort commented 8 years ago

Can you share a figure that shows what you want?

choldgraf commented 8 years ago

Yep - for example, if I plot the brain w/ a call to trisurface and points3d in Mayavi, I get this:

surf.load_geometry()
sx, sy, sz = surf.coords.T  # surface coordinates read by nibabel
x, y, z, s = elecs.values.T  # xyz values of elecs
mysurf = mlab.triangular_mesh(sx, sy, sz, surf.faces, color=(.5, .5, .5))
mlab.points3d(x, y, z, s)

image Really nice brain rendering, crappy looking scatterplot.

If I plot the brain with matplotlib, then I get this:

tri = mpl.tri.Triangulation(sx, sy, triangles=surf.faces)

f, ax = plt.subplots()
ax = Axes3D(f)
# ax.plot_trisurf(sx, sy, sz, surf.faces)  # Commented out because its super slow
# ax.plot_trisurf(tri, sz, color='gray', linewidth=0)  # Same with this one
ax.scatter3D(x, y, z, s=s**2, c='r')

image More standard-looking scatterplot (that I can style etc), but I'm not plotting the brain because doing so makes matplotlib grind to a halt.

I'm hoping to basically have the scatterplot that's shown in the matplotlib case, on top of the brain that's showed in the mayavi case. E.g. I'd like to make plots like this one that is made w/ the above-linked matlab toolbox.

image

agramfort commented 8 years ago

@GaelVaroquaux any idea how to do this with mayavi?

choldgraf commented 8 years ago

Perhaps @aestrivex may have thoughts on this as well.

Been playing around with this all afternoon, haven't found a good way to get matplotlib to do the scatterplot rendering properly (or quickly...it takes about 10-15 seconds on a decently fast computer).

I found that Mayavi actually does have a orient_to_camera capability, but it's only been implemented for Text3D plots. This works kinda the way you'd expect though...kinda. e.g., see the linked movie:

https://www.dropbox.com/s/hbyhera2skpsxpi/text3d_demo.mov?dl=0

There, I've just plotted the string "o" at all the electrode locations and scaled up the size. So maybe it work if there were a scatterplot version of this, but I haven't found such a thing yet...

choldgraf commented 8 years ago

I guess I should also add that mayavi seems really really buggy compared with using Matplotlib. It crashes like once every 10 minutes and isn't able to work with certain plotting backends etc. So maybe if it's possible in mayavi, it's still preferable in matplotlib.

choldgraf commented 8 years ago

Another option I found is to plot electrode positions / the brain in a mayavi 3D scene, and then get the projection of the electrode positions onto the 2D image of it. E.g., following this guide, you can get outputs like this:

(left is the raw Mayavi view after positioning it well, right is a matplotlib imshow + scatter of elec positions) image

That way, you'd just define a "view" for each subject, and then pull the electrode locations on that 2D view. Then, you could do whatever scatterplot routine you'd like.

choldgraf commented 8 years ago

I was looking into this a little bit more, and another person recommended VisPy to me. I looked into it, and while it does have some scatterplot functionality (in their gallery here), it doesn't seem like it's under very active development. @Eric89GXL I noticed you've been working with that project. Any idea if something like a 3-D scatterplot + brain surface is possible in VisPy, or will be sometime soon? (I'm also open to other suggestions too...I'd just really prefer not to have to load matlab every time I want to make a brain plot...)

larsoner commented 8 years ago

VisPy isn't under very active development anymore :(

choldgraf commented 8 years ago

Ah that's a bummer - I thought it was going to be my savior in interactive 3-D plotting in python :-P

agramfort commented 8 years ago

I was hoping the same :( :(

choldgraf commented 7 years ago

Just pinging this thread once again as I had a conversation from another researcher who said that 3D plotting is their main missing link in transitioning from matlab to python for neuro analysis.

Now that Enthought has (theoretically) revived Mayavi once again, do you guys think this will eventually be a viable option for visualizations like the one discussed in this thread? I am a little bearish on plotly as a solution for these problems, so I'm looking at other options. It still looks like VisPy development has died down, so I guess that's not an option. Anything else coming out of the woodwork you all have heard about?

larsoner commented 7 years ago

No new developments AFAIK. Mayavi is under development again, but I haven't seen any reliability improvements at my end.

You could try VisPy -- that would be a mesh with a scatter plot on top. Might not be too terrible to code. Feel free to ping me separately on Gitter (and we'll start a separate room to avoid spamming peopel) if you want to try it. We might be able to get something running with a few minutes of hacking together.

GaelVaroquaux commented 7 years ago

Mayavi is under development again, but I haven't seen any reliability improvements at my end.

IMHO that related to anything that touches graphics cards (hence my lack of love for them). Though the big chunk of C++ that VTK is doesn't really help here :$. The question is whether any powerful 3D visualization tool is not bound to fall in the same trap.

larsoner commented 7 years ago

IMHO that related to anything that touches graphics cards (hence my lack of love for them).

In my experience the graphics cards do contribute some variability and bugs, but...

Though the big chunk of C++ that VTK is doesn't really help here :$.

I don't think Mayavi or graphics cards are most at fault for Mayavi not working, but really the underlying VTK backend (and maybe Traits?):

So I suspect we'll have to wait for VTKPython bugs to be fixed before Mayavi reliability goes up. And given the scope of VTK, I'm not holding my breath. Do you have better luck or see other people have these problems @GaelVaroquaux?

The question is whether any powerful 3D visualization tool is not bound to fall in the same trap.

From working with VisPy, I found that while there were some issues related to graphics cards (ATI oddities, Intel oddities), they were not so common as to be a show-stopper. And Xvfb, which does software rendering, always seemed to work well (used in CI graphical testing/validation). But for VisPy the reliability issues are elsewhere, in the implementation of the elements, as opposed to the lower-level graphics integration through OpenGL. So I'm optimistic that it's possible to get high-quality, consistent, fast 3D graphics in Python through OpenGL, but... we aren't there yet.

@GaelVaroquaux if you're interested in solving this and have some thoughts, we should have a brainstorming session :)

GaelVaroquaux commented 7 years ago

Now that Enthought has (theoretically) revived Mayavi once again,

Mayavi isn't dead and has never been. It's development is slow, and the marketting has been lack-luster, but if you look at the commit timeline, it's been active without long interruption (don't trust before 2011, as it wasn't on github).

GaelVaroquaux commented 7 years ago

@GaelVaroquaux if you're interested in solving this and have some thoughts, we should have a brainstorming session :)

I've given up on this. I see this as a huge loss of time. People want 3D visualization, but it's only a toy, IMHO. And, to me, the failure to find resources to reliably solve it is an indication that it is not mission critical for anything.

It's certainly not the problem that I personnaly want to solve. I don't think that it is a problem that drives scientific progress. @agramfort knows my opinions on that :).

agramfort commented 7 years ago

as you saw on twitter to got a POC alpha alpha stage with pycortex. This basically tells me that it can work very efficiently without huge memory allocated in the browser. It could be a viable solution but the pycortex code is to me not ready for massive dissemination... but it works on my machine (TM) :)

choldgraf commented 7 years ago

Mayavi isn't dead and has never been.

My bad, you are right. Wasn't trying to imply that the project was shut down!

People want 3D visualization, but it's only a toy, IMHO. And, to me, the failure to find resources to reliably solve it is an indication that it is not mission critical for anything.

FWIW, I routinely hear complaints from my (admittedly small) community of ECoG that a big hole in the python pipeline is there aren't good 3D plotting libraries, which is the main reason they still use Matlab in their analyses.

Also to this point, I just heard a talk and chatted a bit with a woman that's one of the project leads for Glue, an interactive multi-dimensional dataviz tool. They're using VizPy under the hood now (thinking of switching to mayavi), and were excited at the prospect of building up surface viz into their interactive viewer. It'd be quite helpful for medical science, I believe.

In case you're curious: https://github.com/glue-viz/glue

as you saw on twitter to got a POC alpha alpha stage with pycortex. This basically tells me that it can work very efficiently without huge memory allocated in the browser.

That was awesome - I've been wanting to do this for a while now, but instead I decided to volunteer to teach a course w/ a bunch of the pycortex devs :P

It could be a viable solution but the pycortex code is to me not ready for massive dissemination... but it works on my machine (TM)

IMO that is very true right now. They're working on it but I haven't seen a huge push from them to stop trying to add features and instead focus on usability, documentation, stability, etc. Maybe that'll come as the bulk of the tool is finalized. Actually, as we're using pycortex in our class it has forced the devs to squash a bunch of bugs. We'll theoretically get it working for a bunch of undergrads, which is like the ultimate "can they break it somehow" bug test ;)

GaelVaroquaux commented 7 years ago

FWIW, I routinely hear complaints from my (admittedly small) community of ECoG that a big hole in the python pipeline is there aren't good 3D plotting libraries, which is the main reason they still use Matlab in their analyses.

People want them, but they are not ready to pay for them. By pay, I mean in many ways: gives jobs to open-source developers who do them, contribute some code, write some grant application to fund them, or actually pay matlab, as these people often pirate matlab.

So, I take such desire as a comment from the peanut gallery. And I say this as someone who has at some point invested a lot to satisfy those desires.

choldgraf commented 7 years ago

+1 to what you said, I think you are correct. On the "pay" side, it sounds like the glue project might have a good shot at getting a fair amount of funding for their project (because of the medical applications) and I wonder if they'll hire devs to contribute to these projects, since their interactive 3D thingamajig is only as good as the underlying 3D plotting library. (just thinking out loud here, not suggesting that anybody on this thread should work with that project)

pelednoam commented 7 years ago

We're are working on a new neuroimaging 3D visualization tool, that is based on Blender + python 3.5. It's mainly based on Freesurfer recon-all outputs and mne-python. The tool supports MEG/EEG, fMRI, PET and iEEG (ECoG): https://github.com/pelednoam/mmvt It's still in alpha, and we are looking for brave alpha users :)

pelednoam commented 7 years ago

Here is an ECoG example, where each electrode is colored according to its most probable ROI:

ecog

choldgraf commented 7 years ago

@pelednoam that's a pretty brain! How is the Blender performance? I've played around w/ using Blender with some pycortex folks, and it seemed kinda kludgy to get working easily and quickly. Either way, it looks like a very neat tool. You might want to speak with either the Glue folks, or @aestrivex, as they've both worked on similar projects

choldgraf commented 7 years ago

Also FWIW, I played around with Mayavi and (at least for me) it seems more stable than it did before (aka, it hasn't crashed on my computer yet). I threw together some quick functions to let you pull an image from a particular view of a brain in mayavi, and then store x/y positions of a scatterplot on top of that screen. E.g., see below. Not too bad, actually!

3d_2d

pelednoam commented 7 years ago

@choldgraf Thanks! Blender performance is generally quite good, depending on what you want to do with it. We wrap it in a way the user doesn't need to be familiar with Blender at all (well, almost). We also changed the layout and added quite a lot of panels and buttons. Currently, we are working on the python interface, to get similar behaviour like pysurfer, in case you don't want to use the interactive GUI. One of our goals is the let mne-python users the option to produce figures and videos without being bound to python 2 and Mayavi. And your Mayavi example really looks good!

choldgraf commented 7 years ago

I updated the ecogtools package to use Mayavi plotting etc under the hood.

Sample image: image E.g.: here's a gif

@kingjr let me know what you think about that (or anybody else that does ECoG plotting). I think Mayavi is a pretty decent solution at this point. Still has some work to do tho.

@pelednoam I'm looking forward to seeing progress on your end! Blender definitely makes nicer renderings of brains and as long as it has trisurf-style inputs and lets you do things like return camera parameters, it won't be a ton of work to change the backend (I think).

Also, I apologize to @GaelVaroquaux for suggesting that Mayavi was dead ;)

GaelVaroquaux commented 7 years ago

Excellent reference!

The animated gif is pretty cool.

By the way, I'd be curious, if you have a recent version of Mayavi and Jupyter, if what you showed us works decently with the embedded viewer (which will loose interactivity and likely be slow): http://blog.enthought.com/general/mayavi-python-3d-data-visualization-and-plotting-library-adds-major-new-features-in-recent-release

choldgraf commented 7 years ago

I just gave it a shot and, while it works, it's pretty sloooow and a little cludgy. It seems like MayaviScene objects are handled differently with inline plotting, so my code for projecting 3D points onto the camera etc didn't work, and there doesn't appear to be a figure object created in the same way. However it is possible to get it to work:

(note that it looks frozen near the end but that's just because it takes a while for the plot to render) 3d_2d_int

Promising for an early version of this feature!

GaelVaroquaux commented 7 years ago

It won't get faster though. I believe that it's a limitation of the client side rendering library. I was indeed afraid that it would be slow.

⁣Sent from my phone. Please forgive brevity and mis spelling​

On Nov 1, 2016, 19:48, at 19:48, Chris Holdgraf notifications@github.com wrote:

I just gave it a shot and, while it works, it's pretty sloooow and a little cludgy. It seems like MayaviScene objects are handled differently with inline plotting, so my code for projecting 3D points onto the camera etc didn't work, and there doesn't appear to be a figure object created in the same way. However it is possible to get it to work:

(note that it looks frozen near the end but that's just because it takes a while for the plot to render) 3d_2d_int

Promising for an early version of this feature!

You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/mne-tools/mne-python/issues/2663#issuecomment-257657893

choldgraf commented 7 years ago

Ah I see. Is there an issue in mayavi somewhere in which this is discussed?

GaelVaroquaux commented 7 years ago

Ah I see. Is there an issue in mayavi somewhere in which this is discussed?

Not that I know. You can open one.

choldgraf commented 7 years ago

@agramfort and @Eric89GXL , can one of you guys run me quickly through how MRI / surface data is represented in MNE-python? If it's relatively straightforward to, e.g., load the surface file for a subject along with 3D channel coordinates for it (e.g. in a Montage object?), then I could commit some of this "2D snapshot" stuff into MNE using Mayavi under the hood. I think it'd be useful now that we have an ecog channel type.

If you guys are friendly to that then I can actually close this issue at some point since I think the more general problem of a 3D scatterplot is probably more of a Mayavi / VizPy / whatever challenge than an MNE challenge.

agramfort commented 7 years ago

you mean how coordinate systems are handled?

choldgraf commented 7 years ago

I guess I'm mostly wondering whether it's possible to package arbitrary 3d sensor information along with subject-specific surface information in Raw objects (and derivations of them).

E.g., it seems like Montage objects assume some kind of EEG-style layout. When I try to create an info object using ecog electrodes, I get an error that no EEG electrodes were found, and if I call plot_sensors after setting the channels to EEG, it assumes a sensor position on a standard top-down EEG view. It doesn't seem like it'd be too hard to modify this to plot, e.g., 2d positions on a view of the person's brain if the ch_type is ecog.

This example maybe is a good start. It seems to have some concept of sources in 3d space, as well as a surface of that subject's brain. I think I could use whatever this is using under the hood in order to create some kind of ECoG electrode plotter. However I'm finding it challenging to parse these docs because I have no background in MEG so it's unclear how much of it relies on MEG machinery vs. how much is just general "plotting stuff on a brain surface" code.

Sorry if that is a bit incoherent, I can just try to go through the code more closely to see how it's being accomplished.

larsoner commented 7 years ago

We should just improve plot_trans. It already does most of what you want. Do you have electrode positions in MRI coordinates in info ['chs']? If so it's probably 5 changed lines to get what you want.

choldgraf commented 7 years ago

ya I'm looking at the code now and it looks like there wouldn't be too much to change. Ya, I have electrodes in the same space as a freesurfer surface (see the GIF above). That was made with a bunch of code I wrote but I bet you could accomplish something similar with minimal extra lines in MNE. Could even update the plot_sensors function to accept the string "image" and make a plot with 2D locations. Though that might require changes to the info object so might not be OK.

larsoner commented 7 years ago

You shouldn't need to change the info object. What do you want to accomplish? Project on some view other than top down? We could add am option for that in the function, it would just require applying a rotation before projection. Something like view=(az, pol) would do it.

choldgraf commented 7 years ago

want to chat about this quickly on gitter? so we aren't spamming everyone...

choldgraf commented 7 years ago

The main thing I'd like to do is:

  1. Make a surface plot of electrodes on a freesurfer-style surface.
  2. Pick a "view" of the camera on that surface, then take a snapshot of the view, and the 2D location of each electrode in that view.
  3. Plot scatterplots on top of the view where each point is an electrode.
  4. Optionally also plot foci of activity on the surface itself in Mayavi (or some other tool but I haven't found a good way to do that)

Basically all the stuff in the gif that I showed above. I've got that code working in my own repo but I'm wondering if it would be useful and relatively low-cost to incorporate some of that into MNE (e.g., if you call plot_sensors with ecog channel types, it does something different than just plotting them on top of an EEG style topo plot.)

larsoner commented 7 years ago

@choldgraf I think we have this now from your PRs, so I'll close. Feel free to reopen if I'm mistaken