Closed choldgraf closed 7 years ago
Can you share a figure that shows what you want?
Yep - for example, if I plot the brain w/ a call to trisurface
and points3d
in Mayavi, I get this:
surf.load_geometry()
sx, sy, sz = surf.coords.T # surface coordinates read by nibabel
x, y, z, s = elecs.values.T # xyz values of elecs
mysurf = mlab.triangular_mesh(sx, sy, sz, surf.faces, color=(.5, .5, .5))
mlab.points3d(x, y, z, s)
Really nice brain rendering, crappy looking scatterplot.
If I plot the brain with matplotlib, then I get this:
tri = mpl.tri.Triangulation(sx, sy, triangles=surf.faces)
f, ax = plt.subplots()
ax = Axes3D(f)
# ax.plot_trisurf(sx, sy, sz, surf.faces) # Commented out because its super slow
# ax.plot_trisurf(tri, sz, color='gray', linewidth=0) # Same with this one
ax.scatter3D(x, y, z, s=s**2, c='r')
More standard-looking scatterplot (that I can style etc), but I'm not plotting the brain because doing so makes matplotlib grind to a halt.
I'm hoping to basically have the scatterplot that's shown in the matplotlib case, on top of the brain that's showed in the mayavi case. E.g. I'd like to make plots like this one that is made w/ the above-linked matlab toolbox.
@GaelVaroquaux any idea how to do this with mayavi?
Perhaps @aestrivex may have thoughts on this as well.
Been playing around with this all afternoon, haven't found a good way to get matplotlib to do the scatterplot rendering properly (or quickly...it takes about 10-15 seconds on a decently fast computer).
I found that Mayavi actually does have a orient_to_camera
capability, but it's only been implemented for Text3D plots. This works kinda the way you'd expect though...kinda. e.g., see the linked movie:
https://www.dropbox.com/s/hbyhera2skpsxpi/text3d_demo.mov?dl=0
There, I've just plotted the string "o" at all the electrode locations and scaled up the size. So maybe it work if there were a scatterplot version of this, but I haven't found such a thing yet...
I guess I should also add that mayavi seems really really buggy compared with using Matplotlib. It crashes like once every 10 minutes and isn't able to work with certain plotting backends etc. So maybe if it's possible in mayavi, it's still preferable in matplotlib.
Another option I found is to plot electrode positions / the brain in a mayavi 3D scene, and then get the projection of the electrode positions onto the 2D image of it. E.g., following this guide, you can get outputs like this:
(left is the raw Mayavi view after positioning it well, right is a matplotlib imshow + scatter of elec positions)
That way, you'd just define a "view" for each subject, and then pull the electrode locations on that 2D view. Then, you could do whatever scatterplot routine you'd like.
I was looking into this a little bit more, and another person recommended VisPy to me. I looked into it, and while it does have some scatterplot functionality (in their gallery here), it doesn't seem like it's under very active development. @Eric89GXL I noticed you've been working with that project. Any idea if something like a 3-D scatterplot + brain surface is possible in VisPy, or will be sometime soon? (I'm also open to other suggestions too...I'd just really prefer not to have to load matlab every time I want to make a brain plot...)
VisPy isn't under very active development anymore :(
Ah that's a bummer - I thought it was going to be my savior in interactive 3-D plotting in python :-P
I was hoping the same :( :(
Just pinging this thread once again as I had a conversation from another researcher who said that 3D plotting is their main missing link in transitioning from matlab to python for neuro analysis.
Now that Enthought has (theoretically) revived Mayavi once again, do you guys think this will eventually be a viable option for visualizations like the one discussed in this thread? I am a little bearish on plotly as a solution for these problems, so I'm looking at other options. It still looks like VisPy development has died down, so I guess that's not an option. Anything else coming out of the woodwork you all have heard about?
No new developments AFAIK. Mayavi is under development again, but I haven't seen any reliability improvements at my end.
You could try VisPy -- that would be a mesh with a scatter plot on top. Might not be too terrible to code. Feel free to ping me separately on Gitter (and we'll start a separate room to avoid spamming peopel) if you want to try it. We might be able to get something running with a few minutes of hacking together.
Mayavi is under development again, but I haven't seen any reliability improvements at my end.
IMHO that related to anything that touches graphics cards (hence my lack of love for them). Though the big chunk of C++ that VTK is doesn't really help here :$. The question is whether any powerful 3D visualization tool is not bound to fall in the same trap.
IMHO that related to anything that touches graphics cards (hence my lack of love for them).
In my experience the graphics cards do contribute some variability and bugs, but...
Though the big chunk of C++ that VTK is doesn't really help here :$.
I don't think Mayavi or graphics cards are most at fault for Mayavi not working, but really the underlying VTK backend (and maybe Traits?):
scipy.linalg
functions will just return junk (so things are no longer reliable/usable) or the code will just failSo I suspect we'll have to wait for VTKPython bugs to be fixed before Mayavi reliability goes up. And given the scope of VTK, I'm not holding my breath. Do you have better luck or see other people have these problems @GaelVaroquaux?
The question is whether any powerful 3D visualization tool is not bound to fall in the same trap.
From working with VisPy, I found that while there were some issues related to graphics cards (ATI oddities, Intel oddities), they were not so common as to be a show-stopper. And Xvfb, which does software rendering, always seemed to work well (used in CI graphical testing/validation). But for VisPy the reliability issues are elsewhere, in the implementation of the elements, as opposed to the lower-level graphics integration through OpenGL. So I'm optimistic that it's possible to get high-quality, consistent, fast 3D graphics in Python through OpenGL, but... we aren't there yet.
@GaelVaroquaux if you're interested in solving this and have some thoughts, we should have a brainstorming session :)
Now that Enthought has (theoretically) revived Mayavi once again,
Mayavi isn't dead and has never been. It's development is slow, and the marketting has been lack-luster, but if you look at the commit timeline, it's been active without long interruption (don't trust before 2011, as it wasn't on github).
@GaelVaroquaux if you're interested in solving this and have some thoughts, we should have a brainstorming session :)
I've given up on this. I see this as a huge loss of time. People want 3D visualization, but it's only a toy, IMHO. And, to me, the failure to find resources to reliably solve it is an indication that it is not mission critical for anything.
It's certainly not the problem that I personnaly want to solve. I don't think that it is a problem that drives scientific progress. @agramfort knows my opinions on that :).
as you saw on twitter to got a POC alpha alpha stage with pycortex. This basically tells me that it can work very efficiently without huge memory allocated in the browser. It could be a viable solution but the pycortex code is to me not ready for massive dissemination... but it works on my machine (TM) :)
Mayavi isn't dead and has never been.
My bad, you are right. Wasn't trying to imply that the project was shut down!
People want 3D visualization, but it's only a toy, IMHO. And, to me, the failure to find resources to reliably solve it is an indication that it is not mission critical for anything.
FWIW, I routinely hear complaints from my (admittedly small) community of ECoG that a big hole in the python pipeline is there aren't good 3D plotting libraries, which is the main reason they still use Matlab in their analyses.
Also to this point, I just heard a talk and chatted a bit with a woman that's one of the project leads for Glue, an interactive multi-dimensional dataviz tool. They're using VizPy under the hood now (thinking of switching to mayavi), and were excited at the prospect of building up surface viz into their interactive viewer. It'd be quite helpful for medical science, I believe.
In case you're curious: https://github.com/glue-viz/glue
as you saw on twitter to got a POC alpha alpha stage with pycortex. This basically tells me that it can work very efficiently without huge memory allocated in the browser.
That was awesome - I've been wanting to do this for a while now, but instead I decided to volunteer to teach a course w/ a bunch of the pycortex devs :P
It could be a viable solution but the pycortex code is to me not ready for massive dissemination... but it works on my machine (TM)
IMO that is very true right now. They're working on it but I haven't seen a huge push from them to stop trying to add features and instead focus on usability, documentation, stability, etc. Maybe that'll come as the bulk of the tool is finalized. Actually, as we're using pycortex in our class it has forced the devs to squash a bunch of bugs. We'll theoretically get it working for a bunch of undergrads, which is like the ultimate "can they break it somehow" bug test ;)
FWIW, I routinely hear complaints from my (admittedly small) community of ECoG that a big hole in the python pipeline is there aren't good 3D plotting libraries, which is the main reason they still use Matlab in their analyses.
People want them, but they are not ready to pay for them. By pay, I mean in many ways: gives jobs to open-source developers who do them, contribute some code, write some grant application to fund them, or actually pay matlab, as these people often pirate matlab.
So, I take such desire as a comment from the peanut gallery. And I say this as someone who has at some point invested a lot to satisfy those desires.
+1 to what you said, I think you are correct. On the "pay" side, it sounds like the glue project might have a good shot at getting a fair amount of funding for their project (because of the medical applications) and I wonder if they'll hire devs to contribute to these projects, since their interactive 3D thingamajig is only as good as the underlying 3D plotting library. (just thinking out loud here, not suggesting that anybody on this thread should work with that project)
We're are working on a new neuroimaging 3D visualization tool, that is based on Blender + python 3.5. It's mainly based on Freesurfer recon-all outputs and mne-python. The tool supports MEG/EEG, fMRI, PET and iEEG (ECoG): https://github.com/pelednoam/mmvt It's still in alpha, and we are looking for brave alpha users :)
Here is an ECoG example, where each electrode is colored according to its most probable ROI:
@pelednoam that's a pretty brain! How is the Blender performance? I've played around w/ using Blender with some pycortex folks, and it seemed kinda kludgy to get working easily and quickly. Either way, it looks like a very neat tool. You might want to speak with either the Glue folks, or @aestrivex, as they've both worked on similar projects
Also FWIW, I played around with Mayavi and (at least for me) it seems more stable than it did before (aka, it hasn't crashed on my computer yet). I threw together some quick functions to let you pull an image from a particular view of a brain in mayavi, and then store x/y positions of a scatterplot on top of that screen. E.g., see below. Not too bad, actually!
@choldgraf Thanks! Blender performance is generally quite good, depending on what you want to do with it. We wrap it in a way the user doesn't need to be familiar with Blender at all (well, almost). We also changed the layout and added quite a lot of panels and buttons. Currently, we are working on the python interface, to get similar behaviour like pysurfer, in case you don't want to use the interactive GUI. One of our goals is the let mne-python users the option to produce figures and videos without being bound to python 2 and Mayavi. And your Mayavi example really looks good!
I updated the ecogtools
package to use Mayavi plotting etc under the hood.
Sample image:
E.g.: here's a gif
@kingjr let me know what you think about that (or anybody else that does ECoG plotting). I think Mayavi is a pretty decent solution at this point. Still has some work to do tho.
@pelednoam I'm looking forward to seeing progress on your end! Blender definitely makes nicer renderings of brains and as long as it has trisurf-style inputs and lets you do things like return camera parameters, it won't be a ton of work to change the backend (I think).
Also, I apologize to @GaelVaroquaux for suggesting that Mayavi was dead ;)
Excellent reference!
The animated gif is pretty cool.
By the way, I'd be curious, if you have a recent version of Mayavi and Jupyter, if what you showed us works decently with the embedded viewer (which will loose interactivity and likely be slow): http://blog.enthought.com/general/mayavi-python-3d-data-visualization-and-plotting-library-adds-major-new-features-in-recent-release
I just gave it a shot and, while it works, it's pretty sloooow and a little cludgy. It seems like MayaviScene
objects are handled differently with inline plotting, so my code for projecting 3D points onto the camera etc didn't work, and there doesn't appear to be a figure
object created in the same way. However it is possible to get it to work:
(note that it looks frozen near the end but that's just because it takes a while for the plot to render)
Promising for an early version of this feature!
It won't get faster though. I believe that it's a limitation of the client side rendering library. I was indeed afraid that it would be slow.
Sent from my phone. Please forgive brevity and mis spelling
On Nov 1, 2016, 19:48, at 19:48, Chris Holdgraf notifications@github.com wrote:
I just gave it a shot and, while it works, it's pretty sloooow and a little cludgy. It seems like
MayaviScene
objects are handled differently with inline plotting, so my code for projecting 3D points onto the camera etc didn't work, and there doesn't appear to be afigure
object created in the same way. However it is possible to get it to work:(note that it looks frozen near the end but that's just because it takes a while for the plot to render)
Promising for an early version of this feature!
You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/mne-tools/mne-python/issues/2663#issuecomment-257657893
Ah I see. Is there an issue in mayavi somewhere in which this is discussed?
Ah I see. Is there an issue in mayavi somewhere in which this is discussed?
Not that I know. You can open one.
@agramfort and @Eric89GXL , can one of you guys run me quickly through how MRI / surface data is represented in MNE-python? If it's relatively straightforward to, e.g., load the surface file for a subject along with 3D channel coordinates for it (e.g. in a Montage
object?), then I could commit some of this "2D snapshot" stuff into MNE using Mayavi under the hood. I think it'd be useful now that we have an ecog
channel type.
If you guys are friendly to that then I can actually close this issue at some point since I think the more general problem of a 3D scatterplot is probably more of a Mayavi / VizPy / whatever challenge than an MNE challenge.
you mean how coordinate systems are handled?
I guess I'm mostly wondering whether it's possible to package arbitrary 3d sensor information along with subject-specific surface information in Raw
objects (and derivations of them).
E.g., it seems like Montage
objects assume some kind of EEG-style layout. When I try to create an info
object using ecog
electrodes, I get an error that no EEG electrodes were found, and if I call plot_sensors
after setting the channels to EEG, it assumes a sensor position on a standard top-down EEG view. It doesn't seem like it'd be too hard to modify this to plot, e.g., 2d positions on a view of the person's brain if the ch_type is ecog
.
This example maybe is a good start. It seems to have some concept of sources in 3d space, as well as a surface of that subject's brain. I think I could use whatever this is using under the hood in order to create some kind of ECoG electrode plotter. However I'm finding it challenging to parse these docs because I have no background in MEG so it's unclear how much of it relies on MEG machinery vs. how much is just general "plotting stuff on a brain surface" code.
Sorry if that is a bit incoherent, I can just try to go through the code more closely to see how it's being accomplished.
We should just improve plot_trans. It already does most of what you want. Do you have electrode positions in MRI coordinates in info ['chs']? If so it's probably 5 changed lines to get what you want.
ya I'm looking at the code now and it looks like there wouldn't be too much to change. Ya, I have electrodes in the same space as a freesurfer surface (see the GIF above). That was made with a bunch of code I wrote but I bet you could accomplish something similar with minimal extra lines in MNE. Could even update the plot_sensors
function to accept the string "image"
and make a plot with 2D locations. Though that might require changes to the info
object so might not be OK.
You shouldn't need to change the info object. What do you want to accomplish? Project on some view other than top down? We could add am option for that in the function, it would just require applying a rotation before projection. Something like view=(az, pol) would do it.
want to chat about this quickly on gitter? so we aren't spamming everyone...
The main thing I'd like to do is:
Basically all the stuff in the gif that I showed above. I've got that code working in my own repo but I'm wondering if it would be useful and relatively low-cost to incorporate some of that into MNE (e.g., if you call plot_sensors
with ecog
channel types, it does something different than just plotting them on top of an EEG style topo plot.)
@choldgraf I think we have this now from your PRs, so I'll close. Feel free to reopen if I'm mistaken
I've been playing around with Freesurfer and the gselu for doing electrode localization in python with ecog grids. It seems to work relatively well, which is really cool.
However, once I've got a FS surface model of a brain, as well as electrode positions, the next step is to use them to make brain plots. The two most common ways the ecog world does this is to
A. Make a scatterplot on top of the brain (where size / color represent your effect and location is the electrode position) B. Create some sort of heatmap on the cortical surface model using the electrode positions
It seems relatively straightforward to figure out B, and if an
ecog
type is ever added to mne-python I think it'd be a great viz addition. However, I'm not so sure how to do "A" effectively. Right now, I've tried using pysurfer to read my surface model. I've then tried two methods for plotting, which both fail for different reasons:plot_trisurf
to do the same thing. In this case, it creates nice "flat" scatterplots in the 3D axis. However, the call toplot_trisurf
takes a really long time, and is unusably slow when it comes to rotating etc.So I wonder, is there some way to combine the 3D capability of mayavi with the "data plotting" capability of matplotlib? I know that packages like this exist for matlab (e.g., here). These use matlab's version of plot_trisurf, which I guess is more usable. Any thoughts on how to make this work?