Support for use of alternate atlases (e.g. Human Connectome Project atlas)

[bcipolli]

Tools / datasets from the Van Essen lab, such as the Human Connectome Project (HCP), Connectome workbench, and SumsDB, make surface data readily available without using a freesurfer installation. These include alternate atlases with different vertex counts.

Currently PySurfer assumes a lot--it assumes a freesurfer-like directory structure for data, it loads curvature information from strongly assumed file locations (base directory path is flexible, but the subdirectory and filename are assumed).

To generalize things and make the tools more useful for the new HCP workflows, I suggest generalizing all paths to be taken as optional arguments. The defaults can be built from the assumed freesurfer locations, but I would like to specify my own atlas (in addition to my own overlays).

I'm hacking a prototype together for this for OHBM (as I've got HCP surface data that needs to be plotted on HCP atlases), but wanted to get interests / comments on the idea.

Thanks!

[mwaskom]

To generalize things and make the tools more useful for the new HCP workflows, I suggest generalizing all paths to be taken as optional arguments. The defaults can be built from the assumed freesurfer locations, but I would like to specify my own atlas (in addition to my own overlays).

IMO this seems radically easier said than done. PySurfer is very closely tied to Freesurfer, not just in the location of the data, but also in what things are called ("annotations", "labels", "morphometry"), etc. Taking what we have and extending it to work with an entirely different sphere of data, with its own conventions and quirks, seems destined to be a massive hack.

I agree that it would be great to have a nice Pythonic interface to visualize HCP-style surface data. I'm not sure that building flexibility into PySurfer is the right way to do it, though.

[bcipolli]

The HCP data use freesurfer and file formats readable by nibabel. It's easy to convert file formats before calling into PySurfer (or using the power of nibabel to read files and make PySurfer agnostic to format); the problem is the directory structure and file naming assumptions.

I believe there's a relatively general way to do this (passing file paths to needed files, rather than passing a subjects_dir and assuming you know how to construct the paths). All the file formats can be assumed still, in the way I'm thinking.

Does that sound more reasonable?

[dengemann]

Note that we have a related discussion elsewhere: https://github.com/mne-tools/mne-python/issues/2116

cc @mwaskom

[dengemann]

PySurfer would be part of the solution somehow or we will eventually find another viz backend.

[bcipolli]

@dengemann Thanks for the comment, but I'm unclear: is there any overlap between the discussion of the challenges with the HCP MEG pipeline and this one? I believe the BTI coordinate system is MEG specific. Here, I'm talking about the HCP anatomical surface maps, which I believe are built with surface files that are easy to translate to freesurfer format (via nibabel). So, the overlap is not clear to me.

[mwaskom]

The HCP data use freesurfer and file formats readable by nibabel. It's easy to convert file formats before calling into PySurfer (or using the power of nibabel to read files and make PySurfer agnostic to format); the problem is the directory structure and file naming assumptions.

If you're already converting to Freesurfer file formats, why not just write into a Freesurfer-like file hierarchy?

[bcipolli]

I don't think it's a very good user experience. Converting files across formats and passing them to a function is a typical user experience. Asking a user to define a SUBJECTS_DIR variable and put their data into a freesurfer-specific directory structure and naming convention does not strike me as a nice user experience.

In my mind, the Freesurfer pathways is a specific subclass, where you can generate the paths for the user by assuming they're processing their surface data via the typical Freesurfer directory structure. Then those paths can be passed to the more general logic above.

[mwaskom]

I don't think it's a very good user experience. Converting files across formats and passing them to a function is a typical user experience. Asking a user to define a SUBJECTS_DIR variable and put their data into a freesurfer-specific directory structure and naming convention does not strike me as a nice user experience.

Sure, it's a hack in user-space to avoid hacks in the library code.

In my mind, the Freesurfer pathways is a specific subclass, where you can generate the paths for the user by assuming they're processing their surface data via the typical Freesurfer directory structure. Then those paths can be passed to the more general logic above.

OK, but you're basically describing a completely different library that what exists currently, which is a little different than "add a parameter that accepts a file path".

[dengemann]

is there any overlap between the discussion of the challenges with the HCP MEG pipeline and this one?

Sorry for not having been sufficiently explicit. I wanted to link this to keep track of the discussion here for resolving our MNE-Python issue. The link might be a bit implicit, but thinking ahead we will have many problems at the PySurfer level once the BTI converison issues are settle in the way discussed there. Plotting might be get done by convenience functions that produce the required freesurfer file and directory structures while simulating no-decimation surfaces as we discussed with @Eric89GXL. But then what about the label functionality? How to map labels to an HCP surface, how about mappig volumes to HCP surfaces, and so on. Thinking about it, I see too many places where usually working pipelines would break with HCP data at the viz and freesurfer-related IO level. That's even the case if we have the HCP-MEG data inside MNE-Python such that we can perform foward modeling and source localization. I hope it's now more obvious why I was linking the issues.

[larsoner]

@mwaskom WDYT about colloborating to create a more general bit of brain visualization software? It seems a bit overkill, but it might not be quite as terrible as it seems. We could start from the ground up with an API that is designed to be file-format agnostic, building on top of nibabel and vispy for IO and visualization backend respectively. The next couple of months might be a good time to do it, since vispy is almost at the point where it would make sense to swap it in as the backend. We would even get to pick a new cool name :) Of course my personal highest priority would be making it work with Freesurfer (followed by HCP data) since that's what we use here at UW, but we could at least design it to be capable of taking other formats.

Right now I actually prefer that solution, because PySurfer as it exists is very good at what it does, but trying to generalize it to do more -- either by way of viz with vispy or IO with nibabel -- is likely to lead to a bunch of difficult API choices and it would be very difficult to avoid breaking people's existing code. If others think it's a good idea, then please also mention how much time / effort you could give to the effort, since it would take some person-hours to get off the ground. I think I could find a few hours per week in July, and probably more in August.

[mwaskom]

I think that would be a more promising route forward. I'm not sure what bandwidth I'd have to contribute; probably not much, though.

Before committing to that kind of effort it would be good to sort out some important details. I don't fully understand the vision at the heart of this issue, but it seems like the idea would still require the user to know how to convert HCP data into something Freesurfer-esque for use in PySurfer. IMO that's not great—if we're gonna take the effort to write a new package to support HCP data, we should support it natively. But that means having someone involved who really understands the way those datasets work, which definitely isn't me.

There's also the question of exactly how useful a new package that supports HCP data is going to be. I'll confess that I don't full understand the whole HCP thing, but from what I do understand the Van Essen lab has put a lot of resources into developing a visualization tool that's tightly integrated with it (Connectome Workbench). Given that this tool exists, is there really a large potential userbase for a viz-only Python tool that does the same thing be maybe not as well, except for better default colormaps?

Also is the goal to have a surface based visualization platform that is totally package agnostic? Or just to support Freesurfer+HCP. Are there going to be users thinking they'll be able to use surface system X with it that end up disappointed or requesting enhancements that violate whatever new assumptions are built into the package? Having a sense of what that landscape looks like is important.

[bcipolli]

Just to follow up, #OHBM2015 hackathon participants seemed to agree that having a JS/web-based visualization tool for surfaces is the most useful route to go. It's useful for freesurfer and HCP data together.

The "HCP" format is really just GITFTI/CIFTI, both of which are either integrated in, or in the process of being integrated in, nibabel. The major issue for freesurfer vs. HCP is really the directory format. Other than that, should be good.

Besides having a lot of buzz around the HCP project, the Van Essen Lab is also working on re-releasing their large, public database of surface-based results. We discussed that interactive, web-based results is the direction things are headed (and that's good!), and that a viz package similar to mpld3 (matplotlib in D3) would be best.

I'd be happy to contribute to such a package, and there are reasonable starting points out there (one was prototyped during the #OHBM2015 hackathon; see https://github.com/binarybottle/mbviewer). My plan is to follow up with @binarybottle and @akeshavan about that tool when I can, then start some discussion on neurostars or the nipy mailing list.

[binarybottle]

Ben -- Would you like to help with the JavaScript front end, or would you prefer to work on the Python format conversion / PySurfer visualization?

[bcipolli]

@binarybottle I think the JS front-end is the best way to go. I'm behind on my other commitments (nibabel / CIFTI and nidata), but I would be interested to stay in the loop and contribute (as possible) to a JS front-end.

[binarybottle]

Great! @akeshavan and I will figure out some next steps...

[mwaskom]

We discussed that interactive, web-based results is the direction things are headed (and that's good!), and that a viz package similar to mpld3 (matplotlib in D3) would be best.

Does this mean something that sits on top of PySurfer and converts its output to javascript?

[bcipolli]

Does this mean something that sits on top of PySurfer and converts its output to javascript?

No. My understanding is that PySurfer is specific to FreeSurfer, and people were resistive to the idea to make it more general-purpose. The proposal is for an alternative to PySurfer, which is JS-based and doesn't depend on a FreeSurfer-like directory structure.

[mwaskom]

Right I'm just confused about the analogy with mpld3, which is just a post-processor for figures made in matplotlib to d3 code.

[bcipolli]

I see. I was thinking of mpld3 in terms of user interface, not underlying implementation. That is, running Python code and getting a browser-based visualization (and perhaps launching a temporary web server to visualize the current result) .

I would work out with @binarybottle, @akeshavan, and others, on what implementation is best to make this happen. All that's clear to me at the moment is, modifying the PySurfer code to be more general is not a desired part of the solution.

[mwaskom]

By the way I assume you're familiar with PyCortex but what you're describing sounds congruent with that package.

[mwaskom]

FWIW I think a javascript-based thing for interactivity will be great, but I do think there is still need for a Python-based library for scripting, ideally with a backend that can handle offscreen drawing.

[binarybottle]

I completely agree that there is a need for a Python-based library for visualizing brain image data. I use Hal Canary's vtkviewer (https://github.com/binarybottle/mindboggle/blob/master/mindboggle/thirdparty/vtkviewer.py) for simple vtk surface viewing within mindboggle (https://github.com/binarybottle/mindboggle) to reduce overhead.

[transpersonify]

Due to open-access ROI timeseries and connectivity matrices data from HCP, there is a whole community that is in need of easy visualization of the ROI data on the known atlas surfaces. I have used pysurfer before for the atlases existent in the Freesurfer using this example code:

I am curious if there have been any development in 3 years to do this quick and dirty for any other atlases given an array of values for each ROI and a Nifti file having labels of atlas or 3-D ROI co-ordinates. Can combination of Nilearn atlas plotting do this?

[larsoner]

If you just want HCPMMP labels and not the subcortical ones, see mne.datasets.fetch_hcpmmp_parcellation and mne.morph_labels.

If you need cortical+subcortical (or to go from volume to surface labels) I think this is essentially a FreeSurfer issue, see example code here for how to get HCPMMP+aseg.mgz atlas on fsaverage in a few lines of MNE-Python plus one FreeSurfer call:

https://github.com/mne-tools/mne-python/pull/7639#issuecomment-625907891

To do the same thing for specific subjects, you can morph the HCPMMP labels using mne.morph_labels, then use the same command to get a HCPMMP+aseg.mgz atlas file.

Finally if you want browser-based viz you could consider MNE-Python's Brain class that is adapted from PySurfer's but uses PyVista rather than Mayavi under the hood. It has jupyter notebook viz + interactions as of a few months ago via ipyvtk_simple:

https://github.com/mne-tools/mne-python/pull/8503#issuecomment-755435268

https://user-images.githubusercontent.com/18143289/103798907-a7ec1100-504a-11eb-8d45-f775836cbb9c.mp4