Support for custom atlases

[FedeClaudi]

Some users manifested interest in using brainrender with atlases other than the Allen Brain atlas.

While any 3d mesh can be rendered in brainrender with: https://github.com/BrancoLab/BrainRender/blob/74fefa9eefd3bdbd7aef377b016e0f631fb184c2/brainrender/scene.py#L394

many functions only work for the Allen atlas because they expect a region's acronym as input: https://github.com/BrancoLab/BrainRender/blob/74fefa9eefd3bdbd7aef377b016e0f631fb184c2/brainrender/scene.py#L428

A first workaround would be to let these function accept filepaths to custom meshes or vtkactors as argument, then most of brainrender's functionality would be working on a custom atlas. However, a more comprehensive and long term solution would be preferable.

[FedeClaudi]

Some of brainrender's functionality depends on external APIs and databases (e.g. mouselight).

In this case it would be hard to use custom atlases, and even if it were possible it might cause problems since the data are registered to the Allen atlas.

So functionality that depends on datasets based on the Allen atlas will probably not be extended.

[adamltyson]

Do you mean "functionality that depends on datasets not based on the Allen atlas will probably not be extended."?

Steve and I were playing with this atlas. It's based on the ARA (and is registered to it), but has greater granularity. We had a prototype working in cellfinder which used an image-based atlas and an amended structures.csv. We could then extract regions and render as surfaces in BR as needed.

Just thinking out loud: I've had success with [GIN}(https://gin.g-node.org/cellfinder) for hosting data (and allowing automatic downloads when needed), so I think if someone were keen to persue this, the steps would be:

• Extract surfaces for every brain region from an image-based atlas (if the surfaces didn't already exist)
• Ensure metadata for hierachy etc. exists
• Host online in a standardised way
• Edit BR in such a way that it could query this downloaded atlas, rather than the Allen API

Is there any reason why this approach shouldn't work for other (non ARA based) atlases? Of course you would need some checks, i.e. don't try to render data from Mouselight with a zebrafish atlas etc.

[FedeClaudi]

So I think that there are two cases:

1. Atlases that are extensions of the Allen atlas for the adult mouse brain.
2. Atlases that are very different (e.g. different species or developmental stages, or specific to a brain structure).

For the first case datasets like mouselight would be okay, since it's roughly the same brain. What I meant is that in the more general case datasets based on ARA should not be used with other atlases (obviously).

I like your approach, though I guess steps 1 -> 3 should happen outside of brainrender. Then brainrender can download/load the data and organize an atlas.

[adamltyson]

Yeah sounds good. 1-3 should definitely happen outside of BR, but there should be some standardized structure (that could maybe also be used with amap and cellfinder).

[FedeClaudi]

Yeah, agreed that would be good.

I think for now I want to deal with the easier case of pre-existing and established atlases (e.g. drosophila's) as:

1. These atlases would probably be relevant to a larger number of people
2. They are already available and hopefully accessible to anyone

Once brainrender can be used with these atlases it should be easy(er) to support custom user-created atlases too.

[adamltyson]

Great. Keep me in the loop, I'd like to see how they could be supported with amap/cellfinder/neuro etc too.

[FedeClaudi]

Sure, I first need to get a hold onto these atlases, do you know where to find them?

I've created a new branch to work on this, if you want to keep an eye on it.

[adamltyson]

There are some here, but they're pretty low-res.

If you're interested in zebrafish etc. I can ask around my old dev neuro department.

[FedeClaudi]

Yeah they are not super high res and they're not segmented. Some of them like the rat's can be downloaded freely and each voxel is annotated with the brain region so it should be easy to make something to visualize selected brain regions. The only thing that it's lacking is a hierarchical structure, but that's okay I guess

[anorak94]

just a few comments @adamltyson mentions that we can "Extract surfaces for every brain region from an image-based atlas (if the surfaces didn't already exist)" does this mean that the image based atlas already has meshes generated for it because i couldnt find any the way i am doing it is getting a neuroanatomist to draw the structures in svg file, parsing those files and then registering it to a reference atlas(which i create), transferring the annotations to itk snap and then interpolating labels followed by manual editing to get the entire mesh for the structures. I also had to write the code for the volumetric brain creation from histological stacks and mri priors i used antspy for it. Do you guys know of an easier way to get around this process this is roughly what the allen brain did for their atlas creation but my meshes look really ugly

[FedeClaudi]

I'll let Adam explain this more in detail but in the meanwhile you can have a look here:

https://github.com/BrancoLab/BrainRender/blob/c6c4fdb740564b2b762e1aa91eb83256c33c9720/brainrender/Utils/image.py#L53

and here (though Neuro is still under development): https://github.com/SainsburyWellcomeCentre/neuro/tree/master/neuro/segmentation/manual_region_segmentation

[anorak94]

So the way to use this function would be to create a mask of the different regions of the cerebellum using manual delineations in histological space i then transfer these to the reference atlas and use this function to create a mesh of the surface i do this for a lot of serial images and simply stack them to get the 3d mesh.?

[adamltyson]

I'm not sure if I fully understand, but I'll try to help clarify what tools we have:

• Brainrender uses the Allen adult mouse atlas, which is defined both as images (each region has a value), and meshes (which brainrender downloads and renders).
• We have used other atlases based on the Allen (e.g. https://www.nature.com/articles/s41467-019-13057-w) which are still in the same coordinate space, but with different labels. We have used the image based atlas to extract meshes for brain regions, and display them manually in brainrender.
• We are developing tools in neuro to allow people to manually create their own meshes (they just draw round a region in 3D. Some of this code is also still in brainrender, but maybe should be removed (@FedeClaudi?).

So I don't think we have tools to do what you want to do, but there is maybe some code that can be reused. Happy to work with you if you wanted to make a PR to neuro to add functionality etc.

[FedeClaudi]

Hey but I think what he meant is that you need a 3D mask for each brain region to use the marching cube stuff and extract the surface's mesh. If that's what you meant @anorak94 that's correct, you need a mask for each brain region defined in the same coordinate space as the atlas.

What we don't have is a way to create these masks though.

And yes @adamltyson that code will be removed soon enough :)

[anorak94]

@FedeClaudi i have a way to extract those regions from svg allen brain atlases or other svg delineated images. so what i understand is create a mask for those regions for all slices in the reference atlas when we say mask do you change the voxels in the original reference atlas to the value of the mask or do you just give the 3d mask with different labels values to the marching cube algorithm as input. Id be happy to share my code for region extraction from histological slices and then alignment to the reference atlas for any of your packages if you guys want it. But drawing a ton of regions on a lot of slices is tiring. Itk snap takes as input spaced delineated regions and then interpolates labels to the slices in the middle and also lets you manually edit the masks is your approach better and more convenient than itk snap

[FedeClaudi]

I see, I'm afraid that's something that we don't have tools for at the moment, sorry

[vigji]

Hi @FedeClaudi! With @adamltyson in the last days we were thinking about some way of having a nice API for dealing with an Atlas representation, defined as a folder with some stacks, regions, metadata, some link for download, etc. I temptatively put a draft for it in the neuro package, I am sure there would be a loot of overlapping for stuff that you already might have or would need for this support of general atlases - what about try to come out with a common solution for dealing with it there or in some other place we can all refer to? I like the idea of it being in a place where on the tools that have to deal with such problem in specific applications can point to

[vigji]

The good practice of posting this before having read all the above messages fro @adamltyson and read you Atlas class code 😅 The point about discussing a multipurpose Atlas class in some module more general than brainrender holds though!

It would mean maintaining together stuff that is generally useful (atlas metadata, files structure, areas hierarchies, meshes reading, assignment of a coordinate to hemisphere/region, files I/O); I would be quite interested in dealing with the zebrafish part, as it's something I am working on at the moment in other libraries for my lab; and have this integrating nicely with more specific applications (in your case, rendering etc).

[FedeClaudi]

hey, yeah that sounds great! I'd be happy to work on it with you guys. I guess it would have to come with a solution for hosting data for atlases that don't have an API like the Allen atlas?

I started doing something similar for brainrender: https://github.com/BrancoLab/BrainRender/blob/custom_atlases/brainrender/atlases/base.py

For just brain render's needs this is enough, but it would for sure be good to have something standardised across packages.

Do you guys think that it should be part of neuro or should it be it's own thing?

[adamltyson]

Hi both,

I think hosting should be fine (GIN has worked well for me so far), we just need a standardised structure.

I vote for a new repo, as it shouldn't have many dependencies, and neuro has some fairly heavy ones now.

[vigji]

I also vote for a new one, so we start with a fresh mind and add only things as we see them suited :)

[FedeClaudi]

allright, sounds great, I'm happy to be part of it!

[vigji]

Great! I am for sure more of a newcomer to the brain atlases enterprise than you; I have tried to look around for something like this already existing and I could not find anything obvious existing in this general form. Does the Allen actually provide something like a general API to deal with their atlas data that we would be overlapping with?

For hosting, should I make a new organisation so that we all have admin access?

[FedeClaudi]

No I'm afraid the Allen's SDK are specific to their stuff.

[adamltyson]

Yeah new organisation is a good idea. Any ideas for a name (clearly the most important thing)?

For a package name - atlases is available on pypi

[vigji]

Something fancier? :D brainglobe? brainchart? 😃

[adamltyson]

I like both of those. Do you mind making an org, then we can discuss code organisation and what packages we would need?

[vigji]

Sure!

[FedeClaudi]

@anorak you can check this to see how you can start using brainrender with custom atlases while vigji Adam and I work on brainglobe.

If you plant to do so, make sure to get the latest code with

pip install -U git+https://github.com/BrancoLab/BrainRender.git

[anorak94]

hey thanks it looks cool

[FedeClaudi]

Hey @anorak94 we've made some progress with brainglobe and we start having more and more atlases supported.

One of the problems we've encountered is that when we create new meshes in our atlases they have some imperfections. I know that you've had the same problem before and was wandering have found a solution?

[anorak94]

What kind of imperfections do you mean. Id be happy to help. I asked a question on the allen brain forum about how they create meshes for their atlases. Maybe you can take a look at that as well. The guy suggested doing decimation for reducing mesh sizes and some structure shrinkage to ensure that the meshes of different structures dont intersect each other.

Apart from that. I have written code which extracts contours from svg images overlays them on mri atlases then does interpolation to propagate labels to the intermediate slices(this step is buggy and the main source of errors in my meshes) i then load the segmentation masks in itk snap and do an export mesh. In itk snap i also do some manual editing to ensure that the structires are well delineated. I then smooth the meshes.

If you can tell me what imperfections you have in your meshes maybe i could tell you more.

Best

[FedeClaudi]

That's great, thanks for the help.

We are working with 3d images so the first part of the workflow is different. We have a 3d mask with the voxels assigned to a brain region, and we extract a mesh with the surface of the mask. However these often look blocky and sometimes they have artifacts (e.g. some geometry under the brain region's surface), so we are looking for ideas on how to clean them up.

I think we will try to implement what was suggested to you here, but were wandering if you had any further info that would help.

By the way, I know that you were working on creating an atlas (was it a developmental mouse atlas?), if you want to develop it as part of brainglobe / have it supported by brainglobe we welcome anyone interested in helping developing the atlas api.

[anorak94]

hey yeah indeed i was working on creating an atlas for the developmental mouse brain. i have written code for creation of a 3d reference volume from histological stack(rn im waiting for some mri data from developing mouse brain to test it, it works on the adult mouse brain i used the pre built 3d volume as a proxy for the mri data). i then wrote scripts for getting meshes from volumetric data(so i neuroanatomists will draw on the reconstructed atlas in step 1 i will transfer these annotations to the reference atlas and then extract the meshes). Thus creating a fully annotated allen brain atlas for every age of the mouse brain (given a histological stack, an mri volume and a neuroanatomist willing to draw structures). i have tested parts of the pipeline individually but i havent run it all together at once and created a final mesh i dont have the mri data to do it. i would be happy to share my code and everything else i just wanted to see it work end to end for once(we have tried to arrange some data and hopefully we should get it soon dont know when but if my pipeline works id be happy to share it , just need to ask my PI).

[FedeClaudi]

That sound cool, let us know when you get it to work and we can integrate it with brainglobe so that it's fully supported by software like cellfinder and brainrender