PlaneSegmentation holds ROI image masks for optical physiology

[bendichter]

here is the corresponding NWB Widgets code for finding the contours of the image masks and plotting them in Plotly: https://github.com/NeurodataWithoutBorders/nwbwidgets/blob/d62bed037e24c75f8a9215b8f0ab81e975763700/nwbwidgets/ophys.py#L198-L303

[bendichter]

Note that even getting to the point where we can navigate to a PlaneSegmentation object in neurosift requires better handling of /general

[magland]

@bendichter Do you have an example nwb with a PlaneSegmentation object?

[bendichter]

https://dandiarchive.s3.amazonaws.com/blobs/a93/0bb/a930bb36-4e47-4158-bc61-6d4a8e77ec9b

PlaneSegmentation is inside ImageSegmentation (this is another example of a MultiContainerInterface)

[magland]

I am not able to find any ImageSegmentation objects in that dataset...

[CodyCBakerPhD]

I see them under 'processing' -> 'ophys' -> 'ImageSegmentation' but I'll admit this file has a lot of contents overall

Perhaps a simpler example, albeit one that might also be a good test of the data engineering capabilities (especially w.r.t. image/video streaming) of NeuroSift, might be: https://dandiarchive.s3.amazonaws.com/blobs/7e3/e07/7e3e0785-4793-4d7f-b07a-bb2d63b127f2 (DANDI set 000350, file sub-20170113-2/sub-20170113-2_ses-20170113T124907_ophys.nwb)

EDIT: pick a file that actually has segmentation

[bendichter]

@CodyCBakerPhD it looks like the file you link doesn't have a PlaneSegmentation object

[CodyCBakerPhD]

@bendichter Whoops, that was one of the unsegmented glia sessions - updated

[bendichter]

OK, well that one is in 3D. Let's get the 2D version working first and maybe put 3D in another issue

[magland]

@bendichter

I'm not sure how this structure corresponds to your source code:

For example, I don't see an "image_mask" dataset.

I do see pixel_mask and pixel_mask_index, but I don't see the dimensions of the image. I'm trying to imagine what this plotly plot should look like - do you have a screenshot example?

[CodyCBakerPhD]

I do see pixel_mask and pixel_mask_index, but I don't see the dimensions of the image. I'm trying to imagine what this plotly plot should look like - do you have a screenshot example?

Aha, good reason for us to find many other examples of this...

Technically NWB has two ways you can convey image mask type information

image_mask is when you specify an array that is the same size as the image and the values are the weights (I believe between 0 and 1) for every pixel

pixel_mask is a more efficient representation where it's just a list of pixel indices (and an optional weight, again 0 to 1) - for plotting in that situation, you'd have to use the linked photon series or reference image (if available) to judge the image shape

[bendichter]

@magland here's an example of an NWB file that uses image_mask:

https://api.dandiarchive.org/api/assets/57e9f6a9-6d06-4e75-8ad7-8e2ab2ab3565/download/

from https://dandiarchive.org/dandiset/000054/0.210819.1547/files?location=sub-F1

[CodyCBakerPhD]

OK here's another example of an image_mask

https://dandiarchive.s3.amazonaws.com/blobs/368/fa7/368fa71e-4c93-4f7e-af15-06776ca07f34 ('MICrONS', DANDI:000402, sub-17797/sub-17797_ses-5-scan-7_behavior+image+ophys.nwb)

Again found in 'processing' -> 'ImageSegmentation' -> 'PlaneSegmentation{pick an index}'

[magland]

@bendichter @CodyCBakerPhD here's a prelim ImageSegmentation view

The above examples:

https://dandiarchive.org/dandiset/000402/0.230307.2132/files?location=sub-17797 sub-17797/sub-17797_ses-5-scan-7_behavior+image+ophys.nwb processing/ophys -> ImageSegmentation https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/368/fa7/368fa71e-4c93-4f7e-af15-06776ca07f34

https://dandiarchive.org/dandiset/000054/0.210819.1547/files?location=sub-F1 https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/cb2/fc5/cb2fc5d3-3a37-4e1c-bc03-c35cf83db68c

Since each unit mask is stored as a full image, it can take a while to load.

[bendichter]

@magland looks great! Could you add the "Loading..." indicator?

[CodyCBakerPhD]

Looks beautiful!

It also finally enables direct side by side comparison against frames from the TwoPhotonSeries

When I do that to 000054 I get

which makes me wonder if perhaps there is some flipping/transposing going on, and if so is it an issue in the data or is it perhaps the image in NeuroSift that needs rotation? (or even rotation buttons on both windows just for good measure and convenience might be nice)

Tried doing it to 000402 as well but the TwoPhotonSeries there gets stuck loading, not sure why

[magland]

Opened a few new issues related to the above comments. Closing this one.