How do we represent regions of interest in OME-NGFF?

[jluethi]

Context: We were discussing OME-NGFF usages recently with @tischi and I noticed we were going in somewhat separate directions on this topic. I’d like to start the broader discussion to see where the OME-NGFF standard may go.

The overarching question: How do we represent regions of interest?

What are regions of interest? (to us)

Defined parts of a dataset. In our use case, that’s mostly the field of views of a microscope or large objects like organoids. But it could be any arbitrary region in an OME-NGFF dataset.

What approaches are there (that I’m aware of)?

1. ROI Tables: Our approach to regions of interest is saving ROIs in the newly proposed OME-NGFF table specification. So far, we’re just looking at rectangular ROIs (⇒ we define a corner and the extent in x, y & z in physical units. See here for details: https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/24)
2. Crops in transformations: Using the proposed OME-NGFF transformation, @tischi is looking to define regions of interests as a crop transformation. By specifying a crop transformation, a clearly defined part of the dataset is described. (@tischi can you explain more detailed?)

Pros & Cons

The benefit of ROI tables are that it’s easy to save additional metadata per ROI as extra columns of the table. It’s also easy to loop over all entries of a table or get a quick summary of available ROIs. ROIs in a table can also be specified easily for each well of an HCS OME-NGFF file. Finally, more general ROIs than rectangles are in the future outlook vision of the OME-NGFF tables proposal.

The benefit of crop transformations are the abilities to chain transformations together. Plus (again, please elaborate further @tischi) the potential to define them in or in relation to different named coordinate spaces (I can’t fully imagine the use case there yet, but that may be the limiting view of the somewhat static coordinate space of HCS data).

Questions

Are other people also looking at saving ROIs? Are there clear preferences of how ROIs should be saved in OME-Zarrs? My rough understanding is that both approaches should enable roughly the same functionality, with small trade-offs in how easy specific things are to achieve. But I'd be looking forward to discuss this further with people who are interested!

cc @kevinyamauchi @joshmoore @ogovicj @gusqgm (also please tag people that would be interested in this discussion)

[tischi]

Adding to that: I further discussed this with @kevinyamauchi and he raised the question whether cropping (or sub-setting) a dataset would actually be a coordinate transformation at all. I can sympathize with this notion...

@bogovicj what's your take on this?

[bogovicj]

Thanks for the ping. Short answer: my preference is for option (1) - ROIs are stored using tables

he raised the question whether cropping (or sub-setting) a dataset would actually be a coordinate transformation at all.

In my view, a coordinate transformation (ct) by itself is not an ROI, but might help define one. The main reason being that ct's don't say anything about the extent of the data, and I don't think they should.

rectangular ROIs (⇒ we define a corner and the extent in x, y & z in physical units.

this sounds great. I'd advocate for also having a name + properties to the coordinate system in which the roi is defined (i think @kevinyamauchi has this in mind ). This is a place where cts might be useful. For example, one could define a non-rectangular roi by transforming a rectangular roi.

Another example of how cts are related to rois. If a tool can do something like 1) select a rectangular roi. 2) resample some array using that roi 3) write a new array of the crop

I will insist that the resulting array have transformation metadata that describes how the new array's coordinates are related to the larger array.

But if a tool only does steps (1) and (2) and displays the crop independent of original, I'd want displayed coordinates to be the same as for the original (e.g. the "origin" of the crop should be the "min corner" of the rectangle roi that defines it). So, depending on the implementation, a ct might help in keeping track of that internally, but the tool wouldn't have to write any new transformation metadata.

Hopefully that's mostly clear.

[tischi]

It is always good to talk ❤️ as this made me rethink of what we actually currently mean by a transformation in mobie. And, in fact, it made me realize that what we actually probably mean is an imageTransformation, with such an interface (@constantinpape, would you agree?):

public interface ImageTransformation< A, B >
{
    Image< B > apply( Image< A > image );
}

One implementation could be the transformation of an image into a new coordinate system, but it may also be the cropping of an image. In fact, this definition would even cover a Gaussian blur. Historically, we started by just having coordinate transformations, cropping came later. I think the general concept is still useful for us (not saying that we ever want to serialize this concept into OME-Zarr). I am not sure whether transformation is still the best word here.

[constantinpape]

for the general discussion: I think the proposed solution of having ROIs in a table sounds good (there might be cases where it could still be good to have a transformation for it, but I think it's fine to go with the table solution first and only if it turns out that the transformation is necessary consider adding it as well)

transformation in mobie. And, in fact, it made me realize that what we actually probably mean is an imageTransformation, with such an interface

Yes, I agree that our MoBIE notion of transformation is fairly broad and fits your interface, which can also include intensity based transformations like smoothing. This is also the reason we chose the name coordinateTransformation in ome ngff to make the scope clear.

[LucaMarconato]

Also relevant for SpatialData. Happy to receive feedback on the code when it's ready (still WIP). We are implementing a tentative representation for all the types of ROIs that Napari currently supports, and we will be happy to discuss/refine it into a commonly agreed representation. FYI @giovp @ivirshup

[jluethi]

@LucaMarconato Very cool! Is there an overview somewhere of ROIs that napari supports? I find the ROI plugin that defines ROIs in 2D as X & Y positions + X & Y height (see here: https://github.com/BodenmillerGroup/napari-roi), but haven't seen a standard for 3D ROIs. We currently use this same approach, just with an extra axis. We have now started adding optional columns of "original position X" etc. when we move ROIs (for different reasons, e.g. issues with stage positions precision, some ways of handling overlapping imaging). Would be curious to get involved in this discussion & give feedback on SpatialData implementation if you can point me to a relevant place! :)

[LucaMarconato]

I deduced the list from the APIs, the 2D/3D regions that napari supports can be displayed as napari.layers.Labels, napari.layers.Points and napari.layers.Shapes. The shapes can be of different types, by specifying shape_type in 'rectangle', 'ellipse' and 'polygons'. There are a few more layers and shape types but they are not for defining areas in 2D or volumes in 3D, rather 1D information in 2D/3D or 2D surfaces in 3D.

Regarding the way things are represented we are at the moment saving everything that is not a labels in anndata, with the geometric description specified in each row, but we are exploring also relying on a pure geometric descriptions, like geojeson, associated to an annotation table. We could discuss this in one the next spatial data meetings.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/tool-for-adding-labels-group-to-ome-ngff/72839/2

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/roi-annotations-and-tracking-information-in-ome-zarr/65975/7