Create Image Segmentation module

[tischi]

I think I have an idea 😸

What about we create two new modules:

Semantic segmentation
Instance segmentation

And the Activities and Exercises in these modules are not about how to create those segmentations, but how such segmentations are typically stored (label mask images) and how to inspect already existing segmentations**; stuff like: how many objects are there, how many semantic classes are there, are they disjoint, what class covers the largest part of the image, ...

I think this could be really good, because it enables the students to understand the output of all the bizillion methods and tools that segment stuff in images.

Maybe better: just one module: "Image segmentation" and there we explain the difference between instance and semantic and that both can be stored as label masks.

[tischi]

@constantinpape May we use some material (text and images) of your presentation about image segmentation (semantic, instance, ...) for creating a publicly accessible teaching module here? If so, could you share the presentation with us?

[constantinpape]

You're welcome to re-use my material, but I mostly have some initial slides about this in talks where I present my own work. Here are some talks that might contain relevant slides:

• CBB Unit Seminar
• Application talk
• Slides for a course I gave at University (with @wolny):
  • Day 1: ML Intro
  • Day 2: DeepLearning and ConvNets
  • Day 3: Segmentation & Denoising
  • Day 4: Denoising

[tischi]

@constantinpape Thanks a lot!

This is my first try:

1. Didactic improvements are welcome!
2. Could we put the membranes and mitochondria into the platybrowser? Then I could make such images myself.

[constantinpape]

• Didactic improvements are welcome!

• Could we put the membranes and mitochondria into the platybrowser? Then I could make such images myself.

• Membranes: a bit of a space issue, this is about the same size as the raw data again, but can be done.
• Mitos, the training data is technically not publicly available, so I haven't made the model and predictions public yet; I will follow up on it

(also a bit of a timeline question, I wanted to have on-the-fly predictions in MoBIE as one of the outcomes of Ekaterina's thesis, which would make this much more convenient)

Didactic: concept map: what exactly is the differnce between "Label mask image" and "Regions of same type" for a Semantic segmentation. Would you store regions as coordinate lists or polygons?

[tischi]

Membranes: a bit of a space issue, this is about the same size as the raw data again, but can be done.

Really? If it is binary it should compress very well?

what exactly is the differnce between "Label mask image" and "Regions of same type" for a Semantic segmentation.

I somehow want to make the point that "Label mask image" is a very common way to store both semantic and instance segmentation; in fact I wanted to make it an exercise that students open an label mask and try to figure out what sort of segmentation it is.

Related to this: For an instance segmentation is it the case that all pixels of one instance MUST be connected or could they be disjoint?

In other words, if you open a label mask image, is there a way to figure out (in some cases) whether this more likely is an instance or a semantic segmentation?

[constantinpape]

Membranes: a bit of a space issue, this is about the same size as the raw data again, but can be done.

Really? If it is binary it should compress very well?

Usually one would store the probability maps, especially for membrane predictions, because binarizing loses information. (Thinking about this, I think probabilities should also be part of your concept map.)

I somehow want to make the point that "Label mask image" is a very common way to store both semantic and instance segmentation; in fact I wanted to make it an exercise that students open an label mask and try to figure out what sort of segmentation it is.

Yes, that makes sense, but does not answer the question what the difference between "Label mask image" and "Regions of same type" is.

Related to this: For an instance segmentation is it the case that all pixels of one instance MUST be connected or could they be disjoint?

No, they may be disconnected and there are genuine applications, e.g. occluded objects in natural images (e.g. a pole in front of a car) and in microscopy (e.g. crowded cells imaged in 2d where the protrusions of one cell are above another cell). You can also have more complex representations where a single pixel can be assigned to multiple objects, which is usually done by giving the per object coordinates (probably corresponding to "Individual objects" in your concepts.)

[tischi]

but does not answer the question what the difference between "Label mask image" and "Regions of same type" is.

I thought about it like this: Label mask image is one implementation to store a Semantic segmentation onto a hard disk. Regions of same type was meant to be a more understandable explanation of what a Semantic segmentation contains, but I think this should be improved or maybe left out completely from the concept map? In other words, I was looking for some very short (few words) explanations for what Semantic segmentation and Instance segmentation are....

[constantinpape]

Regions of same type was meant to be a more understandable explanation of what a Semantic segmentation contains, but I think this should be improved or maybe left out completely from the concept map? In other words, I was looking for some very short (few words) explanations for what Semantic segmentation and Instance segmentation are....

Ok, this makes sense, but than "Regions of same type" should not be on the same level as "Label mask image", because if you just look at it right now it appears that these are two different options of representing a semantic segmentation, but "Regions of same type" is a less technical synonym.