Add and allow mixing paint brushing with polygons for segmentation masks

[Mut1nyJD]

It is great to see someone picking up the visual data annotation problem seriously. However, you are obviously not the first one who has been trying that. I've tried and checked quiet a few data annotation tools in the past and I noticed they are all very weak if at best barely usable when it comes to doing segmentation masks.

As either they go one of the two potential routes:

*Only allow polygonal input to mask polygons

*Only allow painting input via a pixel brush

But both of them have issues. Paint brushes are cumbersome if you wan to mask large areas and take a lot of time. Polygons have an issue if you want to mark small and thin image regions as you simply would have to define so many points and often you can not just hit the edge correctly.

Best would be able to mix both of them so you can draw polygons and draw above the polygons in a multi layer fashion. Or if that is too complicated allow the option to bake the polygons down into the segmentation mask image and then have the option to use a paint brush to paint on top.

[SkalskiP]

Hi, @Mut1nyJD ! Thank you so much for bringing this subject up. One of the ideas for further development of the application is the introduction of labeling by drawing the mask with a brush. I'm not hiding that I planned to implement this functionality in the form of a separate tab in the panel on the right, so as another "type" of label. If I understand your thought correctly, it would be wrong to do it this way. Because these two ways should mix together?

What do you think about doing it a little bit like in MS Paint? It means: you can use both brush and polygon drawing in one functionality, BUT... If you finish editing your polygon, you will no longer be able to edit it as it is now. The polygon will be burned on the mask and the only way to interact with it is to erase it with the eraser.

It is worth to think about what should be the form of export of such a mask? I understand that in the form of a bitmap?

[Mut1nyJD]

@SkalskiP Thanks for your comments. I don't say it is wrong but I think it would be better if you could mix the two, as using a pixel brush even if it is large is really tedious if you have to label large areas (lets say background) but marking it with a polygon is relatively quickly done.

Doing it the MSPaint way is a compromise I guess most people could live with personally I think it ia nicer of course if you would be able to decide for yourself when to burn it onto the mask. But I am not sure how much flexibility of editing polygons you have already considered.

How do you handle multiple polygons on top of each other, right now? Classic example would be face I mark the face contour and now I want to label the eyes inside the face.

Yes segmentation mask are usually as a bitmap (a single channel just containing the labelnr or RGB where each label is assigned to a specific color)

[Marwen-Bhj]

Going back to this, since make-sense is an important tool for me that I use often, and I needed it for a segmentation task that required a coco style json output. I suggest implementing Simple Linear Iterative Clustering (SLIC) along with the polygons since Super Pixels comes super handy for having refined edges of complicated geometry. It will be definitely faster than adding multiple points while creating the polygon, and help avoiding the manual brushing.

[SkalskiP]

Hi, @Marwen-Bhj. Sounds like a very cool idea. Frankly speaking, I will have to examine this subject. Because I wonder what the transition from a bitmap to COCO JSON would have to be like. Other than that I would be happy to add it to our backlog. BTW If you use MakeSense for segmentation with COCO output, check out our new feature on our develop instance: https://develop.makesense.ai/ Now you can not only export to COCO but also import in COCO ;)