Multi Class Semantic Segmentation using Binary Masks

[IkhlasAhmad1998]

@JordanMakesMaps @gazay @zcoder @habi @678098 @mathandy Hi, i need to perform multi label semantic segmentation using unet. but can't figure out how to preprocess my masks. i have an image in rgb format: with respective individual binary masks for 5 classes ( WITHOUT BACKGROUND / unlabeled pixels) with pixel value 1 for ROI and 0 for background:

1. so how to preprocess these masks to train a unet and is there any need to include background/unlabeled pixels as a mask so that the mask shape becomes (none, none, 6) ?
   2. can we merge all these masks into a single array where each pixel value is the index of class that is present...?
   3. do i need to use categorical crossentropy or something else?

have very little time. Thanks

[JordanMakesMaps]

So multi-label and multi-categorical classification are two different things. Multi-label means an object can have two different labels at the same time (e.g., a red shirt is both the color red and is a shirt). Multi-categorical is having multiple categories and an object belongs to only one of them. Here it looks like you're doing the latter.

If you want each class to be recognized by the model, you should train for each class. If there is a class you don't want to be learned, you can set it as a background class, and then in the loss function, set the class weights such that the background class' coefficient is set to 0.

To reshape the mask and get the format you're asking for, you can use Keras' to_categorical function.

You can use any loss function, but it makes since to use those that are for multi-categorical, such as CCE as you mentioned.

Hope this helps!

[IkhlasAhmad1998]

thanks @JordanMakesMaps , but what i actually mean is that i have individual binary masks for every class (also there is overlap among masks) so how should i preprocess them. just stack one upon other to create a mask of shape (none,none5) as 5 classes? does adding a new mask for background makes sense in this case..? Secondly, i am using sigmoid with total-loss and IOU. (should i change)..? and i have small number of images (471) which splitting ratio do i adopt for train-val and test...? lastly, does increasing number of images using augmentation helps..? brother i am running out of time for my project. kindly response asap (Appreciated) Thanks again.

[JordanMakesMaps]

i have individual binary masks for every class (also there is overlap among masks) so how should i preprocess them. just stack one upon other to create a mask of shape (none,none5) as 5 classes?

If you have 5 binary masks of the same image, and each mask contains only [0, 1], where 0 is not the class of interest and 1 is the class of interest then you should should stack them together, do an np.argmax(x, axis=2), and where there is overlap you'll have to figure something out (maybe stack the masks in order of importance, where the most important class is last so that when you do argmax, the last/most important class is given preference).

does adding a new mask for background makes sense in this case..?

If you perform the above stacking AND you still have pixel indices in the final mask that don't belong to any of the classes (for whatever reason), then that's you're background class. If you don't need your model to learn the background class, in the loss function set the class_weights list such that the background index is 0.

Secondly, i am using sigmoid with total-loss and IOU. (should i change)..?

I've found good results using Jaccard() + CCE()

and i have small number of images (471) which splitting ratio do i adopt for train-val and test...?

70/15/15? Try different ratios, you only have 471 images so it shouldn't take long to try 😜

lastly, does increasing number of images using augmentation helps..?

Typically yes, but you'll have to see so for yourself. Use imgaug for this, and make sure to read up on how to provide the same augmentations to both images and masks (if they don't line up then you'll have issues)

brother i am running out of time for my project.

Time machine?

Good luck!

[JordanMakesMaps]

Actually, you might not need to np.argmax, just stack the masks and you'll technically already have what you need in the shape that you need it.

[IkhlasAhmad1998]

@JordanMakesMaps thanks, but i have individual overlapping masks. should i use CCE or BCE..? thanks

[IkhlasAhmad1998]

@JordanMakesMaps i am unable to get good iou score using sigmoid activation even i have increased my dataset by augmentation. what could be the possible reasons..? provide some suggestions to improve my results thanks

[JordanMakesMaps]

@IkhlasAhmad1998 you have multiple classes (not binary), so you should be using CCE not BCE, and a softmax activation function, not a sigmoid.

[fatemehprhm]

Hi Could you get any results in this case? Can you share this part of your code?