Feature Question/Request : augment image according to mask

[julienguegan]

:fire: Feature Question/Request

Hello,

I would like to know if :books: Albumentation :books: is capable of augment only a certain part of the image, especially according to the mask. I did not find it in the documentation and I don't know if it is something usual when doing data augmentation but what I would like to do is, for my binary segmentation problem, augment only where the mask is. For example, if I want to segment a clothing someone is wearing, I was thinking that it would be useful to have different color version. I know that there might be some artefact generated compared to the original version but still I think it could be a cool thing to have ...

Does someone know if it exists already or if it is possible to integrate it ?

I actually know how to do it my self with python, but I am not sure on how to interface with albumentation ... Here, I change randomly the color where the mask is :

image[mask==1,:] = image[mask==1,:] + np.random.randint(low=-50,high=50,size=3)

Thank you !

[Dipet]

Interesting idea. But at this time I do not know how to do so that it would work with any transforms. For now you can use this wrapper:

import numpy as np
import albumentations as A
import matplotlib.pyplot as plt

class MaskWrapper(A.Sequential):
    def __call__(self, **data):
        image = data['image'].copy()

        data = super(MaskWrapper, self).__call__(**data)

        mask = data['mask'].astype(np.bool)
        image[mask] = data['image'][mask]
        data['image'] = image

        return data

transforms = A.Compose([
    MaskWrapper([A.GaussianBlur(p=1), A.RGBShift([50, 50], p=1)]),
    A.LongestMaxSize(512),
    MaskWrapper([A.HueSaturationValue(p=1)])
])

img = A.read_rgb_image("true.png")
mask = np.zeros_like(img)
mask[:500, :500] = 1

res = transforms(image=img, mask=mask)

plt.subplot(211, title="original")
plt.imshow(img, vmin=0, vmax=255)
plt.subplot(212, title="result")
plt.imshow(res['image'], vmin=0, vmax=255)
plt.show()

[julienguegan]

Ok Thank you. If I understand well, this way of doing would allow to apply color transformation only on image[mask]. And if I want to do it also when there is geometrics transformations, it might not work ? Is it possible do just apply transformations one after the other, like :

sample  = augmentation_geometric(image=image, mask=mask) # flip, scale, rotate, shift, perspective, distortion ...
sample  = augmentation_mask_color(image=sample['image'], mask=sample['mask']) # rgbshift where mask is, with MaskWrapper
sample  = augmentation_color(image=sample['image'], mask=sample['mask']) # GaussNoise, ISONoise, luminosity, blur, contrast ...

Though, it might increase the computation time ?

[Dipet]

In example I combine geometric(LongestMaxSize) and color transforms. At this time I do not know how apply geometric transforms like rotations and flips only in mask regions. If you will use my wrapper, you need to wrap transforms, that you want to apply only on mask region.

And yes, this modification increase computation time.

[julienguegan]

Ah yes ok ok, currently I was thinking to only change the color in mask region. Obviously, it seems difficult to apply a rotation only in mask region.

In any case, thank you for your answer. Do I close this issue with your maskwrapper solution ?

[Dipet]

I think it is good feature. So let it be open.