Augmenting sets of images and sets of masks (replay only some transforms in a compose)

[JeremieHuteau]

I work with samples which are tuples of sets: a set of images and a set of masks form a single "row" of my dataset.

I want to apply augmentations that work on both images and masks with the same parameters to all elements of both sets (eg horizontally flip (or not) all images and masks), but apply augmentations that work only on a modality with different parameters for all elements (eg color jitter all images with different parameters).

How do I do that with albumentations? If it cannot be done natively, how should I go about implementing such an augmentation pipeline?

(actually, I'm working with sets of images and texts, and use transforms on text the way it's done in VirTex, but I'm expecting the answer not to depend on that)

[BloodAxe]

A.ReplayCompose looks like a way to go with you task.

Пн, 14 июня 2021 г. в 11:38, JeremieHuteau @.***>:

I work with samples which are tuples of sets: a set of images and a set of masks form a single "row" of my dataset.

I want to apply augmentations that work on both images and masks with the same parameters to all elements of both sets (eg horizontally flip (or not) all images and masks), but apply augmentations that work only on a modality with different parameters for all elements (eg color jitter all images with different parameters).

How do I do that with albumentations? If it cannot be done natively, how should I go about implementing such an augmentation pipeline?

(actually, I'm working with sets of images and texts, and use transforms on text the way it's done in VirTex https://github.com/kdexd/virtex/blob/master/virtex/data/transforms.py, but I'm expecting the answer not to depend on that)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/albumentations-team/albumentations/issues/920, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEB6YGONOL6IUHW5CPCWELTSW5YLANCNFSM46UYR3PQ .

[JeremieHuteau]

To apply the transformations the same way to all images and masks, it indeed seems to be the right tool. However, it is not obvious to me how it would help with only replaying the image&mask transforms but not the image only transforms in a Compose.

transform = A.Compose([
    A.RandomBrightnessContrast(p=0.2),
    A.HorizontalFlip(p=0.5),
    A.RGBShift(p=0.2),
])

data = transform(image=image, image0=image0, ..., imageN=imageN, mask=mask, mask0=mask0, ..., maskM=maskM)

With this transform, I want RandomBrightnessContrast and RGBShift to have different parameters for image, image0, ..., imageN, but HorizontalFlip to have the same parameter for image, ..., imageN and mask, ..., maskM. I feel like ReplayCompose would apply the same image transforms to all images, which is not what I want.

[BloodAxe]

Perhaps, you can split your transformation pipeline into two parts: 1) Photometric transformations, unique for each image: transform = A.Compose([ A.RandomBrightnessContrast(p=0.2), A.RGBShift(p=0.2),])

2) Spatial transformations, same for all images in a 'row' transform = A.Compose([A.HorizontalFlip(p=0.5),])

Then, it becomes way more easier to operate with your data. First, you apply augmentations independently for every image of your set (With this transform, I want RandomBrightnessContrast and RGBShift to have different parameters for image, image0, ..., imageN,) and secondly you can apply random flip to the whole set simultaneously using ReplayCompose or with image=image, image0=image0, ..., imageN=imageN, mask=mask, mask0=mask0, ..., maskM=maskM notation).

[Dipet]

Also you could to use additional_targets. Look example in the documentation https://albumentations.ai/docs/examples/example_multi_target/