Inconsistency in data augmentation using Albumentations

[wetoo-cando]

I noticed an inconsistency in data augmentation using Albumentations.

When you rotate the (cropped) image, Albumentations seems to fill up corner areas (which are empty due to the rotation) by mirroring pixels in the nearby area. In some situations with an object of interest near the image corner, the object in the image gets overwritten by mirrored pixels. However, the transformed keypoints corr. to the object still remain. This results in a training image with keypoints "apparently in the middle of nowhere".

This cannot be good for training. Has anyone noticed this before? For now I am setting A.Rotate(limit=0) (effectively removing the rotation randomization). A better way of course would be to remove the rotated keypoints if the object gets overwritten.

https://github.com/NVlabs/Deep_Object_Pose/blob/1655459de50cfcbf01f7d24775f834cab400aa25/scripts/train2/utils_dope.py#L258C8-L267C10

# data augmentation
transform = A.Compose(
    [
        A.RandomCrop(width=400, height=400),
        A.Rotate(limit=180),
        A.RandomBrightnessContrast(brightness_limit=0.1,contrast_limit=0.15,p=1),
        A.GaussNoise(p=1),

    ], 
    keypoint_params=A.KeypointParams(format='xy',remove_invisible=False)
)

[TontonTremblay]

This is a good catch, I did not think about the problems it could create. We should update the data pipeline, thank you.

[thohemp]

Hello, instead of removing the rotation augmentation completely, I suggest changing the border mode:

A.Rotate(limit=180, border_mode=cv2.BORDER_CONSTANT)

Instead of mirroring, this will just add a black border and should be less confusing during the training.

[wetoo-cando]

Thanks @thohemp !