Augmentations history / Test Time Augmentation with imgaug

[virilo]

I'd like to do TTA with imgaug with a NN to predict bounding boxes.

So, at predicting time, let's say I'll predict 10 different versions (augmentations) of the image, and average the output.

Augmentations would include transformations like translate, zoom, etc.

Once the predictions have been done for the augmented images, how could I reverse transform the bounding boxes back to fit the original image?

For example, If imgaug a apply zoom factor of 1.15 and then shift the image 25 px to the left previous to the model.predict(...). Then the output bounding box should be first translated 25px to the right, and the apply a zoom of 1/1.15.

If it isn't implemented by imgaug yet, it could be enought to have the augmentations history in oder my script to apply to the bounding boxes

Thanks in advance

[aleju]

That is currently not supported by the library. The best one can do is to use non-stochastic parameters and manually invert them. Something comparable to the following might work:

image = ... # must be defined
model = ... # must be defined

bbs_pred = []

flip = [0.0, 1.0]
crop = [0, 5, 10]
scale = [1.0, 1.1, 1.2]
for flip_i, crop_i, scale_i in itertools.product(flip, crop, scale):
    flip_i_inv = 1.0 - flip_i
    crop_i_inv = -crop_i
    scale_i_inv = 1/scale_i
    seq = iaa.Sequential([
        iaa.Fliplr(flip_i),
        iaa.CropAndPad(px=crop_i),
        iaa.Affine(scale=scale_i)
    ])
    seq_inv = iaa.Sequential([
        iaa.Fliplr(flip_i_inv),
        iaa.CropAndPad(px=crop_i_inv),
        iaa.Affine(scale=scale_i_inv)
    ])
    seq_det = seq.to_deterministic()
    seq_inv_det = seq_inv.to_deterministic()
    image_aug = seq_det.augment_image(image)
    bbs_pred_i = model.predict(image_aug)
    bbs_pred_i_inv = seq_inv_det.augment_bounding_boxes(bbs_pred_i)
    bbs_pred.append(bbs_pred_i_inv)

# ... average bounding boxes ...

[EtienneDavid]

First, thank you for this answer ! Is it planned to implement TTA with imgaug ? I would find it very useful !

[aleju]

Will be added at some point. Not sure yet when, as the implementation might end up being difficult and hence time-consuming for some augmenters.

[EtienneDavid]

Thanks ! :)

[ejones18]

Will be added at some point. Not sure yet when, as the implementation might end up being difficult and hence time-consuming for some augmenters.

Hi, any plans to address this issue? Ta