Conflict when using multiple classes in any image

[bryangreener]

I'm putting this here mostly so people can avoid this difficult to spot issue in the future.

When using this method of augmenting a dataset where images in the dataset can contain bounding boxes of different classes, it is possible that the classes in the resulting dataframe will be invalid. This is caused by the augmentation process when removing bboxes using .remove_out_of_image(). Since there is no link between the rows in the main dataframe and bbs_aug, using remove_out_of_image() can delete a bounding box from bbs_aug however it does not update the rows in the main dataframe to reflect this change. Thus when you bring the newly augmented bboxes back into the main dataframe, some of the rows may be shifted up but only for the positional values of the bboxes and not for any of the remaining columns of the dataframe.

To fix this, you need to somehow link the dataframe and the bbox augmentation arrays and remove any rows in the dataframe that contain bounding boxes that are removed using remove_out_of_image().

[bryangreener]

I seem to have resolved this issue by performing the following:

Replace

bb_array = group_df.drop(['filename', 'width', 'height', 'class'], axis=1).values
bbs = BoundingBoxesOnImage.from_xyxy_array(bb_array, shape=image.shape)

with

# BoundingBox takes format (x1=top_left_x, y1=top_left_y, x2=bottom_right_x, y2=bottom_right_y, label=class_name)
bb_array = [BoundingBox(row['xmin'], row['ymax'], row['xmax'], row['ymin'], label=row['class']) for _, row in group_df.iterrows()]
bbs = BoundingBoxesOnImage(bb_array, shape=image.shape)

in both the resize_imgaug and image_aug functions wherever they are used.

To use this new format, I also had to add "class" to the list of items dropped from the dataframe when creating the info_df dataframe.

info_df = group_df.drop(['class', 'xmin', 'ymin', 'xmax', 'ymax'], axis=1)

Finally I had to update the bbs_obj_to_df function as follows:

def bbs_obj_to_df(bbs_obj):
    bbs_arr = [[bb.x1, bb.y2, bb.x2, bb.y1, bb.label] for bb in bbs_obj.bounding_boxes]
    df_bbs = pd.DataFrame(bbs_arr, columns=['xmin', 'ymin', 'xmax', 'ymax', 'class'])
    return df_bbs

Even with all of that, I had to change the way that augmentations are performed on images and bounding boxes. This is because certain augmentations when performed in succession will not allow for proper removal of bboxes when they are out of the image. So to fix this I had to create a list of augmentations then iterate over that list and perform each augmentation manually, removing/clipping after each, then re-applying the augmented bboxes to the image and updating the bbs object. See below:

augmentors = [
        iaa.SomeOf((0, 1), [
            iaa.GaussianBlur(sigma=(1.0, 3.0)),
            iaa.AverageBlur(k=(1, 3)),
            iaa.MedianBlur(k=(1, 3)),
            iaa.imgcorruptlike.GlassBlur(severity=1),
            iaa.imgcorruptlike.DefocusBlur(severity=1),
            iaa.imgcorruptlike.MotionBlur(severity=1),
            #iaa.imgcorruptlike.ZoomBlur(severity=1) # too intense of transformation
        ], random_order=True),
        iaa.Sometimes(0.5, # these are more intense. want to reduce frequency
            iaa.SomeOf((0, 1), [
                iaa.imgcorruptlike.Fog(severity=1),
                iaa.imgcorruptlike.Frost(severity=1),
                iaa.imgcorruptlike.Snow(severity=1),
                iaa.imgcorruptlike.Spatter(severity=1)
            ], random_order=True)
        ),
        iaa.SomeOf((0, 2), [
            iaa.LinearContrast((0.75, 1.5)),
            iaa.imgcorruptlike.Brightness(severity=1),
            iaa.imgcorruptlike.Saturate(severity=1),
            iaa.imgcorruptlike.JpegCompression(severity=1),
            iaa.imgcorruptlike.Pixelate(severity=1)
        ], random_order=True),
        iaa.SomeOf((0, 1), [
            iaa.imgcorruptlike.ShotNoise(severity=1),
            iaa.imgcorruptlike.ImpulseNoise(severity=1),
            iaa.imgcorruptlike.SpeckleNoise(severity=1),
            iaa.AdditiveGaussianNoise(loc=0, scale=(0.03, 0.05*255), per_channel=0.5),
            iaa.AdditiveLaplaceNoise(loc=0, scale=(0.03, 0.05*255), per_channel=0.5),
            iaa.AdditivePoissonNoise(lam=(0.0, 5.0), per_channel=0.5)
        ], random_order=True),
        # use upward of 2 affine transforms to really mess up the image
         iaa.Sometimes(0.5, iaa.Affine(
            scale={'x': (0.8, 1.2), 'y': (0.8, 1.2)},
            translate_percent={'x': (-0.3, 0.3), 'y': (-0.3, 0.3)},
            rotate=(-10, 10),
            shear=(-16, 16),
            order=[0, 1],
            cval=(0, 255),
            mode=ia.ALL
        )),
         iaa.Sometimes(0.5, iaa.Affine(
            scale={'x': (0.8, 1.2), 'y': (0.8, 1.2)},
            translate_percent={'x': (-0.3, 0.3), 'y': (-0.3, 0.3)},
            rotate=(-10, 10),
            shear=(-16, 16),
            order=[0, 1],
            cval=(0, 255),
            mode=ia.ALL
        )),
        iaa.Sometimes(0.5, iaa.PerspectiveTransform(scale=(0.01, 0.15))),
        iaa.Sometimes(0.1, iaa.Multiply((0.8, 1.2), per_channel=0.2)),
        #iaa.Sometimes(0.1, iaa.Crop(percent=(0, 0.05))), # crop breaks bboxes
        iaa.Fliplr(0.25),
        iaa.Flipud(0.25),
]

and to use this list in the image_aug function...

from random import shuffle
def image_aug(.......):
    aug_bbs_xy = pd.DataFrame(columns=COLUMNS)
    grouped = df.groupby('filename')
    for filename in df['filename'].unique():
        group_df = grouped.get_group(filename)
        group_df = group_df.reset_index()
        group_df = group_df.drop(['index'], axis=1)
        image = imageio.imread(images_path + filename)
        # BoundingBox takes format (x1=top_left_x, y1=top_left_y, x2=bottom_right_x, y2=bottom_right_y, label=class_name)
        bb_array = [BoundingBox(row['xmin'], row['ymax'], row['xmax'], row['ymin'], label=row['class']) for _, row in group_df.iterrows()]
        bbs = BoundingBoxesOnImage(bb_array, shape=image.shape)

        shuffle(augmentors)
        for augmentor in augmentors:
            image_aug, bbs_aug = augmentor(image=image, bounding_boxes=bbs)
            bbs_aug = bbs_aug.remove_out_of_image_fraction(0.3) #removes any bbox with 30%+ outside of image 
            bbs_aug = bbs_aug.clip_out_of_image()
            bbs_aug = bbs_aug.on(image_aug)
            image = image_aug
            bbs = BoundingBoxesOnImage(bbs_aug.bounding_boxes, shape=image.shape)

        if re.findall('Image...', str(bbs_aug)) == ['Image([]']:
            pass
        else:
            imageio.imwrite(aug_images_path + image_prefix + filename, image_aug)
            info_df = group_df.drop(['class', 'xmin', 'ymin', 'xmax', 'ymax'], axis=1)
            for index, _ in info_df.iterrows():
                info_df.at[index, 'width'] = image_aug.shape[1]
                info_df.at[index, 'height'] = image_aug.shape[0]
            info_df['filename'] = info_df['filename'].apply(lambda x: image_prefix + x)
            bbs_df = bbs_obj_to_df(bbs_aug)
            aug_df = pd.concat([info_df, bbs_df], axis=1)
            aug_bbs_xy = pd.concat([aug_bbs_xy, aug_df])
    aug_bbs_xy = aug_bbs_xy.reset_index()
    aug_bbs_xy = aug_bbs_xy.drop(['index'], axis=1)
    aug_bbs_xy = aug_bbs_xy.dropna(axis='rows') # removed bboxes result in rows with NaN values. Need to drop rows.
    return aug_bbs_xy

What all of this does is it uses the imgaug.BoundingBox's label attribute to keep track of any bounding boxes that are deleted instead of using the "class" column in the dataframe. This is because the dataframe isn't updated at all when augmentations are performed and the "class" column is dynamic and can change from row to row. Thus when a bounding box is removed, the association between the row in the dataframe and the BoundingBox objects is broken. This new method ensures that the only information that is held in the dataframe during augmentations is information that is the same for every row in the dataframe.

[whiteRave1234]

hello when I changed the code from :

bbs = BoundingBoxesOnImage.from_xyxy_array(bb_array, shape=image.shape)

to :

bbs = BoundingBoxesOnImage(bb_array, shape=image.shape)

i got an error NameError: name 'image' is not defined, I thought it was a typo so I changed it to :

bbs = BoundingBoxesOnImage(bb_array, shape=images[9].shape)

and I got this error instead :

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-28-1517f7eb1763> in <module>
      2 bbs = BoundingBoxesOnImage(bb_array, shape=images[9].shape)
      3 # display the image and draw bounding boxes
----> 4 ia.imshow(bbs.draw_on_image(images[9], size=2))

~\anaconda3\envs\PascalVOCaugmentation\lib\site-packages\imgaug\augmentables\bbs.py in draw_on_image(self, image, color, alpha, size, copy, raise_if_out_of_image, thickness)
   1000                 copy=False,
   1001                 raise_if_out_of_image=raise_if_out_of_image,
-> 1002                 thickness=thickness
   1003             )
   1004 

~\anaconda3\envs\PascalVOCaugmentation\lib\site-packages\imgaug\augmentables\bbs.py in draw_on_image(self, image, color, alpha, size, copy, raise_if_out_of_image, thickness)
    578             rr, cc = skimage.draw.polygon_perimeter(y, x, shape=result.shape)
    579             if alpha >= 0.99:
--> 580                 result[rr, cc, :] = color
    581             else:
    582                 if ia.is_float_array(result):

IndexError: too many indices for array

would you mind sharing your code?