Error when using Cropping Augmentations from Albumentations

[abdksyed]

Thanks for your error report and we appreciate it a lot.

Checklist

1. I have searched related issues but cannot get the expected help.
2. I have read the FAQ documentation but cannot get the expected help.
3. The bug has not been fixed in the latest version.

Describe the bug A clear and concise description of what the bug is.

I am using Albumentations for Augmentations.

I copied the transforms and pipeline from the given Albumentations example on top of mask rcnn. The given example works fine, but the pipeline fails with an error when I add any Albumentation augmentation related to cropping (Added error traceback below)

Reproduction

1. What command or script did you run?

Here are the transformations and pipeline

albu_train_transforms = [
    dict(
        type='ShiftScaleRotate',
        shift_limit=0.0625,
        scale_limit=0.0,
        rotate_limit=0,
        interpolation=1,
        p=0.5),
    dict(
        type='RandomBrightnessContrast',
        brightness_limit=[0.1, 0.3],
        contrast_limit=[0.1, 0.3],
        p=0.2),
    dict(
        type='OneOf',
        transforms=[
            dict(
                type='RGBShift',
                r_shift_limit=10,
                g_shift_limit=10,
                b_shift_limit=10,
                p=1.0),
            dict(
                type='HueSaturationValue',
                hue_shift_limit=20,
                sat_shift_limit=30,
                val_shift_limit=20,
                p=1.0)
        ],
        p=0.1),
    # dict(type='CropNonEmptyMaskIfExists', height = 250, width = 300, p=1),
    dict(type='CenterCrop', height = 250, width = 300, p=1),
    # dict(type='BBoxSafeRandomCrop', erosion_rate=0.2),
    dict(type='JpegCompression', quality_lower=85, quality_upper=95, p=0.2),
    dict(type='ChannelShuffle', p=0.1),
    dict(
        type='OneOf',
        transforms=[
            dict(type='Blur', blur_limit=3, p=1.0),
            dict(type='MedianBlur', blur_limit=3, p=1.0)
        ],
        p=0.1),
]

train_pipeline = [
    dict(type='LoadImageFromFile', backend_args=None),
    dict(
        type='LoadAnnotations',
        with_bbox=True,
        with_mask=True),
        # poly2mask=True),
    # dict(type='RandomChoiceResize', 
    #      scales=[(1200, 1000), (900, 720), (450,300)]),
    dict(
        type='Albu',
        transforms=albu_train_transforms,
        bbox_params=dict(
            type='BboxParams',
            format='pascal_voc',
            label_fields=['gt_bboxes_labels', 'gt_ignore_flags'],
            min_visibility=0.0,
            filter_lost_elements=True),
        keymap={
            'img': 'image',
            'gt_masks': 'masks',
            'gt_bboxes': 'bboxes'
        },
        skip_img_without_anno=True),
    # dict(type='RandomFlip', prob=0.5),
    dict(type='PackDetInputs', 
         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape'))
]

2. Did you make any modifications on the code or config? Did you understand what you have modified?
3. What dataset did you use?

Environment

1. Please run python mmdet/utils/collect_env.py to collect necessary environment information and paste it here.
2. You may add addition that may be helpful for locating the problem, such as
   • How you installed PyTorch [e.g., pip, conda, source]
   • Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Error traceback

Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mmengine/dataset/dataset_wrapper.py", line 278, in __getitem__
    return self.dataset[sample_idx]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 413, in __getitem__
    data = self.prepare_data(idx)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 797, in prepare_data
    return self.pipeline(data_info)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 59, in __call__
    data = t(data)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mmcv/transforms/base.py", line 12, in __call__
    return self.transform(results)
  File "/home/ubuntu/rll-pa-seg/mmdetection/mmdet/structures/bbox/box_type.py", line 267, in wrapper
    return func(self, results)
  File "/home/ubuntu/rll-pa-seg/mmdetection/mmdet/datasets/transforms/transforms.py", line 1508, in transform
    results = self._postprocess_results(results, ori_masks)
  File "/home/ubuntu/rll-pa-seg/mmdetection/mmdet/datasets/transforms/transforms.py", line 1585, in _postprocess_results
    results['masks'] = ori_masks.__class__(
  File "/home/ubuntu/rll-pa-seg/mmdetection/mmdet/structures/mask/structures.py", line 271, in __init__
    self.masks = np.stack(masks).reshape(-1, height, width)
ValueError: cannot reshape array of size 450000 into shape (715,895)

The error is clear, the original image/mask is of size (715,895), but after applying the cropping augmentation, I have 5 masks cropped at 250x300. Which are being stacked and reshaped. But the height and weight when being reshaped is being taken from original shape from ori_mask object. The new height and width should be used since the mask has already been transformed using Cropping augmentation.

Hence the size 450000 should be reshaped to (250,300) rather than original shape of (715,895).

Bug fix If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

[abdksyed]

The workaround for this, which I am using for now is

def __init__(self, masks, height, width):
        self.height = height
        self.width = width
        if len(masks) == 0:
            self.masks = np.empty((0, self.height, self.width), dtype=np.uint8)
        else:
            assert isinstance(masks, (list, np.ndarray))
            if isinstance(masks, list):
                assert isinstance(masks[0], np.ndarray)
                assert masks[0].ndim == 2  # (H, W)
            else:
                assert masks.ndim == 3  # (N, H, W)

            # ----- CHANGES -----
            # self.masks = np.stack(masks).reshape(-1, height, width)
            self.masks = np.stack(masks)
            # assert self.masks.shape[1] == self.height
            # assert self.masks.shape[2] == self.width

            self.height = self.masks.shape[1]
            self.width = self.masks.shape[2]

I am overwriting the height and width parameter with the mask height and width.

[pkyzh2006]

The workaround for this, which I am using for now is

def __init__(self, masks, height, width):
        self.height = height
        self.width = width
        if len(masks) == 0:
            self.masks = np.empty((0, self.height, self.width), dtype=np.uint8)
        else:
            assert isinstance(masks, (list, np.ndarray))
            if isinstance(masks, list):
                assert isinstance(masks[0], np.ndarray)
                assert masks[0].ndim == 2  # (H, W)
            else:
                assert masks.ndim == 3  # (N, H, W)

            # ----- CHANGES -----
            # self.masks = np.stack(masks).reshape(-1, height, width)
            self.masks = np.stack(masks)
            # assert self.masks.shape[1] == self.height
            # assert self.masks.shape[2] == self.width

            self.height = self.masks.shape[1]
            self.width = self.masks.shape[2]

I am overwriting the height and width parameter with the mask height and width.

Now it can run but all mask lose turn to 0. So werid

[abdksyed]

I don't remember the loss going to 0. It's been over a year, so I don't remember much, but I am sure I was able to train the model.

Maybe you can visualise the actual mask it is creating.