Train on COCO Stuff + Things

[kevinj22]

Hello,

I wish to train on the 172 classes of COCO Stuff + Things https://github.com/nightrome/cocostuff.

I downloaded their 2017 training and validation COCO version annotations and added them to the proper folder.

I noticed you provide a function to combine the ROI's from multiple data sets, so I attempted to use this to combine the Stuff + Thing annotations to train with.

First I added a custom dataset to the dataset_catalog.py:

'coco_stuff_2017': { IM_DIR: _DATA_DIR + '/coco/images/train2017', ANN_FN: _DATA_DIR + '/coco/annotations/stuff_train2017.json' },

I then added a dataset argument to train_net_step.py

elif args.dataset == "coco_2017_train_both":
        cfg.TRAIN.DATASETS = ('coco_2017_train','coco_stuff_2017',)
        cfg.MODEL.NUM_CLASSES = 172

In which I provide multiple dataset arguments for the combined_roidb_for_training function to load.

This is the output when I run train_net_step.py:

loading annotations into memory... Done (t=8.68s) creating index... index created! INFO json_dataset.py: 298: Loading cached gt_roidb from /home/kevin/Documents/Segmentation/PANet-master/data/cache/coco_2017_train_gt_roidb.pkl INFO roidb.py: 50: Appending horizontally-flipped training examples... INFO roidb.py: 52: Loaded dataset: coco_2017_train loading annotations into memory... Done (t=7.53s) creating index... index created! INFO json_dataset.py: 298: Loading cached gt_roidb from /home/kevin/Documents/Segmentation/PANet-master/data/cache/coco_stuff_2017_gt_roidb.pkl INFO roidb.py: 50: Appending horizontally-flipped training examples... Traceback (most recent call last): File "train_net_step.py", line 457, in main() File "train_net_step.py", line 233, in main cfg.TRAIN.DATASETS, cfg.TRAIN.PROPOSAL_FILES) File "/home/kevin/Documents/Segmentation/PANet-master/lib/datasets/roidb.py", line 62, in combined_roidb_for_training roidbs = [get_roidb(args) for args in zip(dataset_names, proposal_files)] File "/home/kevin/Documents/Segmentation/PANet-master/lib/datasets/roidb.py", line 62, in roidbs = [get_roidb(args) for args in zip(dataset_names, proposal_files)] File "/home/kevin/Documents/Segmentation/PANet-master/lib/datasets/roidb.py", line 51, in get_roidb extend_with_flipped_entries(roidb, ds) File "/home/kevin/Documents/Segmentation/PANet-master/lib/datasets/roidb.py", line 107, in extend_with_flipped_entries entry['segms'], entry['height'], entry['width'] File "/home/kevin/Documents/Segmentation/PANet-master/lib/utils/segms.py", line 60, in flip_segms flipped_segms.append(_flip_rle(segm, height, width)) File "/home/kevin/Documents/Segmentation/PANet-master/lib/utils/segms.py", line 48, in _flip_rle mask = mask[:, ::-1, :] IndexError: too many indices for array

It indicates that coco_train_2017 loads correctly, but during the data augmentation the coco_stuff_2017 crashes.

Does anyone have an idea of how to fix this?

Thanks,

Kevin

[dongzhang89]

hi, have you solved this problem ? I meet the same one, but i do not know how to do.

[kevinj22]

I have. I added the following lines to segms at line 52:

 n = len(mask.shape)
    if n != 3:
        mask = np.expand_dims(mask,axis=2)

Also make sure the number of classes is 134, it's not actually 172 as a bunch of classes in stuff were merged. In the json_dataset.py file I added the following lines starting at line 80:

self.json_category_id_to_contiguous_id[0] = 0
        for i, v in enumerate(self.COCO.getCatIds()):
            self.json_category_id_to_contiguous_id[v] = i + 1

        # self.json_category_id_to_contiguous_id = {
        #     v: i + 1
        #     for i, v in enumerate(self.COCO.getCatIds())
        # }

        self.contiguous_category_id_to_json_id = {
            v: k
            for k, v in self.json_category_id_to_contiguous_id.items()
        }

There might have been some more changes, but I can't remember as it's been a while. If you run into any problems let me know.

Also if you check the panoptic segmentation evaluation results it seems like all the participants trained on things and stuff separately then merged their results. I'm not sure if this is the better approach. You could get a set of decent "things" results from https://github.com/ShuLiu1993/PANet and just train the "stuff" classes from the panoptic dataset if that's any easier.

[dongzhang89]

Thank you so much for your assistance, and I will try it.