Query when preparing my dataset

[SUYEgit]

Hi, thanks for sharing your work. I am trying to implement the code for my own dataset. I read the article (https://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46) demonstrating by building the balloon dataset, however I don't understand how you make use of the 'region_attributes' information in the JSON file generated from VGG Image Annotator (VIA). In other words, I dont understand how you add label name into the dataset to tell the model the corresponding label of each bounding box. Is it because you only have balloon and it's a binary classification so you don't need the label name?

How should I modify the balloon.py if I have a dataset with multiple labels? I really appreciate your reading and help!

[zungam]

How should I modify the balloon.py if I have a dataset with multiple labels?

Labels are added in the load_mask function, it gets them from the json file. Just add the number of classes (labels) you use in the config (number of classes + background) and it shall work. Say you have cat and dog. Then the number should be 3, becuse you need to count background.

[SUYEgit]

Thanks for your comment. However, I didn't see any code reading the 'region_attributes' from the dataset. Could you please tell me how the load_mask function get the label information? I wish to understand the detail. It seems that labelled balloon images don't have labels in the first place.

[zungam]

As you see in the ballon example here, they do this: np.ones([mask.shape[-1]], dtype=np.int32) on the return statement from load_mask. It just a fast way to convert all the instances in the mask to a np list of [1,1,1,1,1....] where each "1" correspond to the label. Here 1="balloon". This is a bit of a lazy way, as they dont mind getting the balloon name from the json, they just assume each mask to be a "1". To see how you would do it with multiple classes I would recoment looking in the load_mask function in coco.py for inspiration.

Just to make things clear. If you had multiple classes, you would have something like [1,21,2,12,43,7,26]. Say you have a image with three instances of cats, and two dogs. The output would be [1,2,2,1,1] where 1="cat" and 2="dog". Where is these names saved? If you see in the load_ballon function here, you can see that they are adding all number references when they init the dataset. They do the same in coco.py, so you should look on that for reference with multiple classes.

[SUYEgit]

Wow. I really appreciate your detailed explanation! I will try to get more inspiration looking into the codes though I currently still don't really get it. Do I have to change np.ones([mask.shape[-1]], dtype=np.int32) for different classes referring to numbers when initializing the dataset? For example functions like load_cat, load dog.... Or do I only need to add the number of classes in config without other modification? Thank you so much!

[SUYEgit]

And when I am using VGG Image Annotation tool, should I care about the name given to each instance of one class? I am very confused about how the codes load labels from the annotations without using 'region_attributes' key. Is it because they only need balloon(which is only one class?)? Following is the example of the JSON file.

[zungam]

Do I have to change np.ones([mask.shape[-1]], dtype=np.int32) for different classes referring to numbers when initializing the dataset?

At the init of your dataset, call self.add_class(<name_of_dataset>, <your_number_id>,<your_corresponding_label_name>) for each existing class. This is so that the computer will know what "1" is everytime you reference it (or "2", "3" etc). When you then use load_mask, get the label from the json, and reference these numbers. Just use coco.py load_mask line 243 for inspiration.

Output from load_mask should be two values: A 3D numpy_bool_array with all mask instances, and a 1D array with all references (each mask position shall correspond to a reference).

I am very confused about how the codes load labels from the annotations without using 'region_attributes'

I guess the "region_attribute" is the name/class/label right? If so, you should modify the load_mask function to use "region_attribute" for getting the label.

I recomend using coco.py instead of balloon.py as a start.

[SUYEgit]

Great! Yes I mean 'region_attribute' by the label. The balloon.py is truly confusing. I will go through coco.py. Do you suggest me to directly modify the load_mask in coco.py rather than inheriting a class from utils.Dataset? Thanks a lot for your patience and time! It really helps a lot.

[zungam]

I copied coco.py to a new .py file and started from there.

Np

[SUYEgit]

Thanks a lot!

[SUYEgit]

I also found out that the script 'train_shapes.ipynb' very helpful. Hope it could also bring you some intuition!

[mia2mia]

@SUYENus @zungam Can either of you please please share a copy of your .py file. I'm still not able to implement the multiclass segmentation despite the different explanations

[EduardoTayupanta]

Utiliza la ultima version del VGG Image Annotation y cambia esta linea polygons = [r['shape_attributes'] for r in a['regions'].values()] a for r in a['regions']: polygons = [{'all_points_x': r['shape_attributes']['all_points_x'], 'all_points_y': r['shape_attributes']['all_points_y']}]

[amrbenattia]

@zungam any idea what size in annotation.josn file refer to ? I have create the annotation.json file. and for some reason i need to use it with new dataset (same dataset but some preprossing has done to it -same name, same extension and same dimensions but size in KB become different ). so when I import the annotation.json with the new dataset, the new images added not to the old names but to new names means I need to start annotation from the beginning . Any Idea How to avoid repeating the annotation and use the old annotation with the preprocced (new) dataset ?

Thanks in advance

[bramth]

I know this issue is kind of stale, but as it took me quite some time to figure this out, I hope it may benefit some others. Edit the following in your copy of balloon.py

    # Number of classes (including background)
    NUM_CLASSES = 1 + 2  # Background + class 1+ class 2

        # Add classes. We have only one class to add.
        self.add_class("balloon", 1, "class name 1")
        self.add_class("balloon", 2, "class name 2")

        #           'shape_attributes': {
        #               'all_points_x': [...],
        #               'all_points_y': [...],
        #               'classid': id,       # <-- I added this in an external tool. Probably via region_attributes is possible too, but I'm not using VIA for this. 
        #               'name': 'polygon'}},
        #       ... more regions ...
        #   },

        info = self.image_info[image_id]
        mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
                        dtype=np.uint8)
        classid = np.zeros([mask.shape[-1]], dtype=np.int32)

        for i, p in enumerate(info["polygons"]):
            # Get indexes of pixels inside the polygon and set them to 1
            rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
            mask[rr, cc, i] = 1
            if 'classid' in p:
                classid[i] = p['classid']
            else:
                classid[i] = 1

        return mask.astype(np.bool), classid