Balloon Sample: What if we want to train for more than one class

[AliceDinh]

In coco.py, annotation format is coco annotation format. In balloon.py, the annotation is achieved by VIA which is different to coco format. In shapes.py, the annotation is not needed coz there is no dataset I successfully trained, evaluated and tested all samples and notebooks (coco, balloon, shapes) and I also created my own dataset, my annotation by VIA. I could train the model using my own dataset successfully, but only for one class (one category). I need to train for more class and I did try to modify funtions: load_objects() and load_mask() without success. So please help.

[waleedka]

Since you're using VIA, then BalloonDataset is the closest to your case, so I'll use it as a reference.

The key change is to modify load_mask() to return masks that belong to different classes. Look at the last line of this function:

        # Return mask, and array of class IDs of each instance. Since we have
        # one class ID only, we return an array of 1s
        return mask, np.ones([mask.shape[-1]], dtype=np.int32)

The function returns an array of masks and an array of class IDs. The class IDs are all ones because we have one class. You need to modify it to return an array of class IDs where each class_id is either 1 or 2 (since you want to support two classes).

Now, where does load_mask get the mask details from in the first place? It depends on your dataset. In some cases, like in coco.py, it loads the masks from a file. But in balloon.py, it reads the polygon points from self.image_info, which was populated in the load_balloon(). So let's look at that function.

In load_balloon() go to this line:

        self.add_image(
            "balloon",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons)

This is where it adds every image to the dataset. The first three parameters are standard (dataset name, image_id, and image path). The next three are added so they can be used by load_mask() (width, height, and polygons). You just need to also add class_ids and pass a list of IDs that correspond to the list of polygons you're passing. You can get the class IDs from the annotations JSON. I think it's in the category of each region.

Once you get the list of class IDs (and verify it's sorted in the same order as the list of polygons so they match) then you simply pass it in add_image:

        self.add_image(
            "balloon",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons,
            class_ids=class_ids)

And then update the last line of load_mask to use the newly added info:

        # Return mask, and array of class IDs of each instance.
        return mask, info['class_ids']

[AliceDinh]

@waleedka Thanks for everything. I successfully made it. <3

[zxczhkzxc]

@AliceDinh hello,i meet the same problem,can you share your functions about load_objects() and load_mask(),thank you

[derelearnro]

Please see my function below that maybe usefull, currently I have 13 classes, including the BG.

def load_multi_number(self, dataset_dir, subset): """Load a subset of the number dataset. dataset_dir: Root directory of the dataset. subset: Subset to load: train or val """

Add classes

    self.add_class("object", 1, "A")
    self.add_class("object", 2, "B")
    self.add_class("object", 3, "C")
    self.add_class("object", 4, "D")
    self.add_class("object", 5, "E")
    self.add_class("object", 6, "F")
    self.add_class("object", 7, "G")
    self.add_class("object", 8, "H")
    self.add_class("object", 9, "I")
    self.add_class("object", 10, "J")
    self.add_class("object", 11, "K")    
    self.add_class("object", 12, "browl")

    # Train or validation dataset?
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    annotations = json.load(open(os.path.join(dataset_dir, ".../train/via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    # The VIA tool saves images in the JSON even if they don't have any
    # annotations. Skip unannotated images.
    annotations = [a for a in annotations if a['regions']]

    # Add images
    for a in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. There are stores in the
        # shape_attributes (see json format above)
        # for b in a['regions'].values():
        #    polygons = [{**b['shape_attributes'], **b['region_attributes']}]
        # print("string=", polygons)
        # for r in a['regions'].values():
        #    polygons = [r['shape_attributes']]
        #    # print("polygons=", polygons)
        #    multi_numbers = [r['region_attributes']]
            # print("multi_numbers=", multi_numbers)
        polygons = [r['shape_attributes'] for r in a['regions'].values()]
        objects = [s['region_attributes'] for s in a['regions'].values()]
        # print("multi_numbers=", multi_numbers)
        # num_ids = [n for n in multi_numbers['number'].values()]
        # for n in multi_numbers:
        num_ids = [int(n['object']) for n in objects]
        # print("num_ids=", num_ids)
        # print("num_ids_new=", num_ids_new)
        # categories = [s['region_attributes'] for s in a['regions'].values()]
        # load_mask() needs the image size to convert polygons to masks.
        # Unfortunately, VIA doesn't include it in JSON, so we must read
        # the image. This is only managable since the dataset is tiny.
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "object",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons,
            num_ids=num_ids)

def load_mask(self, image_id):
    """Generate instance masks for an image.
   Returns:
    masks: A bool array of shape [height, width, instance count] with
        one mask per instance.
    class_ids: a 1D array of class IDs of the instance masks.
    """
    # If not a number dataset image, delegate to parent class.
    info = self.image_info[image_id]
    if info["source"] != "object":
        return super(self.__class__, self).load_mask(image_id)
    num_ids = info['num_ids']
    # Convert polygons to a bitmap mask of shape
    # [height, width, instance_count]
    mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
                    dtype=np.uint8)

    for i, p in enumerate(info["polygons"]):
        # Get indexes of pixels inside the polygon and set them to 1
        rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
        mask[rr, cc, i] = 1
    # print("info['num_ids']=", info['num_ids'])
    # Map class names to class IDs.
    num_ids = np.array(num_ids, dtype=np.int32)
    return mask, num_ids

def image_reference(self, image_id):
    """Return the path of the image."""
    info = self.image_info[image_id]
    if info["source"] == "object":
        return info["path"]
    else:
        super(self.__class__, self).image_reference(image_id)

[zxczhkzxc]

@ldthan many thanks

[hwjung92]

ldthan, Thank you for the providing code.

[chohaku84]

many thanks, @waleedka and @ldthan

[domtx]

Guys, I got the code working with multiple classes. I have some issues with the json file though. I am using VIA. Could somebody please post a snippet of a json file for multiple classes. Thanks.

[derelearnro]

Look at my simple file here:

{"Webp.net-resizeimage (24).jpg422460":{"fileref":"","size":422460,"filename":"Webp.net-resizeimage (24).jpg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[529,647,665,499,529],"all_points_y":[305,313,554,548,305]},"region_attributes":{"person":"2"}},"1":{"shape_attributes":{"name":"polygon","all_points_x":[544,631,618,545,544],"all_points_y":[317,325,397,394,317]},"region_attributes":{"person":"1"}}}},"Webp.net-resizeimage (25).jpg495440":{"fileref":"","size":495440,"filename":"Webp.net-resizeimage (25).jpg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[328,436,486,289,328],"all_points_y":[242,254,541,546,242]},"region_attributes":{"person":"2"}},"1":{"shape_attributes":{"name":"polygon","all_points_x":[344,430,426,350,344],"all_points_y":[262,267,335,334,262]},"region_attributes":{"person":"1"}}}},"Webp.net-resizeimage (26).jpg442743":{"fileref":"","size":442743,"filename":"Webp.net-resizeimage (26).jpg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[328,448,514,311,328],"all_points_y":[273,274,630,629,273]},"region_attributes":{"person":"2"}},"1":{"shape_attributes":{"name":"polygon","all_points_x":[366,422,436,367,366],"all_points_y":[301,297,369,366,301]},"region_attributes":{"person":"1"}}}}}

[domtx]

Thanks very much @ldthan ... Appreciated.

[domtx]

I still have a problem ... the code below 'works' in the sense that I get no error messages, but after the print "Training network heads" I get to Epoch 1/30 and than the app kind of stalls. I do see virtually no activity on the GPU but I do see the python threads via ps. If I train with 1 class (basically the balloon example) all works and I do see major activity on the GPU. Hope somebody can tell me what I am doing wrong. Thanks!

-> The code I use (based on @ldthan):

############################################################

Configurations

############################################################

class VehicleConfig(Config): """Configuration for training on the toy dataset. Derives from the base Config class and overrides some values. """

Give the configuration a recognizable name

NAME = "vehicle"

# We use a GPU with 12GB memory, which can fit two images.
# Adjust down if you use a smaller GPU.
IMAGES_PER_GPU = 2

# Number of classes (including background)
NUM_CLASSES = 1 + 2

# Number of training steps per epoch
STEPS_PER_EPOCH = 100

# Skip detections with < 90% confidence
DETECTION_MIN_CONFIDENCE = 0.9

############################################################

Dataset

############################################################

class VehicleDataset(utils.Dataset):

def load_vehicle(self, dataset_dir, subset):
    """Load a subset of the Vehicle dataset.
    dataset_dir: Root directory of the dataset.
    subset: Subset to load: train or val
    """
    # Add classes. We have 2 classes
    self.add_class("vehicle", 1, "truck")
    self.add_class("vehicle", 2, "bus")

    # Train or validation dataset?
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    # The VIA tool saves images in the JSON even if they don't have any
    # annotations. Skip unannotated images.
    annotations = [a for a in annotations if a['regions']]

    # Add images
    for a in annotations:
        polygons = [r['shape_attributes'] for r in a['regions'].values()]
        objects = [s['region_attributes'] for s in a['regions'].values()]
        print("objects=", objects)
        num_ids = [int(n['vehicle']) for n in objects]
        print("num_ids=", num_ids)
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "vehicle",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons,
            num_ids=num_ids)

def load_mask(self, image_id): """Generate instance masks for an image. Returns: masks: A bool array of shape [height, width, instance count] with one mask per instance. class_ids: a 1D array of class IDs of the instance masks. """

If not a number dataset image, delegate to parent class.

info = self.image_info[image_id]
if info["source"] != "vehicle":
    return super(self.__class__, self).load_mask(image_id)
num_ids = info['num_ids']
# Convert polygons to a bitmap mask of shape
# [height, width, instance_count]
mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
                dtype=np.uint8)

for i, p in enumerate(info["polygons"]):
    # Get indexes of pixels inside the polygon and set them to 1
    rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
    mask[rr, cc, i] = 1
# Map class names to class IDs.
num_ids = np.array(num_ids, dtype=np.int32)
return mask, num_ids

def image_reference(self, image_id): """Return the path of the image.""" info = self.image_info[image_id] if info["source"] == "vehicle": return info["path"] else: super(self.class, self).image_reference(image_id)

def train(model): """Train the model."""

Training dataset.

dataset_train = VehicleDataset()
dataset_train.load_vehicle(args.dataset, "train")
dataset_train.prepare()

# Validation dataset
dataset_val = VehicleDataset()
dataset_val.load_vehicle(args.dataset, "val")
dataset_val.prepare()

# *** This training schedule is an example. Update to your needs ***
# Since we're using a very small dataset, and starting from
# COCO trained weights, we don't need to train too long. Also,
# no need to train all layers, just the heads should do it.
print("Training network heads")
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=30,
            layers='heads')

[domtx]

A snippet of my VIA json file is here:

{"DSCN0931.JPG7959217":{"fileref":"","size":7959217,"filename":"DSCN0931.JPG","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[ 488,325,1239,1007,3168,3339,4431,4571,488],"all_points_y":[697,1449,1658,2456,2928,1975,2146,1294,697]},"region_attributes":{"vehicle":"1"}}}},"DSCN3522.JPG4203101":{"fileref":"","size":420 3101,"filename":"DSCN3522.JPG","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[992,961,2099,2146,2099,2146,1495,1526,984,992], "all_points_y":[968,1239,1704,1394,1363,999,883,798,635,968]},"region_attributes":{"vehicle":"2"}},"1":{"shape_attributes":{"name":"polygon","all_points_x":[2944,2843,4888,4865,2944],"all_p oints_y":[1689,2061,2983,2487,1689]},"region_attributes":{"vehicle":"1"}}}},"DSCN3512.JPG4229421":{"fileref":"","size":4229421,"filename":"DSCN3512.JPG","base64_img_data":"","file_attribute s":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[674,751,2448,2363,2208,1487,1472,612,674],"all_points_y":[1209,1487,1278,999,728,868,829,930,1209]},"region_attrib utes":{"vehicle":"2"}},"1":{"shape_attributes":{"name":"polygon","all_points_x":[3029,3106,4183,4028,3029],"all_points_y":[922,1170,1015,813,922]},"region_attributes":{"vehicle":"1"}},"2":{ "shape_attributes":{"name":"polygon","all_points_x":[4493,4617,4888,4826,4493],"all_points_y":[767,953,891,713,767]},"region_attributes":{"vehicle":"1"}}}},"DSCN3521.JPG4267617":{"fileref": "","size":4267617,"filename":"DSCN3521.JPG","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[1821,1743,3021,3068,1821],"all_poi nts_y":[1766,2216,2448,1975,1766]},"region_attributes":{"vehicle":"1"}},"1":{"shape_attributes":{"name":"polygon","all_points_x":[4222,4245,4873,4888,4222],"all_points_y":[2177,2618,2696,23 01,2177]},"region_attributes":{"vehicle":"1"}}}}}

[rahul-321]

I did the same thing as stated by @ldthan but I am getting an error -

Traceback (most recent call last): File "student.py", line 262, in train(model) File "student.py", line 112, in train dataset_train.load_student(args.dataset, "train") File "student.py", line 62, in load_student num_ids = [int(n['student']) for n in objects] File "student.py", line 62, in num_ids = [int(n['student']) for n in objects] KeyError: 'student'

I have 3 classes - BG, Student, bag

class StudentDataset(utils.Dataset):

def load_student(self, dataset_dir, subset):
    """Load a subset of the Balloon dataset.
    dataset_dir: Root directory of the dataset.
    subset: Subset to load: train or val
    """
    # Add classes. We have only one class to add.
    self.add_class("student", 1, "student")

    # Train or validation dataset?
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    # Load annotations
    # VGG Image Annotator saves each image in the form:
    # { 'filename': '28503151_5b5b7ec140_b.jpg',
    #   'regions': {
    #       '0': {
    #           'region_attributes': {},
    #           'shape_attributes': {
    #               'all_points_x': [...],
    #               'all_points_y': [...],
    #               'name': 'polygon'}},
    #       ... more regions ...
    #   },
    #   'size': 100202
    # }
    # We mostly care about the x and y coordinates of each region
    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    # The VIA tool saves images in the JSON even if they don't have any
    # annotations. Skip unannotated images.
    annotations = [a for a in annotations if a['regions']]

    # Add images
    for a in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. There are stores in the
        # shape_attributes (see json format above)
        polygons = [r['shape_attributes'] for r in a['regions'].values()]
        students = [s['region_attributes'] for s in a['regions'].values()]

        num_ids = [int(n['student']) for n in students]

        # load_mask() needs the image size to convert polygons to masks.
        # Unfortunately, VIA doesn't include it in JSON, so we must read
        # the image. This is only managable since the dataset is tiny.
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "student",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons,
            num_ids = num_ids)

def load_mask(self, image_id):
    """Generate instance masks for an image.
   Returns:
    masks: A bool array of shape [height, width, instance count] with
        one mask per instance.
    class_ids: a 1D array of class IDs of the instance masks.
    """
    # If not a balloon dataset image, delegate to parent class.
    image_info = self.image_info[image_id]
    if image_info["source"] != "student":
        return super(self.__class__, self).load_mask(image_id)
    num_ids = info['num_ids']

    # Convert polygons to a bitmap mask of shape
    # [height, width, instance_count]
    info = self.image_info[image_id]
    mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
                    dtype=np.uint8)
    for i, p in enumerate(info["polygons"]):
        # Get indexes of pixels inside the polygon and set them to 1
        rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
        mask[rr, cc, i] = 1

    # Return mask, and array of class IDs of each instance. Since we have
    # one class ID only, we return an array of 1s
    #return mask.astype(np.bool), np.ones([mask.shape[-1]], dtype=np.int32)
    return mask, num_ids

[mia2mia]

I am getting the same error as @rahul-321. @rahul-321 @ldthan how did you avoid this?

[rahul-321]

@mia2mia

Here is the code which worked for me -

def load_student(self, dataset_dir, subset):
    self.add_class("student", 1, "student")
    self.add_class("student", 2, "bag")

    # Train or validation dataset?
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    annotations = [a for a in annotations if a['regions']]

        # Add images
    for a in annotations:
        polygons = [r['shape_attributes'] for r in a['regions'].values()]
        objects = [s['region_attributes'] for s in a['regions'].values()]

        print(objects)
        num_ids=[]
        for n in objects:
            #print(n)
            #print(type(n))
            try:
                if n['object_name']=='student':
                    num_ids.append(1)
                elif n['object_name']=='bag':
                    num_ids.append(2)
            except:
                pass

        #num_ids = [int(n['object_name']) for n in objects]
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "student",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons,
            num_ids=num_ids)

I have 2 classes student and bag. Each class has only one region attribute which is 'object_name'

Now consider this line in the code -

objects = [s['region_attributes'] for s in a['regions'].values()]

When I run my code, the object is of the type dict and the key is the region attribute i.e object_name and the value is the object_name's values i.e either bag or student.

Apparently, in the code referred by @ldthan objects is not a dictionary. This I guess happens because of the type of annotator one is using, I am using Vgg annotator version 1.0.4. So the code given by me will work. For version Vgg version 2, this might not work and instead, the code given by @ldthan may work.

[bellakate]

@rahul-321 can u please share your load mask function?

[kalanka29]

@derelearnro : I did as yours but it gives the following error converting sparse indexedslices to a dense tensor of unknown shape (Note that I have used about 514 images for both training and testing ).What may be the case... https://stackoverflow.com/questions/51764713/mask-rcnn-not-working-for-images-with-large-resolution And also the my annotation json is with only x and y values.(not as all_points_y and all_points_x

[aakashkodes]

Hi @AliceDinh Can you please share the code which worked for you.

[AliceDinh]

@aakashkodes I gave @ldthan the code and he already posted above (not sure his account on Github is @ldthan or @derelearnro)

[synioe]

@aakashkodes I gave @ldthan the code and he already posted above (not sure his account on Github is @ldthan or @derelearnro)

@AliceDinh excuse me,have you run succed with the code mentioned above?I find some issues and cannot work successfully.If you find a solution, could you please push it online?

[AliceDinh]

@synioe Yes I already run it successfully. What issues do you have, may I know more specific?

[synioe]

@AliceDinh morning alice! Since I use the VIA2.0.2,the json file have some different format as those use VIA1.0.x,such as the polygons = [r['shape_attributes'] for r in a['regions'].values()] objects = [s['region_attributes'] for s in a['regions'].values()]I should use polygons = [r['shape_attributes'] for r in a['regions']] objects = [s['region_attributes'] for s in a['regions']]to get the polygons and objects,then for multi number class,in the load_objects function,I don't know exactly how to get the class_id deliver to add_image() and I try several methods just can not make it,could you give some advice on how to get the class_idin the load_object function with the VIA2.0 json file format.

here is one attempt I made just now: ` def load_object(self, dataset_dir, subset):

    # Add classes. We have 6 classes to add.
    self.add_class("object", 1, "porosity")
    self.add_class("object", 2, "crack")
    self.add_class("object", 3, "porosity array")
    self.add_class("object", 4, "lacking of sintering")
    self.add_class("object", 5, "surface hollow")
    self.add_class("object", 6, "surface scratch")

    # Train or validation dataset
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    # Load annotations
    # We mostly care about the x and y coordinates of each region
    # Note: In VIA 2.0, regions was changed from a dict to a list.
    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    # The VIA tool saves images in the JSON even if they don't have any
    # annotations. Skip unannotated images.
    annotations = [a for a in annotations if a['regions']]
    # print("ANNOTATIONS INFO:", annotations)
    idlist = []
    # Add images
    for a in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. These are stores in the
        # shape_attributes (see json format above)
        # The if condition is needed to support VIA versions 1.x and 2.x.
        # print(type(a['regions'])) #list
        '''if type(a['regions']) is dict:
            polygons = [r['shape_attributes'] for r in a['regions'].values()]
        else:
            polygons = [r['shape_attributes'] for r in a['regions']]'''

        polygons = [r['shape_attributes'] for r in a['regions']]
        name = [r['region_attributes']['type'] for r in a['regions']]
        #print("[NAME INFO:]", name)
        #print(type(name)) #list
        '''
        [NAME INFO:] [{'porosity': True}, {'porosity': True}, {'surface hollow': True}, 
        {'porosity': True}, {'lacking of sintering': True}, {'porosity': True}, 
        {'porosity': True}, {'porosity': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}]
        <class 'list'>
        '''
        name_dict = {"porosity": 1,
                     "crack": 2,
                     "porosity array": 3,
                     "lacking of sintering": 4,
                     "surface hollow": 5,
                     "surface scratch": 6}
        name_id = [name_dict[a] for a in name]

        '''class_names = [r['region_attributes'] for r in a['regions']]
        for i in range(len(class_names)):
            if class_names[i]["type"] == "porosity":
                idlist.append(1)
            elif class_names[i]["type"] == "crack":
                idlist.append(2)
            elif class_names[i]["type"] == "porosity":
                idlist.append(3)
            elif class_names[i]["type"] == "lacking of sintering":
                idlist.append(4)
            elif class_names[i]["type"] == "surface hollow":
                idlist.append(5)
            elif class_names[i]["type"] == "surface scratch":
                idlist.append(6)'''

        # load_mask() needs the image size to convert polygons to masks.
        # Unfortunately, VIA doesn't include it in JSON, so we must read
        # the image. This is only managable since the dataset is tiny.
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "object",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            class_id=name_id,
            width=width, height=height,
            polygons=polygons)`

then I get the error Traceback (most recent call last): File "object.py", line 407, in <module> train(model) File "object.py", line 226, in train dataset_train.load_defect(args.dataset, "train") File "object.py", line 143, in load_defect name_id = [name_dict[a] for a in name] File "object.py", line 143, in <listcomp> name_id = [name_dict[a] for a in name] TypeError: unhashable type: 'dict' So how can I solve it?

[synioe]

@AliceDinh And I find above someone said that @ldthan gave a solution for the VIA2.0,but I cannot find his code anymore.Could you please share the code mentions above by @ldthan?thank U very much.

[amullapu]

when trying to add a class. I'm getting the following error and I have attached the code and json from VGG annotator version-1 below for reference @AliceDinh @waleedka @derelearnro @rahul-321. Can you guys tell me what is the mistake

Error: Epoch 1/30 ERROR:root:Error processing image {'id': '2.jpeg', 'source': 'object', 'path': '/root/Code/test/Image_segmented/Dataset/val/2.jpeg', 'width': 225, 'height': 225, 'polygons': [{'name': 'polygon', 'all_points_x': [47, 85, 93, 97, 102, 111, 112, 112, 112, 113, 198, 218, 219, 207, 110, 110, 101, 93, 85, 51, 32, 33, 15, 14, 35, 42, 47], 'all_points_y': [69, 70, 75, 82, 86, 86, 89, 92, 93, 94, 95, 113, 114, 129, 127, 134, 135, 143, 150, 151, 147, 146, 110, 106, 74, 70, 69]}], 'num_ids': []} Traceback (most recent call last): File "/root/Code/test/mrcnn/model.py", line 1710, in data_generator use_mini_mask=config.USE_MINI_MASK) File "/root/Code/test/mrcnn/model.py", line 1266, in load_image_gt class_ids = class_ids[_idx] IndexError: boolean index did not match indexed array along dimension 0; dimension is 0 but corresponding boolean dimension is 1

99/100 [============================>.] - ETA: 2s - loss: 0.9285 - rpn_class_loss: 0.0016 - rpn_bbox_loss: 0.1541 - mrcnn_class_loss: 0.0748 - mrcnn_bbox_loss: 0.5152 - mrcnn_mask_loss: 0.1827multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 626, in next_sample return six.next(_SHARED_SEQUENCES[uid]) File "/root/Code/test/mrcnn/model.py", line 1710, in data_generator use_mini_mask=config.USE_MINI_MASK) File "/root/Code/test/mrcnn/model.py", line 1266, in load_image_gt class_ids = class_ids[_idx] IndexError: boolean index did not match indexed array along dimension 0; dimension is 0 but corresponding boolean dimension is 1 """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "balloon.py", line 381, in train(model) File "balloon.py", line 216, in train layers='heads') File "/root/Code/test/mrcnn/model.py", line 2375, in train use_multiprocessing=True, File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1418, in fit_generator initial_epoch=initial_epoch) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py", line 234, in fit_generator workers=0) File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1472, in evaluate_generator verbose=verbose) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py", line 330, in evaluate_generator generator_output = next(output_generator) File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 709, in get six.reraise(sys.exc_info()) File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise raise value File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 685, in get inputs = self.queue.get(block=True).get() File "/usr/lib/python3.6/multiprocessing/pool.py", line 670, in get raise self._value IndexError: boolean index did not match indexed array along dimension 0; dimension is 0 but corresponding boolean dimension is 1.

code:

############################################################

Configurations

############################################################

class objectConfig(Config): """Configuration for training on the toy dataset. Derives from the base Config class and overrides some values. """

Give the configuration a recognizable name

NAME = "object"

# We use a GPU with 12GB memory, which can fit two images.
# Adjust down if you use a smaller GPU.
IMAGES_PER_GPU = 2

# Number of classes (including background)
NUM_CLASSES = 1 + 2  # Background + balloon

# Number of training steps per epoch
STEPS_PER_EPOCH = 100

# Skip detections with < 90% confidence
DETECTION_MIN_CONFIDENCE = 0.9

############################################################

Dataset

############################################################

class objectDataset(utils.Dataset):

def load_object(self, dataset_dir, subset):
    """Load a subset of the Balloon dataset.
    dataset_dir: Root directory of the dataset.
    subset: Subset to load: train or val
    """
    # Add classes. We have only one class to add.
    self.add_class("object", 1, "Key")
    self.add_class("object", 2, "Image")

    # Train or validation dataset?
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    # Load annotations
    # VGG Image Annotator (up to version 1.6) saves each image in the form:
    # { 'filename': '28503151_5b5b7ec140_b.jpg',
    #   'regions': {
    #       '0': {
    #           'region_attributes': {},
    #           'shape_attributes': {
    #               'all_points_x': [...],
    #               'all_points_y': [...],
    #               'name': 'polygon'}},
    #       ... more regions ...
    #   },
    #   'size': 100202
    # }
    # We mostly care about the x and y coordinates of each region
    # Note: In VIA 2.0, regions was changed from a dict to a list.
    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    # The VIA tool saves images in the JSON even if they don't have any
    # annotations. Skip unannotated images.
    annotations = [a for a in annotations if a['regions']]

    # Add images
    for a in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. These are stores in the
        # shape_attributes (see json format above)
        # The if condition is needed to support VIA versions 1.x and 2.x.
        polygons = [r['shape_attributes'] for r in a['regions'].values()]
        objects = [s['region_attributes'] for s in a['regions'].values()]

        #num_ids=[int(n['object']) for n in objects]
        num_ids = []
        for n in objects:
            try:
                if 'Key' in n.keys():
                    num_ids.append(1)

                elif 'Image' in n.keys():
                    num_ids.append(2)

            except:
                pass

        # load_mask() needs the image size to convert polygons to masks.
        # Unfortunately, VIA doesn't include it in JSON, so we must read
        # the image. This is only managable since the dataset is tiny.
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "object",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons,
            num_ids=num_ids)

def load_mask(self, image_id):
    """Generate instance masks for an image.
   Returns:
    masks: A bool array of shape [height, width, instance count] with
        one mask per instance.
    class_ids: a 1D array of class IDs of the instance masks.
    """
    # If not a balloon dataset image, delegate to parent class.
    info = self.image_info[image_id]
    if info["source"] != "object":
        return super(self.__class__, self).load_mask(image_id)
    num_ids = info['num_ids']

    # Conver polygons to a bitmap mask of shape
    # [height, width, instance_count]
    #info = self.image_info[image_id]
    mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
                    dtype=np.uint8)
    for i, p in enumerate(info["polygons"]):
        # Get indexes of pixels inside the polygon and set them to 1
        rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
        mask[rr, cc, i] = 1
    print(info['num_ids'])
    #num_ids = np.array(num_ids, dtype=np.int32)
    # Return mask, and array of class IDs of each Since we have
    # one class ID only, we return an array of 1s
    #return mask.astype(np.bool), np.ones([mask.shape[-1]], dtype=np.int32)
    return mask,num_ids

def image_reference(self, image_id):
    """Return the path of the image."""
    info = self.image_info[image_id]
    if info["source"] == "object":
        return info["path"]
    else:

super(self.class, self).image_reference(image_id)

*json snippet of Class Image and Key***

19.jpeg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[46,57,60,73,76,87,102,118,122,123,125,129,134,150,156,164,188,189,208,223,237,246,264,266,265,220,217,215,213,208,202,190,176,170,165,162,146,141,116,112,105,90,80,74,74,57,48,40,36,38,37,42,44,46],"all_points_y":[62,55,52,40,27,24,20,18,18,15,15,17,15,18,12,10,15,17,22,26,38,44,60,64,124,129,125,126,140,153,161,164,163,161,151,137,136,134,133,138,142,144,140,131,126,123,124,121,110,109,90,84,78,62]},"region_attributes":{"Image":"1"}}}},"20.jpeg6341":{"fileref":"","size":6341,"filename":"20.jpeg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[54,69,81,92,116,146,176,187,195,217,225,230,235,239,239,241,240,238,233,231,230,227,220,211,203,197,195,190,186,171,160,159,144,136,124,126,125,123,114,101,98,95,94,94,73,51,48,45,34,27,22,20,21,23,25,41,45,41,41,43,47,52,54],"all_points_y":[78,63,57,54,52,51,55,59,63,79,82,87,89,97,104,114,132,140,143,142,149,155,158,158,157,152,144,144,146,147,147,145,145,147,146,146,152,159,163,161,156,145,144,143,139,139,143,145,145,144,136,129,117,109,95,85,84,83,81,77,74,74,78]},"region_attributes":{"Image":"1"}}}},"21.jpeg4667":{"fileref":"","size":4667,"filename":"21.jpeg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[50,50,185,206,213,235,256,273,282,284,295,308,318,323,320,308,292,282,281,263,231,216,202,187,179,178,172,166,163,164,142,133,126,118,112,102,92,84,77,70,63,52,38,29,20,3,18,50],"all_points_y":[52,52,51,36,16,3,2,11,28,41,42,46,55,68,91,105,112,111,129,148,149,137,116,104,106,111,114,115,106,102,100,95,95,96,90,90,97,93,93,99,95,93,84,84,90,67,53,52]},"region_attributes":{"Key":"1"}}}},"22.jpeg4603":{"fileref":"","size":4603,"filename":"22.jpeg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[25,192,205,232,292,301,300,294,281,238,209,191,163,158,16,14,0,14,25],"all_points_y":[77,77,46,39,41,59,122,144,149,153,152,120,121,118,120,107,86,89,77]},"region_attributes":{"Key":"1"}}}},"26.jpeg3282":{"fileref":"","size":3282,"filename":"26.jpeg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[58,45,42,40,42,49,58,88,98,100,100,98,90,81,81,83,81,79,71,63,61,58],"all_points_y":[81,73,62,48,35,25,24,24,32,43,54,66,78,82,87,90,95,154,168,156,83,81]},"region_attributes":{"Key":"1"}},"1":{"shape_attributes":{"name":"polygon","all_points_x":[171,158,150,149,152,156,165,181,194,202,208,208,203,199,192,187,188,183,179,172,171],"all_points_y":[83,78,64,46,36,28,24,25,25,28,37,54,70,75,80,82,147,157,157,147,83]},"region_attributes":{"Key":"2"}}}},"29.jpeg5023":{"fileref":"","size":5023,"filename":"29.jpeg","base64_img_data":"","file_attributes":{},"regions":{"0":{"shape_attributes":{"name":"polygon","all_points_x":[22,36,46,76,94,99,99,87,73,73,76,76,70,71,68,67,65,65,68,68,70,70,60,54,47,48,44,46,32,22,20,22],"all_points_y":[51,33,28,29,44,54,84,100,107,114,116,125,126,141,143,149,152,155,157,162,167,183,197,196,189,125,123,109,101,89,77,51]},"region_attributes":{"Key":"1"}},"1":{"shape_attributes":{"name":"polygon","all_points_x":[138,151,179,186,195,203,207,206,200,186,181,181,184,181,178,178,176,171,166,156,156,158,159,160,161,158,157,155,156,148,150,143,133,127,128,138],"all_points_y":[41,29,29,31,38,48,56,84,96,107,109,118,123,127,127,189,195,197,197,186,164,162,156,155,151,149,143,140,128,122,109,102,93,82,54,41]},"region_attributes":{"Key":"2"}}}}

[josiahls]

@waleedka Thank you, this took longer to solve than intended (looked at a dozen other issues / stackoverfow Q/A). I wish there was a single simple multiclass example of a Dataset class that explained this. The balloon example seemed the most straightforward image load -> bounding box extraction -> save, however since it is (pointlessly) hardcoded to handle single classes this can make things unnecessarily confusing.

[bat3a]

should all classes be sequential? i mean there ids should be like 1,2,3,4,5..., cause im having errors with 1,5,6,...! any ideas?

[darvida]

@amullapu did you solve the problem ? i get the same kind of error: IndexError: boolean index did not match indexed array along dimension 0; dimension is 56 but corresponding boolean dimension is 57

[Nandhiny-Duraikannu]

@darvida im facing the same issue. Did you find a solution for this error?

[darvida]

Yes it was a problem with the # Number of classes (including background) NUM_CLASSES = 1 + 2 # Background + balloon try to adjust these values in order to get it right :)

[Nandhiny-Duraikannu]

@darvida thank you

[NorthLatitudeOne]

Hello, everyone, I have one problem about my annotations mixed of polygon and rect, the sample annotations as below: "NG0069.jpg195141": { "filename": "NG0069.jpg", "size": 195141, "regions": [{ "shape_attributes": { "name": "polygon", "all_points_x": [365, 366, 456], "all_points_y": [258, 268, 262] }, "region_attributes": { "object": "3" } }, { "shape_attributes": { "name": "rect", "all_points_x": 696, "all_points_y": 431, "width": 8, "height": 100 }, "region_attributes": { "object": "8" } }] } It seems the Balloon sample code could not handle the case of rect ? It's running with error, can someone give me the tips how to modify the sample code to handle the case of rect ?

[josiahls]

Why do you need it as a rectangle object? Just convert it to a polygon. Also you mentioned an error, but no one can help you without showing what the error is.

[NorthLatitudeOne]

@josiahls thanks for your answer, the error message as below, I can convert rectangle to a polygon as temporary solutions, but I really want to know how to modify the code to identify the rectangle object? 226 356 ERROR:root:Error processing image {'id': 'NG0071.jpg', 'source': 'object', 'path': 'dataset\train\NG0071.jpg', 'width': 800, 'height': 600, 'polygons': [{'name': 'rect', 'all_points_x': 356, 'all_points_y': 226, 'width': 94, 'height': 9}, {'name': 'rect', 'all_points_x': 531, 'all_points_y': 227, 'width': 105, 'height': 10}], 'num_ids': [1, 2]} Traceback (most recent call last): File "E:\Anaconda3\lib\site-packages\mask_rcnn-2.1-py3.6.egg\mrcnn\model.py", line 1710, in data_generator use_mini_mask=config.USE_MINI_MASK) File "E:\Anaconda3\lib\site-packages\mask_rcnn-2.1-py3.6.egg\mrcnn\model.py", line 1213, in load_image_gt mask, class_ids = dataset.load_mask(image_id) File "wzballoon.py", line 190, in load_mask rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x']) File "E:\Anaconda3\lib\site-packages\skimage\draw\draw.py", line 457, in polygon return _polygon(r, c, shape) File "skimage/draw/_draw.pyx", line 217, in skimage.draw._draw._polygon (skimage\draw_draw.c:4411) OverflowError: Python int too large to convert to C ssize_t

[abdelDebug]

@waynezhang72 did you find a solution for your problem ? I want to identify the rectangle object

[qq4060]

@darvida thank you

do you solve this ?change it in the Config? NUM_CLASSES = 1 + 2 # Background + balloon

Thanks

[HAMZARaouia]

Since you're using VIA, then BalloonDataset is the closest to your case, so I'll use it as a reference.

The key change is to modify load_mask() to return masks that belong to different classes. Look at the last line of this function:

        # Return mask, and array of class IDs of each instance. Since we have
        # one class ID only, we return an array of 1s
        return mask, np.ones([mask.shape[-1]], dtype=np.int32)

The function returns an array of masks and an array of class IDs. The class IDs are all ones because we have one class. You need to modify it to return an array of class IDs where each class_id is either 1 or 2 (since you want to support two classes).

Now, where does load_mask get the mask details from in the first place? It depends on your dataset. In some cases, like in coco.py, it loads the masks from a file. But in balloon.py, it reads the polygon points from self.image_info, which was populated in the load_balloon(). So let's look at that function.

In load_balloon() go to this line:

        self.add_image(
            "balloon",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons)

This is where it adds every image to the dataset. The first three parameters are standard (dataset name, image_id, and image path). The next three are added so they can be used by load_mask() (width, height, and polygons). You just need to also add class_ids and pass a list of IDs that correspond to the list of polygons you're passing. You can get the class IDs from the annotations JSON. I think it's in the category of each region.

Once you get the list of class IDs (and verify it's sorted in the same order as the list of polygons so they match) then you simply pass it in add_image:

        self.add_image(
            "balloon",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons,
            class_ids=class_ids)

And then update the last line of load_mask to use the newly added info:

        # Return mask, and array of class IDs of each instance.
        return mask, info['class_ids']

this does not change ?

[nyck33]

Can someone tell me where to get ldthan's code for annotator version 2?

[hugoferrero]

Hello, i´m trying to train the model for multiclass segmentation. What is the structure of the folders in order to train the model?..how are the class images and the .json files ordered?. Thank you in advance.

[glala981]

What should I do if I want to cover only human objects in Mask R-CNN? I am currently running Mask R-CNN. I want to recognize only human objects and then cover only human objects.

So I tried to erase 80 objects that were learned in the coco file, but they didn't work correctly. I tried using the index number of a person object, but it didn't work.

class_names= [
    'BG', 'person',
    'bicycle', 'car', 'motorcycle', 'airplane',
    'bus', 'train', 'truck', 'boat', 'traffic light',
    'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird',
    'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear',
    'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie',
    'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball',
    'kite', 'baseball bat', 'baseball glove', 'skateboard',
    'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup',
    'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple',
    'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza',
    'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed',
    'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote',
    'keyboard', 'cell phone', 'microwave', 'oven', 'toaster',
    'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors',
    'teddy bear', 'hair drier', 'toothbrush'
]

 def apply_mask(image, mask, color, alpha=1):    #alpha값 0.5->1
    """apply mask to image"""
    for n, c in enumerate(color):
          image[:, :, n] = np.where(
                mask == 1,
               image[:, :, n] *(1 - alpha) + alpha * c,
               image[:, :, n]
            )

    return image

Even if I modify the code, the program continues to recognize objects that are not human objects.

[NarendraBabuVL]

@mia2mia

Here is the code which worked for me -

def load_student(self, dataset_dir, subset):
    self.add_class("student", 1, "student")
    self.add_class("student", 2, "bag")

    # Train or validation dataset?
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    annotations = [a for a in annotations if a['regions']]

        # Add images
    for a in annotations:
        polygons = [r['shape_attributes'] for r in a['regions'].values()]
        objects = [s['region_attributes'] for s in a['regions'].values()]

        print(objects)
        num_ids=[]
        for n in objects:
          #print(n)
          #print(type(n))
            try:
              if n['object_name']=='student':
                  num_ids.append(1)
              elif n['object_name']=='bag':
                  num_ids.append(2)
            except:
                pass

        #num_ids = [int(n['object_name']) for n in objects]
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "student",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            width=width, height=height,
            polygons=polygons,
            num_ids=num_ids)

I have 2 classes student and bag. Each class has only one region attribute which is 'object_name'

Now consider this line in the code -

objects = [s['region_attributes'] for s in a['regions'].values()]

When I run my code, the object is of the type dict and the key is the region attribute i.e object_name and the value is the object_name's values i.e either bag or student.

Apparently, in the code referred by @ldthan objects is not a dictionary. This I guess happens because of the type of annotator one is using, I am using Vgg annotator version 1.0.4. So the code given by me will work. For version Vgg version 2, this might not work and instead, the code given by @ldthan may work.

hey, I got this error. Please help me Epoch 1/30 ERROR:root:Error processing image {'id': 'IMG_20191112_104159.jpg', 'source': 'traffic', 'path': '../../datasets/traffic/train/IMG_20191112_104159.jpg', 'width': 4000, 'height': 3000, 'polygons': [{'name': 'polygon', 'all_points_x': [1668, 1668, 1673, 1982, 2559, 2553, 2517, 2234, 2224, 1668], 'all_points_y': [1735, 1735, 1941, 2018, 2013, 1920, 1699, 1719, 1761, 1735]}, {'name': 'polygon', 'all_points_x': [3012, 3346, 3665, 3604, 3356, 2996, 3012], 'all_points_y': [2023, 2069, 2023, 1740, 1719, 1704, 2023]}], 'class_ids': [1, 1]} Traceback (most recent call last): File "/content/drive/My Drive/Colab Notebooks/mj_pro_mid/mrcnn/model.py", line 1705, in data_generator use_mini_mask=config.USE_MINI_MASK) File "/content/drive/My Drive/Colab Notebooks/mj_pro_mid/mrcnn/model.py", line 1261, in load_image_gt class_ids = class_ids[_idx] TypeError: only integer scalar arrays can be converted to a scalar index ERROR:root:Error processing image {'id': 'IMG_20191112_104147.jpg', 'source': 'traffic', 'path': '../../datasets/traffic/val/IMG_20191112_104147.jpg', 'width': 4000, 'height': 3000, 'polygons': [{'name': 'polygon', 'all_points_x': [1024, 1019, 1714, 1689, 1575, 1472, 1189, 1081, 911, 1024], 'all_points_y': [2636, 2646, 2456, 2090, 2008, 1822, 1956, 2106, 2564, 2636]}, {'name': 'polygon', 'all_points_x': [2734, 3017, 3161, 3135, 3027, 2785, 2692, 2764, 2734], 'all_points_y': [2095, 2121, 2075, 1915, 1833, 1838, 1987, 2100, 2095]}, {'name': 'polygon', 'all_points_x': [2270, 2394, 2394, 2270, 2250, 2270], 'all_points_y': [1997, 1997, 1874, 1869, 1956, 1997]}], 'class_ids': [4, 3, 4]} Traceback (most recent call last): File "/content/drive/My Drive/Colab Notebooks/mj_pro_mid/mrcnn/model.py", line 1705, in data_generator use_mini_mask=config.USE_MINI_MASK) File "/content/drive/My Drive/Colab Notebooks/mj_pro_mid/mrcnn/model.py", line 1261, in load_image_gt class_ids = class_ids[_idx] TypeError: only integer scalar arrays can be converted to a scalar index

[SriRamGovardhanam]

@AliceDinh morning alice! Since I use the VIA2.0.2,the json file have some different format as those use VIA1.0.x,such as the polygons = [r['shape_attributes'] for r in a['regions'].values()] objects = [s['region_attributes'] for s in a['regions'].values()]I should use polygons = [r['shape_attributes'] for r in a['regions']] objects = [s['region_attributes'] for s in a['regions']]to get the polygons and objects,then for multi number class,in the load_objects function,I don't know exactly how to get the class_id deliver to add_image() and I try several methods just can not make it,could you give some advice on how to get the class_idin the load_object function with the VIA2.0 json file format.

here is one attempt I made just now: ` def load_object(self, dataset_dir, subset):

    # Add classes. We have 6 classes to add.
    self.add_class("object", 1, "porosity")
    self.add_class("object", 2, "crack")
    self.add_class("object", 3, "porosity array")
    self.add_class("object", 4, "lacking of sintering")
    self.add_class("object", 5, "surface hollow")
    self.add_class("object", 6, "surface scratch")

    # Train or validation dataset
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    # Load annotations
    # We mostly care about the x and y coordinates of each region
    # Note: In VIA 2.0, regions was changed from a dict to a list.
    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    # The VIA tool saves images in the JSON even if they don't have any
    # annotations. Skip unannotated images.
    annotations = [a for a in annotations if a['regions']]
    # print("ANNOTATIONS INFO:", annotations)
    idlist = []
    # Add images
    for a in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. These are stores in the
        # shape_attributes (see json format above)
        # The if condition is needed to support VIA versions 1.x and 2.x.
        # print(type(a['regions'])) #list
        '''if type(a['regions']) is dict:
            polygons = [r['shape_attributes'] for r in a['regions'].values()]
        else:
            polygons = [r['shape_attributes'] for r in a['regions']]'''

        polygons = [r['shape_attributes'] for r in a['regions']]
        name = [r['region_attributes']['type'] for r in a['regions']]
        #print("[NAME INFO:]", name)
        #print(type(name)) #list
        '''
        [NAME INFO:] [{'porosity': True}, {'porosity': True}, {'surface hollow': True}, 
        {'porosity': True}, {'lacking of sintering': True}, {'porosity': True}, 
        {'porosity': True}, {'porosity': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}]
        <class 'list'>
        '''
        name_dict = {"porosity": 1,
                     "crack": 2,
                     "porosity array": 3,
                     "lacking of sintering": 4,
                     "surface hollow": 5,
                     "surface scratch": 6}
        name_id = [name_dict[a] for a in name]

        '''class_names = [r['region_attributes'] for r in a['regions']]
        for i in range(len(class_names)):
            if class_names[i]["type"] == "porosity":
                idlist.append(1)
            elif class_names[i]["type"] == "crack":
                idlist.append(2)
            elif class_names[i]["type"] == "porosity":
                idlist.append(3)
            elif class_names[i]["type"] == "lacking of sintering":
                idlist.append(4)
            elif class_names[i]["type"] == "surface hollow":
                idlist.append(5)
            elif class_names[i]["type"] == "surface scratch":
                idlist.append(6)'''

        # load_mask() needs the image size to convert polygons to masks.
        # Unfortunately, VIA doesn't include it in JSON, so we must read
        # the image. This is only managable since the dataset is tiny.
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "object",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            class_id=name_id,
            width=width, height=height,
            polygons=polygons)`

then I get the error Traceback (most recent call last): File "object.py", line 407, in <module> train(model) File "object.py", line 226, in train dataset_train.load_defect(args.dataset, "train") File "object.py", line 143, in load_defect name_id = [name_dict[a] for a in name] File "object.py", line 143, in <listcomp> name_id = [name_dict[a] for a in name] TypeError: unhashable type: 'dict' So how can I solve it? hey , did you executed multiple classes with new version of VGG successfully? if yes , can you please tell me the changes!

[synioe]

@AliceDinh morning alice! Since I use the VIA2.0.2,the json file have some different format as those use VIA1.0.x,such as the polygons = [r['shape_attributes'] for r in a['regions'].values()] objects = [s['region_attributes'] for s in a['regions'].values()]I should use polygons = [r['shape_attributes'] for r in a['regions']] objects = [s['region_attributes'] for s in a['regions']]to get the polygons and objects,then for multi number class,in the load_objects function,I don't know exactly how to get the class_id deliver to add_image() and I try several methods just can not make it,could you give some advice on how to get the class_idin the load_object function with the VIA2.0 json file format. here is one attempt I made just now: ` def load_object(self, dataset_dir, subset):

    # Add classes. We have 6 classes to add.
    self.add_class("object", 1, "porosity")
    self.add_class("object", 2, "crack")
    self.add_class("object", 3, "porosity array")
    self.add_class("object", 4, "lacking of sintering")
    self.add_class("object", 5, "surface hollow")
    self.add_class("object", 6, "surface scratch")

    # Train or validation dataset
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    # Load annotations
    # We mostly care about the x and y coordinates of each region
    # Note: In VIA 2.0, regions was changed from a dict to a list.
    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    # The VIA tool saves images in the JSON even if they don't have any
    # annotations. Skip unannotated images.
    annotations = [a for a in annotations if a['regions']]
    # print("ANNOTATIONS INFO:", annotations)
    idlist = []
    # Add images
    for a in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. These are stores in the
        # shape_attributes (see json format above)
        # The if condition is needed to support VIA versions 1.x and 2.x.
        # print(type(a['regions'])) #list
        '''if type(a['regions']) is dict:
            polygons = [r['shape_attributes'] for r in a['regions'].values()]
        else:
            polygons = [r['shape_attributes'] for r in a['regions']]'''

        polygons = [r['shape_attributes'] for r in a['regions']]
        name = [r['region_attributes']['type'] for r in a['regions']]
        #print("[NAME INFO:]", name)
        #print(type(name)) #list
        '''
        [NAME INFO:] [{'porosity': True}, {'porosity': True}, {'surface hollow': True}, 
        {'porosity': True}, {'lacking of sintering': True}, {'porosity': True}, 
        {'porosity': True}, {'porosity': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}]
        <class 'list'>
        '''
        name_dict = {"porosity": 1,
                     "crack": 2,
                     "porosity array": 3,
                     "lacking of sintering": 4,
                     "surface hollow": 5,
                     "surface scratch": 6}
        name_id = [name_dict[a] for a in name]

        '''class_names = [r['region_attributes'] for r in a['regions']]
        for i in range(len(class_names)):
            if class_names[i]["type"] == "porosity":
                idlist.append(1)
            elif class_names[i]["type"] == "crack":
                idlist.append(2)
            elif class_names[i]["type"] == "porosity":
                idlist.append(3)
            elif class_names[i]["type"] == "lacking of sintering":
                idlist.append(4)
            elif class_names[i]["type"] == "surface hollow":
                idlist.append(5)
            elif class_names[i]["type"] == "surface scratch":
                idlist.append(6)'''

        # load_mask() needs the image size to convert polygons to masks.
        # Unfortunately, VIA doesn't include it in JSON, so we must read
        # the image. This is only managable since the dataset is tiny.
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "object",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            class_id=name_id,
            width=width, height=height,
            polygons=polygons)`

then I get the error Traceback (most recent call last): File "object.py", line 407, in <module> train(model) File "object.py", line 226, in train dataset_train.load_defect(args.dataset, "train") File "object.py", line 143, in load_defect name_id = [name_dict[a] for a in name] File "object.py", line 143, in <listcomp> name_id = [name_dict[a] for a in name] TypeError: unhashable type: 'dict' So how can I solve it? hey , did you executed multiple classes with new version of VGG successfully? if yes , can you please tell me the changes!

Hi, SriRam, I did executed multi-classes with the VIA-2.0.2, but I'm sorry to say the project is too old too find the modified code, I just remind that the difference is the data-format of VIA-2.0 and VIA-1.0, you need to change the way of JSON data parse without a lot of work. Best luck!

[SriRamGovardhanam]

@AliceDinh morning alice! Since I use the VIA2.0.2,the json file have some different format as those use VIA1.0.x,such as the polygons = [r['shape_attributes'] for r in a['regions'].values()] objects = [s['region_attributes'] for s in a['regions'].values()]I should use polygons = [r['shape_attributes'] for r in a['regions']] objects = [s['region_attributes'] for s in a['regions']]to get the polygons and objects,then for multi number class,in the load_objects function,I don't know exactly how to get the class_id deliver to add_image() and I try several methods just can not make it,could you give some advice on how to get the class_idin the load_object function with the VIA2.0 json file format. here is one attempt I made just now: ` def load_object(self, dataset_dir, subset):

    # Add classes. We have 6 classes to add.
    self.add_class("object", 1, "porosity")
    self.add_class("object", 2, "crack")
    self.add_class("object", 3, "porosity array")
    self.add_class("object", 4, "lacking of sintering")
    self.add_class("object", 5, "surface hollow")
    self.add_class("object", 6, "surface scratch")

    # Train or validation dataset
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)

    # Load annotations
    # We mostly care about the x and y coordinates of each region
    # Note: In VIA 2.0, regions was changed from a dict to a list.
    annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
    annotations = list(annotations.values())  # don't need the dict keys

    # The VIA tool saves images in the JSON even if they don't have any
    # annotations. Skip unannotated images.
    annotations = [a for a in annotations if a['regions']]
    # print("ANNOTATIONS INFO:", annotations)
    idlist = []
    # Add images
    for a in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. These are stores in the
        # shape_attributes (see json format above)
        # The if condition is needed to support VIA versions 1.x and 2.x.
        # print(type(a['regions'])) #list
        '''if type(a['regions']) is dict:
            polygons = [r['shape_attributes'] for r in a['regions'].values()]
        else:
            polygons = [r['shape_attributes'] for r in a['regions']]'''

        polygons = [r['shape_attributes'] for r in a['regions']]
        name = [r['region_attributes']['type'] for r in a['regions']]
        #print("[NAME INFO:]", name)
        #print(type(name)) #list
        '''
        [NAME INFO:] [{'porosity': True}, {'porosity': True}, {'surface hollow': True}, 
        {'porosity': True}, {'lacking of sintering': True}, {'porosity': True}, 
        {'porosity': True}, {'porosity': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}, {'surface hollow': True}, 
        {'surface hollow': True}, {'surface hollow': True}]
        <class 'list'>
        '''
        name_dict = {"porosity": 1,
                     "crack": 2,
                     "porosity array": 3,
                     "lacking of sintering": 4,
                     "surface hollow": 5,
                     "surface scratch": 6}
        name_id = [name_dict[a] for a in name]

        '''class_names = [r['region_attributes'] for r in a['regions']]
        for i in range(len(class_names)):
            if class_names[i]["type"] == "porosity":
                idlist.append(1)
            elif class_names[i]["type"] == "crack":
                idlist.append(2)
            elif class_names[i]["type"] == "porosity":
                idlist.append(3)
            elif class_names[i]["type"] == "lacking of sintering":
                idlist.append(4)
            elif class_names[i]["type"] == "surface hollow":
                idlist.append(5)
            elif class_names[i]["type"] == "surface scratch":
                idlist.append(6)'''

        # load_mask() needs the image size to convert polygons to masks.
        # Unfortunately, VIA doesn't include it in JSON, so we must read
        # the image. This is only managable since the dataset is tiny.
        image_path = os.path.join(dataset_dir, a['filename'])
        image = skimage.io.imread(image_path)
        height, width = image.shape[:2]

        self.add_image(
            "object",
            image_id=a['filename'],  # use file name as a unique image id
            path=image_path,
            class_id=name_id,
            width=width, height=height,
            polygons=polygons)`

then I get the error Traceback (most recent call last): File "object.py", line 407, in <module> train(model) File "object.py", line 226, in train dataset_train.load_defect(args.dataset, "train") File "object.py", line 143, in load_defect name_id = [name_dict[a] for a in name] File "object.py", line 143, in <listcomp> name_id = [name_dict[a] for a in name] TypeError: unhashable type: 'dict' So how can I solve it? hey , did you executed multiple classes with new version of VGG successfully? if yes , can you please tell me the changes!

Hi, SriRam, I did executed multi-classes with the VIA-2.0.2, but I'm sorry to say the project is too old too find the modified code, I just remind that the difference is the data-format of VIA-2.0 and VIA-1.0, you need to change the way of JSON data parse without a lot of work. Best luck!

hey thanks that you replied, i sorted things out .. now its working fine

[samrawitergrtr]

I was using the balloon sample and i couldn't understand what are the full blue background did in the Display Samples section to view masked balloon which is found in the link below https://github.com/matterport/Mask_RCNN/blob/master/samples/balloon/inspect_balloon_data.ipynb and if the python code is found in this link https://github.com/matterport/Mask_RCNN/blob/master/samples/balloon/balloon.py
please i rely need your help Thank you!

[nagoyan-am]

Sorry, I might get confused about these discussing points. So if my question is the one repeated, please let me apology first.

I would like to detect what “class” detected in splash results.

I have run the balloon sample code successfully, and I also could add multiple class something like:

self.add_class("object", 1, "porosity")
self.add_class("object", 2, "crack")
self.add_class("object", 3, "porosity array")
self.add_class("object", 4, "lacking of sintering")
self.add_class("object", 5, "surface hollow")

In splash mode, my tested images successfully detect the target object. However, I cannot extract what class are detected in the images.

For example, if image 1 contains the “porosity”, image 2 contains “surface hollow”, and image 3 contains “porosity array” and “crack.”

So far, the trained model could detect all four objects from three images in splash mode. Now, I would like to know further information such as image 1 contains “porosity” and so on. But, what I have got is always “1” in classID although the model detected “surface hollow” or “crack” That is wired to me...

I have investigated it might be related to “detection” and “unmold_detection” functions which are in original balloon code. But unsuccessfully modify the code.

I would appreciate it if someone helps me to find out a way to solve my issues and show proper classID of the detected object.

Thank you in advance

[Adblu]

An interesting example when we have an ellipse in annotations, this code wont work.

        if p['all_points_x'] not in p:
            rr, cc = skimage.draw.ellipse(p['cy'], p['cx'], p['ry'], p['rx'])
            mask[rr, cc, i] = 1
        else:
            rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
            mask[rr, cc, i] = 1

Maybe it will help to somebody ;)

[soumyaiitkgp]

I have created a custom dataset module for training on any desired dataset. Here is the link. I hope it helps.

[SriRamGovardhanam]

Hey, really sorry that I replied you lately

Here are some code changes to be done for multiple classes

https://medium.com/analytics-vidhya/training-your-own-data-set-using-mask-r-cnn-for-detecting-multiple-classes-3960ada85079

https://github.com/SriRamGovardhanam/wastedata-Mask_RCNN-multiple-classes

Hope this helps

On Mon, Jun 29, 2020, 3:14 PM rupa1118 notifications@github.com wrote:

Hi @SriRamGovardhanam https://github.com/SriRamGovardhanam I am trying to add 2 classes but the problem is while loading the mask it's not displaying the class name?

please check the code I have used below:

def load_dataset(self, dataset_dir, subset):

Add classes. We have only one class to add.

self.add_class("leaf blast", 1, "leafblast")
self.add_class("brown spot", 2, "brownspot")

"""if hc is True:
    for i in range(0,4):
        self.add_class("Paddy leaf", 2, "brownspot")
    self.add_class("Paddy leaf", 3, "leafblast")"""

assert subset in ["train", "val", "test"]
dataset_dir = os.path.join(dataset_dir, subset)

annotations = json.load(open(os.path.join(dataset_dir, "via_region_data.json")))
annotations = list(annotations.values())  # don't need the dict keys

annotations = [a for a in annotations if a['regions']]

# Add images
for a in annotations:
    polygons = [r['shape_attributes'] for r in a['regions']]
    names = [s['region_attributes'] for s in a['regions']]
    print("names", names)
    # print("multi_numbers=", multi_numbers)
    # num_ids = [n for n in multi_numbers['number'].values()]
    # for n in multi_numbers:
    #num_ids = [int(n['Paddy leaf']) for n in objects]
    #print("num_ids=", num_ids)
    # print("num_ids_new=", num_ids_new)
    # categories = [s['region_attributes'] for s in a['regions'].values()]
    image_path = os.path.join(dataset_dir, a['filename'])
    image = skimage.io.imread(image_path)
    height, width = image.shape[:2]

    self.add_image("paddyleaf",
                   image_id=a['filename'],
                   path=image_path,
                   width=width, height=height,
                   polygons=polygons,
                   names=names)

def load_mask(self, image_id): image_info = self.image_info[image_id] if image_info["source"] != "paddyleaf": return super(self.class, self).load_mask(image_id)

# Convert polygons to a bitmap mask of shape
# [height, width, instance_count]
info = self.image_info[image_id]
print("info", info)
class_names = info["names"]
print("Class_ names", class_names)
mask = np.zeros([info["height"], info["width"], len(info["polygons"])],
                dtype=np.uint8)
for i, p in enumerate(info["polygons"]):
    # Get indexes of pixels inside the polygon and set them to 1
    rr, cc = skimage.draw.polygon(p['all_points_y'], p['all_points_x'])
    mask[rr, cc, i] = 1
# Assign class_ids by reading class_names
class_ids = np.zeros([len(info["polygons"])])
# In the surgery dataset, pictures are labeled with name 'a' and 'r' representing arm and ring.
for i, p in enumerate(class_names):
    if p['Paddy leaf'] == 'leafblast':
        class_ids[i] = 1
    elif p['Paddy leaf'] == 'brownspot':
        class_ids[i] = 2
class_ids = class_ids.astype(int)
print("class_ids", class_ids)
# Return mask, and array of class IDs of each instance. Since we have
# one class ID only, we return an array of 1s
return mask.astype(np.bool), class_ids

Any suggestions please help me.......thanks in advance

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matterport/Mask_RCNN/issues/372#issuecomment-651034078, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIQKXPIQHMVRMSDW6XMQW5TRZBO75ANCNFSM4EXZ3AZA .

[gethubwy]

In coco.py, annotation format is coco annotation format. In balloon.py, the annotation is achieved by VIA which is different to coco format. In shapes.py, the annotation is not needed coz there is no dataset I successfully trained, evaluated and tested all samples and notebooks (coco, balloon, shapes) and I also created my own dataset, my annotation by VIA. I could train the model using my own dataset successfully, but only for one class (one category). I need to train for more class and I did try to modify funtions: load_objects() and load_mask() without success. So please help.

How to train on only one class? How many pics to labled?

[soumyaiitkgp]

In coco.py, annotation format is coco annotation format. In balloon.py, the annotation is achieved by VIA which is different to coco format. In shapes.py, the annotation is not needed coz there is no dataset I successfully trained, evaluated and tested all samples and notebooks (coco, balloon, shapes) and I also created my own dataset, my annotation by VIA. I could train the model using my own dataset successfully, but only for one class (one category). I need to train for more class and I did try to modify funtions: load_objects() and load_mask() without success. So please help.

How to train on only one class? How many pics to labled?

Hi @gethubwy,

To train in one class, first, you need to annotate the images using the image annotation tool. After the annotation is completed follow step by step procedure as described in this article to train your model. You can train single as well as multi-class object detection as well. This guide covers all the part from image annotation to model training. Hope it helps. Cheers!