no accuracy metrics while training

[cpoptic]

Under model.py we see the keras model compile

        self.keras_model.compile(
            optimizer=optimizer,
            loss=[None] * len(self.keras_model.outputs),
            metrics=['accuracy'])

And then the metrics for the Mask RCNN metrics are added:

        # Add metrics for losses
        for name in loss_names:
            if name in self.keras_model.metrics_names:
                continue
            layer = self.keras_model.get_layer(name)
            self.keras_model.metrics_names.append(name)
            loss = (
                tf.reduce_mean(layer.output, keepdims=True)
                * self.config.LOSS_WEIGHTS.get(name, 1.))
            self.keras_model.metrics_tensors.append(loss)
        print("metrics: ", self.keras_model.metrics_names)

But when training the model, neither train_accuracy nor val_accuracy are reported. How to add these metrics and report them with each epoch, alongside the mrcnn metrics?

[VtlNmnk]

Hello guys, thanks for sharing the code to have accuracy metrics. I try to implement the code but I have errors :( I pasted the class "MeanAveragePrecisionCallbacks" at the end of the model.py code. Then, in train.py I defined the variable like you did @VtlNmnk: mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1) But when I launch my training, I have this error: Traceback (most recent call last): File "Taraudage.py", line 372, in <module> train(model) File "Taraudage.py", line 196, in train mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(train_model=model, inference_model=model_inference, NameError: name 'model_inference' is not defined What variable do you expect for model inference ? Thanks for you time, Regards, Antoine

I have the same issue , how did you solve it please !

model_inference = modellib.MaskRCNN(mode="inference", config=_InfConfig(), model_dir=MODEL_DIR)

[zaiternouara]

Hello guys, thanks for sharing the code to have accuracy metrics. I try to implement the code but I have errors :( I pasted the class "MeanAveragePrecisionCallbacks" at the end of the model.py code. Then, in train.py I defined the variable like you did @VtlNmnk: mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1) But when I launch my training, I have this error: Traceback (most recent call last): File "Taraudage.py", line 372, in <module> train(model) File "Taraudage.py", line 196, in train mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(train_model=model, inference_model=model_inference, NameError: name 'model_inference' is not defined What variable do you expect for model inference ? Thanks for you time, Regards, Antoine

I have the same issue , how did you solve it please !

model_inference = modellib.MaskRCNN(mode="inference", config=_InfConfig(), model_dir=MODEL_DIR)

Thank you ,i tried this :

def train(dataset_train, dataset_val, model):
    """Train the model."""
    # Training dataset.
    dataset_train.prepare()

    # Validation dataset
    dataset_val.prepare()
    model_inference = modellib.MaskRCNN(mode="inference", config=InferenceConfig(), model_dir=DEFAULT_LOGS_DIR)

    mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model,model_inference, dataset_val,calculate_map_at_every_X_epoch=1, verbose=1)

    # *** This training schedule is an example. Update to your needs ***
    print("Training network heads")
    model.train(dataset_train, dataset_val,
                learning_rate=config.LEARNING_RATE,
                epochs=300,
                layers='heads',
                custom_callbacks=[mean_average_precision_callback])

But still the same error

[zaiternouara]

Hello guys, thanks for sharing the code to have accuracy metrics. I try to implement the code but I have errors :( I pasted the class "MeanAveragePrecisionCallbacks" at the end of the model.py code. Then, in train.py I defined the variable like you did @VtlNmnk: mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1) But when I launch my training, I have this error: Traceback (most recent call last): File "Taraudage.py", line 372, in <module> train(model) File "Taraudage.py", line 196, in train mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(train_model=model, inference_model=model_inference, NameError: name 'model_inference' is not defined What variable do you expect for model inference ? Thanks for you time, Regards, Antoine

I have the same issue , how did you solve it please !

model_inference = modellib.MaskRCNN(mode="inference", config=_InfConfig(), model_dir=MODEL_DIR)

This is my costumTrain :

The Script is modified to train on custom Labelme Data.

import os import sys import json import wget import datetime import numpy as np import skimage.draw

Root directory of the project

ROOT_DIR = os.path.dirname(os.path.abspath(os.path.abspath(file))) print(ROOT_DIR) from mrcnn.utils import compute_ap from mrcnn.model import load_image_gt from mrcnn.model import mold_image from mrcnn.config import Config from mrcnn import model as modellib from mrcnn import utils from mrcnn import visualize from numpy import expand_dims from numpy import mean

In case of configuration

git clone the repo and append the path Mask_RCNN repo to use the custom configurations

git clone https://github.com/matterport/Mask_RCNN

sys.path.append("path_to_mask_rcnn")

maskrrr=sys.path.append("/content/drive/MyDrive/business_card_extract_information/buisness_card_extract_information/maskrcnn/Mask-RCNN-Implementation")

Download Link

https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5

if not os.path.isfile("mask_rcnn_coco.h5"): url = "https://github.com/ayoolaolafenwa/PixelLib/releases/download/1.2/mask_rcnn_coco.h5" print("Downloading the mask_rcnn_coco.h5") filename = wget.download(url) print(f"[INFO]: Download complete >> {filename}")

Path to trained weights file

COCO_WEIGHTS_PATH = os.path.join(ROOT_DIR, "mask_rcnn_coco.h5")

Directory to save logs and model checkpoints, if not provided

through the command line argument --logs

DEFAULT_LOGS_DIR = os.path.join(ROOT_DIR, "logs")

Change it for your dataset's name

source="Dataset2021" ############################################################

My Model Configurations (which you should change for your own task)

############################################################

class ModelConfig(Config): """Configuration for training on the toy dataset. Derives from the base Config class and overrides some values. """

Give the configuration a recognizable name

NAME = "ImageSegmentationMaskRCNN"

# We use a GPU with 12GB memory, which can fit two images.
# Adjust down if you use a smaller GPU.
IMAGES_PER_GPU = 2 # 1

# Number of classes (including background)
NUM_CLASSES = 1 + 1 # Background,
# typically after labeled, class can be set from Dataset class
# if you want to test your model, better set it corectly based on your trainning dataset

# Number of training steps per epoch
STEPS_PER_EPOCH = 100

# Skip detections with < 90% confidence
DETECTION_MIN_CONFIDENCE = 0.9

class InferenceConfig(ModelConfig):

Set batch size to 1 since we'll be running inference on

# one image at a time. Batch size = GPU_COUNT * IMAGES_PER_GPU
GPU_COUNT = 1
IMAGES_PER_GPU = 1

############################################################

Dataset (My labelme dataset loader)

############################################################

class LabelmeDataset(utils.Dataset):

Load annotations

# Labelme Image Annotator v = 3.16.7 
# different version may have different structures
# besides, labelme's annotation setup one file for 
# one picture not for all pictures
# and this annotations  are all dicts after Python json load 
# {
#   "version": "3.16.7",
#   "flags": {},
#   "shapes": [
#     {
#       "label": "balloon",
#       "line_color": null,
#       "fill_color": null,
#       "points": [[428.41666666666674,  875.3333333333334 ], ...],
#       "shape_type": "polygon",
#       "flags": {}
#     },
#     {
#       "label": "balloon",
#       "line_color": null,
#       "fill_color": null,
#       "points": [... ],
#       "shape_type": "polygon",
#       "flags": {}
#     },
#   ],
#   "lineColor": [(4 number)],
#   "fillColor": [(4 number)],
#   "imagePath": "10464445726_6f1e3bbe6a_k.jpg",
#   "imageData": null,
#   "imageHeight": 2019,
#   "imageWidth": 2048
# }
# We mostly care about the x and y coordinates of each region
def load_labelme(self, dataset_dir, subset):
    """
    Load a subset of the dataset.
    source: coustomed source id, exp: load data from coco, than set it "coco",
            it is useful when you ues different dataset for one trainning.(TODO)
            see the prepare func in utils model for details
    dataset_dir: Root directory of the dataset.
    subset: Subset to load: train or val
    """
    assert subset in ["train", "val"]
    dataset_dir = os.path.join(dataset_dir, subset)
    # Train or validation dataset
    filenames = os.listdir(dataset_dir)
    jsonfiles,annotations=[],[]
    for filename in filenames:
        if filename.endswith(".json"):
            jsonfiles.append(filename)
            annotation = json.load(open(os.path.join(dataset_dir,filename)))
            # Insure this picture is in this dataset
            imagename = annotation['imagePath']
            if not os.path.isfile(os.path.join(dataset_dir,imagename)):
                continue
            if len(annotation["shapes"]) == 0:
                continue
            # you can filter what you don't want to load
            annotations.append(annotation)

    print("In {source} {subset} dataset we have {number:d} annotation files."
        .format(source=source, subset=subset,number=len(jsonfiles)))
    print("In {source} {subset} dataset we have {number:d} valid annotations."
        .format(source=source, subset=subset,number=len(annotations)))

    # Add images and get all classes in annotation files
    # typically, after labelme's annotation, all same class item have a same name
    # this need us to annotate like all "ball" in picture named "ball"
    # not "ball_1" "ball_2" ...
    # we also can figure out which "ball" it is refer to.
    labelslist = []
    for annotation in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. These are stores in the
        # shape_attributes (see json format above)
        shapes = [] 
        classids = []

        for shape in annotation["shapes"]:
            # first we get the shape classid
            label = shape["label"]
            if labelslist.count(label) == 0:
                labelslist.append(label)
            classids.append(labelslist.index(label)+1)
            shapes.append(shape["points"])

        # load_mask() needs the image size to convert polygons to masks.
        width = annotation["imageWidth"]
        height = annotation["imageHeight"]
        self.add_image(
            source,
            image_id=annotation["imagePath"],  # use file name as a unique image id
            path=os.path.join(dataset_dir,annotation["imagePath"]),
            width=width, height=height,
            shapes=shapes, classids=classids)

    print("In {source} {subset} dataset we have {number:d} class item"
        .format(source=source, subset=subset,number=len(labelslist)))

    for labelid, labelname in enumerate(labelslist):
        self.add_class(source,labelid,labelname)

def load_mask(self,image_id):
    """
    Generate instance masks for an image.
   Returns:
    masks: A bool array of shape [height, width, instance count] with one mask per instance.
    class_ids: a 1D array of class IDs of the instance masks.
    """
    # If not the source dataset you want, delegate to parent class.
    image_info = self.image_info[image_id]
    if image_info["source"] != source:
        return super(self.__class__, self).load_mask(image_id)

    # Convert shapes to a bitmap mask of shape
    # [height, width, instance_count]
    info = self.image_info[image_id]
    mask = np.zeros([info["height"], info["width"], len(info["shapes"])], dtype=np.uint8)
    #printsx,printsy=zip(*points)
    for idx, points in enumerate(info["shapes"]):
        # Get indexes of pixels inside the polygon and set them to 1
        pointsy,pointsx = zip(*points)
        rr, cc = skimage.draw.polygon(pointsx, pointsy)
        mask[rr, cc, idx] = 1
    masks_np = mask.astype(np.bool)
    classids_np = np.array(image_info["classids"]).astype(np.int32)
    # Return mask, and array of class IDs of each instance. Since we have
    # one class ID only, we return an array of 1s
    return masks_np, classids_np

def image_reference(self,image_id):
    """Return the path of the image."""
    info = self.image_info[image_id]
    if info["source"] == source:
        return info["path"]
    else:
        super(self.__class__, self).image_reference(image_id)

def train(dataset_train, dataset_val, model): """Train the model."""

Training dataset.

dataset_train.prepare()

# Validation dataset
dataset_val.prepare()
model_inference = modellib.MaskRCNN(mode="inference", config=InferenceConfig(), model_dir=DEFAULT_LOGS_DIR)

mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model,model_inference, dataset_val,calculate_map_at_every_X_epoch=1, verbose=1)

# *** This training schedule is an example. Update to your needs ***
print("Training network heads")
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=300,
            layers='heads',
            custom_callbacks=[mean_average_precision_callback])

def test(model, image_path = None, video_path=None, savedfile=None): assert image_path or video_path

 # Image or video?
if image_path:
    # Run model detection and generate the color splash effect
    print("Running on {}".format(args.image))
    # Read image
    image = skimage.io.imread(args.image)
    # Detect objects
    r = model.detect([image], verbose=1)[0]
    # Colorful
    import matplotlib.pyplot as plt

    _, ax = plt.subplots()
    visualize.get_display_instances_pic(image, boxes=r['rois'], masks=r['masks'], 
        class_ids = r['class_ids'], class_number=model.config.NUM_CLASSES,ax = ax,
        class_names=None,scores=None, show_mask=True, show_bbox=True)
    # Save output
    if savedfile == None:
        file_name = "test_{:%Y%m%dT%H%M%S}.png".format(datetime.datetime.now())
    else:
        file_name = savedfile
    plt.savefig(file_name)
    #skimage.io.imsave(file_name, testresult)
elif video_path:
    pass
print("Saved to ", file_name)

############################################################

Training and Validating

############################################################

if name == 'main': import argparse

# Parse command line arguments
parser = argparse.ArgumentParser(description='Train Mask R-CNN to detect balloons.')
parser.add_argument("command",
                    metavar="<command>",
                    help="'train' or 'test'")
parser.add_argument('--dataset', required=False,
                    metavar="/path/to/dataset/",
                    help='Directory of your dataset')
parser.add_argument('--weights', required=True,
                    metavar="/path/to/weights.h5",
                    help="Path to weights .h5 file or 'coco', 'last' or 'imagenet'")
parser.add_argument('--logs', required=False,
                    default=DEFAULT_LOGS_DIR,
                    metavar="/path/to/logs/",
                    help='Logs and checkpoints directory (default=./logs/)')
parser.add_argument('--image', required=False,
                    metavar="path or URL to image",
                    help='Image to test and color splash effect on')
parser.add_argument('--video', required=False,
                    metavar="path or URL to video",
                    help='Video to test and color splash effect on')
parser.add_argument('--classnum', required=False,
                    metavar="class number of your detect model",
                    help="Class number of your detector.")
args = parser.parse_args()

# Validate arguments
if args.command == "train":
    assert args.dataset, "Argument --dataset is required for training"
elif args.command == "test":
    assert args.image or args.video or args.classnum, \
        "Provide --image or --video and  --classnum of your model to apply testing"

print("Weights: ", args.weights)
print("Dataset: ", args.dataset)
print("Logs: ", args.logs)

# Configurations
if args.command == "train":
    config = ModelConfig()
    dataset_train, dataset_val = LabelmeDataset(), LabelmeDataset()
    dataset_train.load_labelme(args.dataset,"train")
    dataset_val.load_labelme(args.dataset,"val")
    config.NUM_CLASSES = len(dataset_train.class_info)
elif args.command == "test":
    config = InferenceConfig()
    config.NUM_CLASSES = int(args.classnum)+1 # add backgrouond

config.display()

# Create model
if args.command == "train":
    model = modellib.MaskRCNN(mode="training", config=config,model_dir=args.logs)
else:
    model = modellib.MaskRCNN(mode="inference", config=config, model_dir=args.logs)

# Select weights file to load
if args.weights.lower() == "coco":
    weights_path = COCO_WEIGHTS_PATH
elif args.weights.lower() == "last":
    # Find last trained weights
    weights_path = model.find_last()
else:
    weights_path = args.weights

# Load weights
print("Loading weights ", weights_path)
if args.command == "train":
    if args.weights.lower() == "coco":
        # Exclude the last layers because they require a matching
        # number of classes if we change the backbone?
        model.load_weights(weights_path, by_name=True, exclude=[
            "mrcnn_class_logits", "mrcnn_bbox_fc",
            "mrcnn_bbox", "mrcnn_mask"])
    else:
        model.load_weights(weights_path, by_name=True)
    # Train or evaluate
    train(dataset_train, dataset_val, model)
elif args.command == "test":
    # we test all models trained on the dataset in different stage
    print(os.getcwd())
    filenames = os.listdir(args.weights)
    for filename in filenames:
        if filename.endswith(".h5"):
            print(f"Load weights from {filename}")
            model.load_weights(os.path.join(args.weights,filename),by_name=True)
            savedfile_name = os.path.splitext(filename)[0] + ".jpg"
            test(model, image_path=args.image,video_path=args.video, savedfile=savedfile_name)
else:
    print("'{}' is not recognized.Use 'train' or 'test'".format(args.command))    

[kecyarchana]

I incorporated mean average precision as a callback (as mentioned above) for my validation dataset during training in every 5 epochs but I am getting nan values and a runtime warning as below.

I cannot figure out what might be the cause as my validation dataset has only the images with the objects in it and ground truth files as shapefiles . Any help would be greatly appreciated.

Thanks

[vineet131]

hi @mhtarora39

I am trying to use your code to calculate map, precision, and recall but get the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-23-a2f082d763b6> in <module>
     64 config = InferenceConfig()
     65 evel = EvalImage(dataset_vall,modelinf,config)
---> 66 evel.evaluate_model()
     67 

<ipython-input-23-a2f082d763b6> in evaluate_model(self, len)
     51       r = yhat[0]
     52       # calculate statistics, including AP
---> 53       AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
     54       precisions_dict[image_id] = np.mean(precisions)
     55       recall_dict[image_id] = np.mean(recalls)

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_ap(gt_boxes, gt_class_ids, gt_masks, pred_boxes, pred_class_ids, pred_scores, pred_masks, iou_threshold)
    728         gt_boxes, gt_class_ids, gt_masks,
    729         pred_boxes, pred_class_ids, pred_scores, pred_masks,
--> 730         iou_threshold)
    731 
    732     # Compute precision and recall at each prediction box step

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_matches(gt_boxes, gt_class_ids, gt_masks, pred_boxes, pred_class_ids, pred_scores, pred_masks, iou_threshold, score_threshold)
    680 
    681     # Compute IoU overlaps [pred_masks, gt_masks]
--> 682     overlaps = compute_overlaps_masks(pred_masks, gt_masks)
    683 
    684     # Loop through predictions and find matching ground truth boxes

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_overlaps_masks(masks1, masks2)
    113 
    114     # intersections and union
--> 115     intersections = np.dot(masks1.T, masks2)
    116     union = area1[:, None] + area2[None, :] - intersections
    117     overlaps = intersections / union

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (5,1048576) and (3136,6) not aligned: 1048576 (dim 1) != 3136 (dim 0)

This is my implementation:

from mrcnn.utils import compute_ap

class EvalImage():
  def __init__(self,dataset,model,cfg):
    self.dataset = dataset
    self.model   = model
    self.cfg     = cfg

  def evaluate_model(self , len = 50):
    APs = list()
    precisions_dict = {}
    recall_dict     = {}
    for index,image_id in enumerate(self.dataset.image_ids):
      if(index > len):
         break; 
      # load image, bounding boxes and masks for the image id
      image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(self.dataset, self.cfg,image_id)
      # convert pixel values (e.g. center)
      #scaled_image = modellib.mold_image(image, self.cfg)
      # convert image into one sample
      sample = np.expand_dims(image, 0)

     # print(len(image))
      # make prediction
      yhat = self.model.detect(sample, verbose=1)
      # extract results for first sample
      r = yhat[0]
      # calculate statistics, including AP
      AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
      precisions_dict[image_id] = np.mean(precisions)
      recall_dict[image_id] = np.mean(recalls)
      # store
      APs.append(AP)

    # calculate the mean AP across all images
    mAP = np.mean(APs)
    return mAP,precisions_dict,recall_dict

dataset_vall = CocoLikeDataset()
dataset_vall.load_data('D:/Desktop/aktwelve_mask_rcnn/datasets/Acacia dataset', 'val')
dataset_vall.prepare()

config = InferenceConfig()

modelinf = modellib.MaskRCNN(mode="inference", 
                          config=config,
                          model_dir=MODEL_DIR)

modelinf.load_weights(os.path.join(ROOT_DIR, "mask_rcnn_acacia-stp73-epch150_0075.h5"), by_name=True)

evel = EvalImage(dataset_vall,modelinf,config)
evel.evaluate_model()

Any suggestions would be appreciated

Hey I have the exact same issue. Did you find a solution to this error??

@sohinimallick @tomjordan and anyone else having this issue:

I just had this same issue today, and was stuck on it for a while. Turns out, something was overriding the default value of False passed to use_mini_mask at this line (in my case, the code had been changed to fetch it from elsewhere):

image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(self.dataset, self.inference_model.config, image_id, use_mini_mask=False)

So just pass False and it should be fixed. Although now I'm getting an error that my mAP is NaN, which I need to check.

[filmonmehari]

@VtlNmnk That's awsome idea but can u help me to plot the accuracy and losses seen on the output arguments. I have have a topic to submit for such parameters after 3 weeks.

thanks in advance!

[VtlNmnk]

@VtlNmnk That's awsome idea but can u help me to plot the accuracy and losses seen on the output arguments. I have have a topic to submit for such parameters after 3 weeks.

thanks in advance! Hi, @filmonmehari! Why don't you use the tensorboard? https://github.com/matterport/Mask_RCNN/issues/2714

[WangariKimotho]

Hello @VtlNmnk and all, I tried implementing the code, made the necesary changes for my workflow but I still cannot seem to get the mask and rpn class losses etc

Also I get the feeling that part of the reason lies in that the compile function predefines the loss var as None on lines 2188-2190.... correct me if I'm wrong. I appreciate any insight moving forward