Creating Precision Recall Curve from Predictions

[patrickcgray]

I'm really struggling with how to generate a precision recall curve from the outputs of the utils.compute_ap function. Does anyone have an example of outputting this plot for the precision/recall of multiple images?

[shriadke]

Hi, if you have the output of utils.compute_ap(), I suggest using plot_precision_recall function provided in visualize.py . And then modify it (if required) as per your needs. Cheers!!

[kesaroid]

Hey @shriadke, How do you use the plot_precision_recall for the nucleus dataset. I'm using a similar method from here but image_id is giving me an error.

[shriadke]

The PR curve is for single detected image in inspect_model.ipynb .

# Draw precision-recall curve
AP, precisions, recalls, overlaps = utils.compute_ap(gt_bbox, gt_class_id, gt_mask,
                                          r['rois'], r['class_ids'], r['scores'], r['masks'])
visualize.plot_precision_recall(AP, precisions, recalls)

image_id is nothing but the id you have given to the image while preparing the dataset using dataset.load_nucleus().

@kesaroid Why do you want to do exactly? Do you want to plot a Single PR curve for entire dataset? Or different PR cuves for different images from test set?

[kesaroid]

I am trying to get the use the compute_batch_ap for the validation dataset. But even though the predictions are close to the gt, the AP returned is almost zero, with precision being close to zero. Even when I try for a single image using plot_precision_recall Also, Is it possible to get the PR curve for test set? when we don't have their groundtruth?

[shriadke]

Basically, we need GT to calculate Precision and recall values. So I don't think it is possible to plot curve without that.

Also, It is easy to use compute_batch_ap for validation set as given in inspect_nucleus_model.ipynb . Compare your code with it, if you are doing it properly, and your Predictions are close to GT, it should give you the correct mAP. If possible you can share your code here, but I think if you compare it carefully, you will be able to resolve the issue.

Cheers!!

[kesaroid]

Yeah, Following the same notebook, I got it to work. But the APs are close to zero. I am using AP = 1 - AP to resolve this. The detect method is predicting most of the nuclei, but I still get 0 or 0.12345

path = os.getcwd()
model_tar = "nuclei_datasets.tar.gz"
data_path = os.path.join(path + '/nuclei_datasets')
model_path = os.path.join(path + '/logs/nucleus')
weights_path = os.path.join(model_path + '/mask_rcnn_nucleus.h5') #My weights file

DEVICE = "/gpu:0" 

config = nucleus.NucleusConfig()
class InferenceConfig(config.__class__):
    # Run detection on one image at a time
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1

config = InferenceConfig()
dataset = nucleus.NucleusDataset()

with tf.device(DEVICE):
    model = modellib.MaskRCNN(mode="inference", model_dir=model_path, config=config)

model.load_weights(weights_path, by_name=True)

def compute_batch_ap(image_ids):
    APs = []
    for image_id in image_ids:
        # Load image
        image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(dataset, config, image_id, use_mini_mask=False)
        # Run object detection
        results = model.detect([image], verbose=0)
        # Compute AP
        r = results[0]
        AP, precisions, recalls, overlaps = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r['rois'], r['class_ids'], r['scores'], r['masks'])
        AP = 1 - AP
        APs.append(AP)
    return APs, precisions, recalls

    dataset.load_nucleus(data_path, 'val')
    dataset.prepare()
    print("Images: {}\nClasses: {}".format(len(dataset.image_ids), dataset.class_names))
    print("Loading weights ", weights_path)

    image_ids = np.random.choice(dataset.image_ids, 25)
    APs, precisions, recalls = compute_batch_ap(image_ids)
    print("mAP @ IoU=50: ", APs)

    AP = np.mean(APs)
    visualize.plot_precision_recall(AP, precisions, recalls)
    plt.show()

AP = 1 - AP

[shriadke]

@kesaroid Have you changed anything in utils.compute_ap() or the related methods? Also, check your utils.py is updated. Because I'm doing a similar thing with my dataset (not nucleus) and getting proper values for AP.

[kesaroid]

There are no changes in any other related methods. The detection is always similar to the GT

[amrbenattia]

thanks every one for you great collaboration. I have a problem to calculate the mAP for a batch of validation images since there is an error regarding (results = model.detect([image], verbose=0)) and

could anyone helps please?

[amrbenattia]

@kesaroid for batch AP, Precision and Recall estimation, why AP is append while Precision and Recall not?

[chant0]

是的，按照相同的笔记本，我得到了它的工作。但是AP接近于零。我AP = 1 - AP用来解决这个问题。检测方法是预测大多数原子核，但我仍然得到0或0.12345

path = os.getcwd()
model_tar = "nuclei_datasets.tar.gz"
data_path = os.path.join(path + '/nuclei_datasets')
model_path = os.path.join(path + '/logs/nucleus')
weights_path = os.path.join(model_path + '/mask_rcnn_nucleus.h5') #My weights file

DEVICE = "/gpu:0" 

config = nucleus.NucleusConfig()
class InferenceConfig(config.__class__):
    # Run detection on one image at a time
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1

config = InferenceConfig()
dataset = nucleus.NucleusDataset()

with tf.device(DEVICE):
    model = modellib.MaskRCNN(mode="inference", model_dir=model_path, config=config)

model.load_weights(weights_path, by_name=True)

def compute_batch_ap(image_ids):
    APs = []
    for image_id in image_ids:
        # Load image
        image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(dataset, config, image_id, use_mini_mask=False)
        # Run object detection
        results = model.detect([image], verbose=0)
        # Compute AP
        r = results[0]
        AP, precisions, recalls, overlaps = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r['rois'], r['class_ids'], r['scores'], r['masks'])
        AP = 1 - AP
        APs.append(AP)
    return APs, precisions, recalls

    dataset.load_nucleus(data_path, 'val')
    dataset.prepare()
    print("Images: {}\nClasses: {}".format(len(dataset.image_ids), dataset.class_names))
    print("Loading weights ", weights_path)

    image_ids = np.random.choice(dataset.image_ids, 25)
    APs, precisions, recalls = compute_batch_ap(image_ids)
    print("mAP @ IoU=50: ", APs)

    AP = np.mean(APs)
    visualize.plot_precision_recall(AP, precisions, recalls)
    plt.show()

AP = 1 - AP

I want to know if the "dataset = nucleus.NucleusDataset()" you selected is the validation set? How do I get mAP of a test set?

[PNeigel]

Maybe I have some error in my thinking, but isn't the way compute_batch_ap computes the AP wrong? It computes the AP for every image and averages the results, while usually the AP should be computed over the whole test set, i.e. we collect all true positives, false positives and false negatives for the complete test set, make the prec/recall curve from that and calculate AP

[hungdaica77]

help me

[WillianaLeite]

I customized the "https://github.com/matterport/Mask_RCNN.git" repository to train with my own data set, for object detection, ignoring the mask segmentation part. Now I am evaluating my results, I can calculate the MAP, but I cannot calculate the F1 score. I have this function: compute_ap, from "https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/utils.py" which returns the "mAP, details, memories, overlays" for each image. The point is that I cannot apply the F1 score formula, because the variables "precision" and "recalls" are lists.

def compute_ap(gt_boxes, gt_class_ids, gt_masks,
               pred_boxes, pred_class_ids, pred_scores, pred_masks,
               iou_threshold=0.5):

    # Get matches and overlaps
    gt_match, pred_match, overlaps = compute_matches(
        gt_boxes, gt_class_ids, gt_masks,
        pred_boxes, pred_class_ids, pred_scores, pred_masks,
        iou_threshold)

    # Compute precision and recall at each prediction box step
    precisions = np.cumsum(pred_match > -1) / (np.arange(len(pred_match)) + 1)
    recalls = np.cumsum(pred_match > -1).astype(np.float32) / len(gt_match)

    # Pad with start and end values to simplify the math
    precisions = np.concatenate([[0], precisions, [0]])
    recalls = np.concatenate([[0], recalls, [1]])

    # Ensure precision values decrease but don't increase. This way, the
    # precision value at each recall threshold is the maximum it can be
    # for all following recall thresholds, as specified by the VOC paper.
    for i in range(len(precisions) - 2, -1, -1):
        precisions[i] = np.maximum(precisions[i], precisions[i + 1])

    # Compute mean AP over recall range
    indices = np.where(recalls[:-1] != recalls[1:])[0] + 1
    mAP = np.sum((recalls[indices] - recalls[indices - 1]) *
                 precisions[indices])

    return mAP, precisions, recalls, overlaps

[gizemtanriver]

Maybe I have some error in my thinking, but isn't the way compute_batch_ap computes the AP wrong? It computes the AP for every image and averages the results, while usually the AP should be computed over the whole test set, i.e. we collect all true positives, false positives and false negatives for the complete test set, make the prec/recall curve from that and calculate AP

how do we implement a prec/recall curve from a batch of images here? do you mean we need a new function in utils.py to keep track all TP, FP etc to compute prec/recall curve? or can we just iterate over all images using the compute_ap function, collect precisions and recalls that the function returns and average the values at every threshold point for all images?

[MadioubiWalid]

Hello, I used the code for create a precision recall but i found the value "1" (mAP @ IoU = 50 : 1.0) the thing that imposible do you have any idea about this problem

[Behnam72]

@shriadke, I'm trying to plot the precision-recall for my entire dataset. Would you please tell me how to do it?

[ManooshSamiei]

Hi @shriadke, I have the same question how one can plot the average precision recall curve for the whole set of test images? Why only AP is appended above?

[software93]

@Behnam72 @ManooshSamiei I have same question how can you draw precision and recall curve for whole test set???