facebookresearch / maskrcnn-benchmark

Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch.
MIT License
9.3k stars 2.49k forks source link

How to draw pr curve and output tested images? #94

Open guanbin1994 opened 6 years ago

guanbin1994 commented 6 years ago

❓ Questions and Help

I can get the AP, but i do not know how to draw pr curve ?

And because the dataset i used to train is medical dataset, so I also want to look the detected test images which can show bbox on images, what should i do to output tested images into a folder.

Thanks!

fmassa commented 6 years ago

I have not yet added functionality to draw PR curves, nor to output the tested images predictions. It is not hard to add, but I didn't have the time yet.

ChenjieXu commented 6 years ago

question Questions and Help

I can get the AP, but i do not know how to draw pr curve ?

And because the dataset i used to train is medical dataset, so I also want to look the detected test images which can show bbox on images, what should i do to output tested images into a folder.

Thanks!

You may refer to maskrcnn_demo.ipynb in demo folder. Here is the code from predictor.py

`def compute_colors_for_labels(labels): """ Simple function that adds fixed colors depending on the class """ colors = labels[:, None] * torch.tensor([2 25 - 1, 2 15 - 1, 2 ** 21 - 1]) colors = (colors % 255).numpy().astype("uint8") return colors

def overlay_boxes(image, predictions): """ Adds the predicted boxes on top of the image Arguments: image (np.ndarray): an image as returned by OpenCV predictions (BoxList): the result of the computation by the model. It should contain the field labels. """ labels = predictions.get_field("labels") boxes = predictions.bbox

    colors = compute_colors_for_labels(labels).tolist()

    for box, color in zip(boxes, colors):
        box = box.to(torch.int64)
        top_left, bottom_right = box[:2].tolist(), box[2:].tolist()
        image = cv2.rectangle(
            image, tuple(top_left), tuple(bottom_right), tuple(color), 3
        )

    return image`
guanbin1994 commented 5 years ago

question Questions and Help

I can get the AP, but i do not know how to draw pr curve ? And because the dataset i used to train is medical dataset, so I also want to look the detected test images which can show bbox on images, what should i do to output tested images into a folder. Thanks!

You may refer to maskrcnn_demo.ipynb in demo folder. Here is the code from predictor.py

`def compute_colors_for_labels(labels): """ Simple function that adds fixed colors depending on the class """ colors = labels[:, None] * torch.tensor([2 25 - 1, 2 15 - 1, 2 ** 21 - 1]) colors = (colors % 255).numpy().astype("uint8") return colors

def overlay_boxes(image, predictions): """ Adds the predicted boxes on top of the image Arguments: image (np.ndarray): an image as returned by OpenCV predictions (BoxList): the result of the computation by the model. It should contain the field labels. """ labels = predictions.get_field("labels") boxes = predictions.bbox

    colors = compute_colors_for_labels(labels).tolist()

    for box, color in zip(boxes, colors):
        box = box.to(torch.int64)
        top_left, bottom_right = box[:2].tolist(), box[2:].tolist()
        image = cv2.rectangle(
            image, tuple(top_left), tuple(bottom_right), tuple(color), 3
        )

    return image`

hello, my problem i s how to draw the precision-recall curve

fmassa commented 5 years ago

@guanbin1994 didn't https://github.com/facebookresearch/maskrcnn-benchmark/issues/353 address your question?

milongo commented 5 years ago

@fmassa issue I have a similar question. From issue #353 I can get the coco_eval object that has the precision and recall values. However, the shape of these two arrays is different, and it is not clear how to use them to plot a PR curve. I can see the arrays as being of shape T, R, K, A, M and T, K, A, M for precision and recall, respectively. So I have 101 precision values at each IoU threshold but only one recall value. How can I plot the PR curve from these?

ayman-saleh commented 5 years ago

pycocotools gives you the eval at IoU thresholds staring at 0.5 up to 0.95 with step sizes of 0.05. The dimensions of the precision array correspond to the following:

So, if you wanted to plot the curves, you could write something like this:

import numpy as np
import matplotlib.pyplot as plt 

all_precision = coco_eval.eval['precision']

pr_5 = all_precision[0, :, 0, 0, 2] # data for IoU@0.5
pr_7 = all_precision[4, :, 0, 0, 2] # data for IoU@0.7
pr_9 = all_precision[8, :, 0, 0, 2] # data for IoU@0.9

x = np.arange(0, 1.01, 0.01)
plt.plot(x, pr_5, label='IoU@0.5')
plt.plot(x, pr_7, label='IoU@0.7')
plt.plot(x, pr_9, label='IoU@0.9')

plt.show()
isalirezag commented 4 years ago

@ayman-saleh Thank you for your answer! Just for my understanding, is M the confidence thresholds that is used? in other words, does it means that when M=100 we just consider the objects with confidence score of 100? Hmmm is it possible to compute the mAP for a certain threshold?

ayman-saleh commented 4 years ago

@isalirezag M is a list, so by default, i believe it is M = [1, 10, 100]. This list corresponds to the thresholds on max detections per image.

https://github.com/cocodataset/cocoapi/blob/636becdc73d54283b3aac6d4ec363cffbb6f9b20/PythonAPI/pycocotools/cocoeval.py#L28

xabirizar9 commented 4 years ago

@ayman-saleh sorry that I got here so late, but how would you go about plotting the precision against the recall? Here you are just plotting the precision against an array from 0 to 1.01, but to my knowledge, recall also has different values, which would make the plot different. Following your code, I plotted this in my program and this came out, which looks a bit odd to me compared to the typical precision vs recall curves.

precisionVSrecall

jpainam commented 3 years ago

Hi @ayman-saleh how about the recall array?, the dimensions of precision array is 5 while the dimensions of the recall array is 4. ie. [TxKxAxM]. How to extract the recall in a shape that can be plot. Thank you.

Eldad27 commented 3 years ago

@ayman-saleh The code you provided computes precision. Please how about the recall points?

  1. how to compute it? I tried to use your suggested code to generate PR curve on custom COCO dataset and had two extremely odd curves for two different classes (Please see below). The dimension of precision returned a shape of [10, 101, 2, 4, 3] against a recall dimension of [10, 2, 4, 3]. how does a 5 dimension shape plot against a 4 dimension shape? Could this be the reason for the Odd curves?
  2. Class 2 PR curve
Class 1 PR curve

?

Btw, The array of precision returns values full of ones. something probably associated with my dataset. This seem a strange to me. Have you by any chance come across such behaviour?

jessicametzger commented 3 years ago

Hi @ayman-saleh how about the recall array?, the dimensions of precision array is 5 while the dimensions of the recall array is 4. ie. [TxKxAxM]. How to extract the recall in a shape that can be plot. Thank you.

Instead of parametrizing the arrays by score threshold, I believe they are parametrized by recall threshold. So you can just do as in here and the result will be a precision vs. recall curve.

manukastratta commented 2 years ago

Hi @ayman-saleh how about the recall array?, the dimensions of precision array is 5 while the dimensions of the recall array is 4. ie. [TxKxAxM]. How to extract the recall in a shape that can be plot. Thank you.

Instead of parametrizing the arrays by score threshold, I believe they are parametrized by recall threshold. So you can just do as in here and the result will be a precision vs. recall curve.

Hi. Do you know how we could parameterize the precision arrays by score threshold instead of by recall threshold? Thanks