How I compute mAP@0.7 using val.py?

[EslamHaourn]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

How I compute mAP@0.7 using val.py?

Additional

No response

[glenn-jocher]

Hello,

To compute mAP @EslamHaourn.7 using val.py, simply add the parameter -iou_t 0.70 to the command. For example: python val.py --data coco.yaml --weights yolov5s.pt --iou-thres 0.70

This will evaluate the validation set using IoU threshold of 0.7 for the mAP calculation.

Let me know if you have any further questions.

Best regards.

[EslamHaourn]

Thanks for reply I would like to compute mAP@.7 beside mAP@.5 and mAP@.5:.95 I mean the iou of mAP not NMS iou threshold. Thanks

[glenn-jocher]

@EslamHaourn hello,

Thank you for your question. It is possible to compute mAP at an IoU threshold other than 0.5 or (0.5-0.95) by specifying the desired IoU threshold using the --iou-thres flag when running the validation script val.py. You can set this flag to the desired value for your computation, for example:

python val.py --data coco.yaml --weights yolov5s.pt --iou-thres 0.7

This will run validation using the weights specified by --weights, the dataset specified by --data, and an IoU threshold of 0.7 for mAP calculation.

I hope this helps. Let me know if you have any further questions.

[EslamHaourn]

@glenn-jocher hello, Is there a difference between iou for mAP and iou for NMS? Thank you in advance.

[glenn-jocher]

@EslamHaourn hello,

Yes, there is a difference between IoU for mAP and IoU for NMS.

IoU (Intersection over Union) is a measure of how much two bounding boxes overlap. In the case of object detection, IoU is used to compare the predicted bounding boxes with the ground-truth bounding boxes to determine if the object has been correctly detected.

When calculating mAP, IoU is used to determine whether an object has been correctly detected in the image. A predicted bounding box is considered a true positive if its IoU with any ground-truth bounding box is greater than a certain threshold, usually set to 0.5 or a range of 0.5-0.95.

On the other hand, when performing non-maximum suppression (NMS), IoU is used to select the best bounding box among the set of overlapping boxes. NMS helps prevent multiple detections of the same object and keep only the best bounding box. In NMS, a lower IoU threshold than in mAP is usually used, typically around 0.4-0.5.

I hope this clears up your question. Let me know if you have any further doubts.

Best, Glenn Jocher

[EslamHaourn]

@glenn-jocher hello, If I set --iou-thres 0.7, then what does the mAP@.5:.95 mean? Isn't --iou-thres 0.7 used to determine NMS iou threshold? If not, how to determine NMS iou? Thank you

[glenn-jocher]

@EslamHaourn hello,

Thank you for your question. Yes, setting --iou-thres to 0.7 in the val.py script will change the IoU threshold used for the mAP calculation from the default value of 0.5 to 0.7, but it does not affect the IoU threshold used in the NMS algorithm.

By default, the NMS algorithm in YOLOv5 uses an IoU threshold of 0.6 to remove duplicate detections. This value can be adjusted by setting the --conf-thres flag, which is the confidence threshold used to filter out low-confidence detections, and the --iou-thres flag, which is the IoU threshold used for NMS.

So if you want to change the IoU threshold used for NMS, you would set the --iou-thres flag to the desired value. For example, to change the NMS IoU threshold to 0.5, you would run the detect.py script with the following command:

python detect.py --weights yolov5s.pt --img 640 --conf 0.25 --iou-thres 0.5

I hope this clears up your question. Let me know if you have any further doubts or questions.

Best regards.

[EslamHaourn]

@glenn-jocher Thank you for your clarification and patience. I would like to verify that I got it. For val.py the --iou-threshold used to adjust iou for mAP and the new computed value appear at mAP@.5 column, but --conf-threshold used to adjust both confidence and iou (must have the same value) for NMS. But in detect.py --iou-threshold and --conf-threshold used to adjust both iou and confidence for NMS. Another question: Is the dropout a mask layer? What is the difference between dropout and different model sizes? Are model sizes differs in number of layer or connections between the same number of layers or both? Thanks in advance.

[glenn-jocher]

Hello,

You are correct that in val.py, setting the --iou-thres flag will change the IoU threshold used for the mAP calculation. The mAP@.5 value in the resulting metrics table will be computed based on this IoU threshold. However, in detect.py, the --iou-thres and --conf-thres flags are used to adjust both the IoU and confidence thresholds for the NMS algorithm.

Regarding your question about dropout, it is a regularization technique commonly used in deep learning to prevent overfitting. Dropout randomly selects some neurons in a neural network and sets their output to zero during training. This prevents those neurons from overly relying on certain inputs and encourages the network to learn more robust and generalizable features.

The difference between YOLOv5 model sizes (such as yolov5s, yolov5m, yolov5l, and yolov5x) largely lies in the number of convolutional layers and the number of filters used in those layers. Generally, larger models have more layers and more filters, and therefore have the potential to learn more complex features and achieve higher accuracy. However, they also require more computing resources, take longer to train, and are usually slower during inference.

I hope this answers your questions. Let me know if you have any further doubts or questions.

Kind regards.

[EslamHaourn]

@glenn-jocher Hello, Thanks for reply. You said the --iou-threshold flag in val.py that is by default set to 0.6, is used for NMS IoU threshold. And you also said that the same flag calculate mAP for other IoU values and doesn't affect NMS IoU threshold. Second, I would like to know about dropout layers: size on disk, location at structure, number of dropout layers? Third: differences between YOLOv5 and YOLOv4? Thank you.

[glenn-jocher]

@EslamHaourn hello,

To clarify, the --iou-thres flag in val.py is used to adjust the IoU threshold used to calculate mAP at the desired IoU threshold. It does not affect the NMS IoU threshold. The NMS IoU threshold can be set using the --conf-thres and --iou-thres flags in both val.py and detect.py scripts.

Regarding your question about dropout layers, each dropout layer only uses a small amount of memory because its purpose is to randomly zero out input activations. The size on disk will depend on the number of neurons being dropped out, but it should be relatively small compared to the size of the entire model. The location of dropout layers in the network structure can vary depending on the specific architecture being used, and the number of dropout layers used can also vary depending on the needs of the model.

Finally, the main differences between YOLOv5 and YOLOv4 are improvements in accuracy and speed. YOLOv5 is built on top of a smaller, more efficient architecture compared to YOLOv4, which allows for faster inference times while still achieving state-of-the-art accuracy. Additionally, YOLOv5 introduces new training techniques such as AutoML-based architecture search and hyperparameter optimization, which can help to further improve the model's accuracy and efficiency.

I hope this helps. Let me know if you have any further questions.

Kind regards.

[EslamHaourn]

@glenn-jocher Hello, Thank you for your reply. I would like to get results on test set, so I only mean val.py. When I set the two flags --iou-thres and --conf-thresh, then the --iou-thresh belong to mAP or NMS? Do you mean that I can use just --iou-thres flag to set iou mAP? If that so, What does the --conf-thres flag do? And if I set both flags for iouNMS, then how I set iou for mAP? Second, What does the following line: x[i] = self.mi do in forward method in Detect class in yolo.py in terms of number of dropout layers, their sizes, location in structure. Third, Could you refer to structures for both YOLOv5 and YOLOv4 with differences? Thank you.

[glenn-jocher]

@EslamHaourn hello,

Thank you for your follow-up questions. To clarify, when using val.py, the --iou-thres flag controls the IoU threshold used for mAP calculation, and the --conf-thres flag controls the confidence threshold used for both NMS and mAP computation. When the --iou-thres flag is set to a value greater than 0.5, it changes the IoU threshold used for mAP calculation, and does not affect the IoU threshold used in NMS.

Regarding your question about the forward method in the Detect class in yolo.py, the line you mentioned applies dropout to the output tensor of each convolutional block in the YOLOv5 network. The number of dropout layers and their sizes depend on the specific architecture being used, and their location in the network structure is interspersed throughout the convolutional blocks.

Finally, regarding your question about YOLOv5 and YOLOv4, both models use similar object detection techniques based on the YOLO (You Only Look Once) algorithm. However, YOLOv5 introduces a more efficient architecture compared to YOLOv4, which allows for faster inference times while still achieving state-of-the-art accuracy. Additionally, YOLOv5 introduces new training techniques such as AutoML-based architecture search and hyperparameter optimization, which can help to further improve the model's accuracy and efficiency.

I hope this answers your questions. Let me know if you have any additional questions or doubts.

Best regards.

[EslamHaourn]

@glenn-jocher Hello, Thak you for your reply. If we set --iou-thres less than 0.5, it controls the iou used for both NMS and mAP? Thank you.

[glenn-jocher]

@EslamHaourn hello,

Thank you for your question. When using the val.py script in the YOLOv5 repository, the default IoU threshold used for both NMS and mAP computation is 0.5. If you set --iou-thres to a value less than 0.5, this value will be used for both NMS and mAP computation.

For example, if you run the following command:

python val.py --data coco.yaml --weights yolov5s.pt --iou-thres 0.4

The val.py script will use an IoU threshold of 0.4 for both NMS and mAP calculation.

I hope this answers your question. Let me know if you have any further doubts or questions.

Kind regards.

[EslamHaourn]

@glenn-jocher Hello, Thank you for your reply. I think the default value is 0.6. But if I set it to a value more than the default value, then what is the iou for both mAP and NMS? Thank you.

[glenn-jocher]

@EslamHaourn hello,

Thank you for your question. You are correct that the default IoU threshold used for both NMS and mAP calculation in the val.py script is 0.6. If you set --iou-thres to a value greater than 0.6, this value will be used for mAP calculation, while the NMS IoU threshold remains unchanged at 0.6.

For example, if you run the following command:

python val.py --data coco.yaml --weights yolov5s.pt --iou-thres 0.7

The val.py script will use an IoU threshold of 0.7 for mAP calculation, while the NMS IoU threshold remains at 0.6.

I hope this answers your question. Please let me know if you have any further doubts or questions.

Kind regards.

[EslamHaourn]

@glenn-jocher Hello, I set the iou threshold to 0.95 and get mAP@0.95 results(0.839) higher than mAP@0.5:0.95 results(0.695)? And, How could changing IoU threshold change mAP@0.95 result? Second, Could I test my model by setting the task flag in val.py to test using the val set that is used to validate after every epoch in training process? If not, how could I test my model? Third, What does the task flag mean in every case? Fourth, could I use train set in validating in training process? Lastly, Does input size affect the model speed? Thank you.

[glenn-jocher]

@EslamHaourn hello,

Regarding your first question, changing the IoU threshold in the val.py script affects the mAP scores at the specific IoU threshold you set. For example, if you set the --iou-thres flag to 0.95, the val.py script will calculate the mAP score at an IoU threshold of 0.95, and return the mAP value for that specific threshold. It's possible that the mAP score at 0.95 is different than the mAP score at 0.5:0.95, because they are computed using different IoU thresholds. Higher IoU thresholds can be more strict, meaning there are fewer true positives, but also fewer false positives, resulting in a different mAP score.

Regarding your second question, the task flag in the val.py script is currently not a valid option. To test your model, you can run the detect.py script using the --source flag to specify the path to your testing images or videos. Alternatively, if you have annotations for your testing images, you can create a new YAML file and specify the path to your testing images and annotations, then use that YAML file with the val.py script as you did for your validation set.

Regarding the third question, the --task flag in the val.py script was an experimental feature that was removed in recent versions. It's no longer a valid option.

Regarding your fourth question, in general, it's not recommended to use your training set to validate your model, because it can result in overfitting - the model may perform well for the training data, but poorly in real-world data. It's better to use a separate validation set that the model has not been exposed to during training.

Regarding your last question, changing the input size can affect the model speed - larger input sizes will take longer to process, while smaller input sizes will be faster. However, the impact on performance may be more prominent for some models than others. Larger models may benefit more from larger input sizes, while smaller models may not see as much benefit.

I hope this answers your questions. Let me know if you have any additional doubts or questions.

Kind regards.

[EslamHaourn]

@glenn-jocher Hello, Thank you for your reply. If higher threshols are more strict, then mAP@0.95 should give lower results than mAP@0.5:0.95 range that contain lower thresholds. Second, when I test using different iou thresholds, they affect the mAP@0.5:0.95 range column in resuls matrix. You mentioned that it only affects the mAP@0.5 column. Third, why larger input sizes take longer to process, while smaller input sizes will be faster. And how larger models benefit from them. How does number of objects affect the speed? Fourth, I used others anchor boxes and get higher results than my kmean anchors. Lastly, when I compare my results with others, the input size give advantage to whom, assuming we got the same mAP? Thank you.

[glenn-jocher]

Hello,

I apologize for any confusion in my previous response. You are correct that when you set a higher IoU threshold, mAP @EslamHaourn.95 should be lower than mAP@0.5:0.95, as it requires stricter conditions to consider a detection to be a true positive. Thank you for bringing this to my attention.

Regarding your second question, I apologize for the confusion in my earlier explanation. When you set a different IoU threshold using the --iou-thres flag in val.py, it will affect the mAP scores at specific IoU threshold(s) you set, including the mAP@0.5:0.95 range column in the results matrix.

Regarding your third question, larger input sizes take longer to process because they have more pixels that need to be processed. Larger models may benefit from larger input sizes because they may contain more parameters, and larger input sizes can provide more information to the model. However, the impact on speed and performance may depend on the specific model and input size being used. The number of objects in an image may also affect the speed, as there are more regions of interest to process.

Regarding your fourth question, it's not uncommon to obtain different results when using different anchor boxes. The choice of anchor boxes can have a significant impact on the accuracy of your model, depending on the size and shape of the objects you are detecting.

Finally, regarding your last question, the input size can give an advantage to a model, as larger input sizes can provide more information to the model and potentially improve its accuracy. However, the impact may depend on the specific model and the size and type of objects being detected. Comparing results between different models can be difficult, as there are many factors that can affect performance.

I hope this answers your questions. Let me know if you have any further questions or doubts.

Kind regards.

[EslamHaourn]

@glenn-jocher Hello, Does setting iou threshold flag to 0.7change the mAP@0.5:0.95 range to mAP@0.7:0.95. If that is the case, then so why I got higher result for mAP@0.95 than the range. Why iou threshold not only change mAP@0.5 in results matrix but also change mAP@0.95? Second, the training process give one matrix results for trainset and valset, how could I got two matrix results for both? Third, if I set iou to 0.6 when validating train set and set iou to 0.7 at test time, then what should I said about the used iou thresh 0.6 or 0.7. Shoud I use one iou thresh for validating and testing? What if I train without determining model.yaml, which model it will train s,m,x, or xl? Lastly, how could I train my model without internet connection? Thank you.

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[arkerman]

@glenn-jocher @EslamHaourn Hey, guys! Sorry to bother your talk. I have another question about mAP(mask). How to calculate it in segment task? Any suggestion will be appreciated.

[glenn-jocher]

@arkerman

No problem at all! Calculating mAP (mean Average Precision) in the context of a segmentation task is a bit different compared to object detection. In segmentation tasks, you typically evaluate the Intersection over Union (IoU) between the predicted mask and the ground truth mask.

To calculate the mAP(mask), you can follow these general steps:

1. Compute the IoU between each predicted mask and its corresponding ground truth mask.
2. Sort the predicted masks in descending order based on their confidence scores (if available).
3. Calculate precision and recall at different IoU thresholds (e.g., 0.5, 0.6, 0.7, etc.) by considering predicted masks as true positives if their IoU exceeds the threshold.
4. Compute average precision (AP) for each class, which is the area under the precision-recall curve.
5. Finally, calculate the mAP(mask) by averaging the AP values across all classes.

Keep in mind that the implementation specifics may depend on the framework or tools you are using for evaluation. It's always a good idea to consult the documentation or resources specific to your segmentation task.

I hope this helps! Let me know if you have any further questions or need additional clarification.

[arkerman]

@glenn-jocher Thanks for your reply! I really appreciate! Based on your suggestions, i found that the detection's mAP and segmentation's mAP is only different when calculating iou. And i collect some codes from yolov5 7.0:

def mask_iou(mask1, mask2, eps=1e-7):
    """
    mask1: [N, n] m1 means number of predicted objects
    mask2: [M, n] m2 means number of gt objects
    Note: n means image_w x image_h

    return: masks iou, [N, M]
    """
    intersection = torch.matmul(mask1, mask2.t()).clamp(0)
    union = (mask1.sum(1)[:, None] + mask2.sum(1)[None]) - intersection  # (area1 + area2) - intersection
    return intersection / (union + eps)

def box_iou(box1, box2, eps=1e-7):
    # https://github.com/pytorch/vision/blob/master/torchvision/ops/boxes.py
    """
    Return intersection-over-union (Jaccard index) of boxes.
    Both sets of boxes are expected to be in (x1, y1, x2, y2) format.
    Arguments:
        box1 (Tensor[N, 4])
        box2 (Tensor[M, 4])
    Returns:
        iou (Tensor[N, M]): the NxM matrix containing the pairwise
            IoU values for every element in boxes1 and boxes2
    """

    # inter(N,M) = (rb(N,M,2) - lt(N,M,2)).clamp(0).prod(2)
    (a1, a2), (b1, b2) = box1.unsqueeze(1).chunk(2, 2), box2.unsqueeze(0).chunk(2, 2)
    inter = (torch.min(a2, b2) - torch.max(a1, b1)).clamp(0).prod(2)

    # IoU = inter / (area1 + area2 - inter)
    return inter / ((a2 - a1).prod(2) + (b2 - b1).prod(2) - inter + eps)

So, can I assume that when calculating mAP (mask), I only need to replace the calculation part of iou?

[glenn-jocher]

@arkerman absolutely! Based on the code snippets you shared from yolov5 7.0, it seems that the only difference when calculating mAP (mask) is the calculation of the IoU.

In the function mask_iou, it computes the IoU between the predicted masks and the ground truth masks. This is different from the traditional IoU calculation used in object detection, where bounding boxes are compared.

Similarly, in the function box_iou, it calculates the IoU between bounding boxes, which may not be directly applicable to the segmentation task.

To compute mAP (mask), you can replace the calculation of IoU in the relevant part of your code with the appropriate mask IoU calculation.

By making this adjustment, you should be able to accurately calculate mAP (mask) for your segmentation task.

I hope this clarifies your question. Let me know if you have any further inquiries or need additional assistance.

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐