How to correctly evaluate the custom model?

[myasser63]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I trained yolov5s on custom dataset I have few questions to clarify my understanding.

1-mAP metrics during traing is for the training data or the validation data?

2-The summary results at the end is on the training or validation data and is it the best.py or last.py?

`200 epochs completed in 1.077 hours. Optimizer stripped from runs/train/yolov5s_test2/weights/last.pt, 14.5MB

Validating runs/train/yolov5s_test2/weights/best.pt... Optimizer stripped from runs/train/yolov5s_test2/weights/best.pt, 14.5MB Fusing layers... Model Summary: 213 layers, 7020913 parameters, 0 gradients, 15.8 GFLOPs Class Images Labels P R mAP@.5 mAP@.5:.95: 100% 3/3 [00:04<00:00, 1.61s/it] all 70 70 0.873 0.884 0.952 0.815`

3- Run summary from wandb is is on the training or validation data and is it the best.py or last.py?

'ex.wandb: Run summary: wandb: metrics/mAP_0.5 0.93316 wandb: metrics/mAP_0.5:0.95 0.80008 wandb: metrics/precision 0.84023 wandb: metrics/recall 0.86327 wandb: train/box_loss 0.00939 wandb: train/cls_loss 0.00543 wandb: train/obj_loss 0.00761 wandb: val/box_loss 0.00408 wandb: val/cls_loss 0.00668 wandb: val/obj_loss 0.0025 wandb: x/lr0 0.001 wandb: x/lr1 0.001 wandb: x/lr2 0.001`

3- the function of Val.py

I appreciate any assistance with these questions.

Additional

No response

[github-actions[bot]]

👋 Hello @myasser63, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python>=3.6.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

$ git clone https://github.com/ultralytics/yolov5
$ cd yolov5
$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[myasser63]

@glenn-jocher Appreciate if you could assist me

[glenn-jocher]

@myasser63 mAP is computed on validation set, metrics after training run on best.pt

[myasser63]

Thanks for your reply.

May I know the function of Val.py

[glenn-jocher]

@myasser63 val runs automatically during training after each epoch to compute your validation mAP. You can also run it directly to evaluate your trained models, i.e.:

python val.py --weights yolov5s.pt --data coco128.yaml --img 640

See our Colab Notebook Val section for details: https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb?hl=en

[myasser63]

@glenn-jocher val.py could be used on test data instead of detect.py to get the testing results metrics?

[glenn-jocher]

@myasser63 yes with python val.py --task test

[fatbringer]

i get these numbers using the val.py codes for yolov5 Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|████ all 3396 281595 0.803 0.585 0.691 0.349

is there any way i can also make it display MAE MSE ?

[glenn-jocher]

@fatbringer Currently, YOLOv5 does not compute MAE (Mean Absolute Error) or MSE (Mean Squared Error) as these are regression metrics, and YOLOv5 is primarily focused on object detection metrics like Precision, Recall, and mAP. If you need to calculate MAE or MSE, you might need to modify the code or process the output detections manually.

[fatbringer]

@glenn-jocher ah ok I understand. Thanks! I've been looking at YoloV5 and YoloV8 for awhile now. I tried training both on the same custom dataset (a combination of VisDrone and MOT Det).

Is it expected for YoloV8 to perform better in terms of mAP50, but worse in terms of mAP50-95?

[glenn-jocher]

@fatbringer Performance can vary depending on the dataset and training configurations. YOLOv8 may perform better on mAP@.5 due to improved object localization, but not necessarily on mAP@.5:.95 which includes stricter IoU thresholds. It's important to ensure both models are trained under similar conditions for a fair comparison. If you see discrepancies, consider hyperparameter tuning, dataset balance, and augmentation strategies. 👍

[fatbringer]

@glenn-jocher thanks for answering my queries! I was working with the understanding that mAP@50-95 was a stricter and thus better measurement of object localisation because a better score on it means more accurate location of bounding boxes.

How might I understand these 2 metrics better?

[glenn-jocher]

@fatbringer You're welcome! mAP@.5 measures mean Average Precision at an IoU threshold of 0.5, which is less strict and focuses more on whether the object is generally located correctly. mAP@.5:.95 averages mAP calculated at IoU thresholds from 0.5 to 0.95 (in steps of 0.05), which indeed provides a more comprehensive assessment of localization accuracy across various levels of strictness. A higher mAP@.5:.95 indicates better precision in bounding box placement across these thresholds. It's a more holistic metric for evaluating performance, especially for applications requiring precise localization. 🎯