how to find why mAP suddenly increased

[MiNaMisan]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I trained YOLOv5s for 500 epochs, and around the 385th to 387th epochs, there was a sudden increase in mAP, resulting in the best result at about 80%. After this peak, the mAP gradually decreased.

I've repeated this training several times to see if this sudden increase would appear again, but it didn't. The best results after these subsequent trainings, without the sudden increase, decreased from 80% to 70%.

My questions are:

How can this phenomenon be explained? How can I identify the specific reason for this sudden increase in mAP? I suspect that an inappropriate learning rate might have caused this issue. Should I adjust the learning rate or other hyperparameters? Attached are images showing the mAP increase during the initial training (around the 385th to 387th epochs) and subsequent trainings where the sudden increase did not appear.

👆 Sudden increase at about the 385th to 387th epochs in the initial training

👆 No sudden increase in subsequent trainings with the same dataset and parameters

👆batch size and epoch was change from 16 to 96 , 500 to 1000 respectively, but the same

Additional

No response

[github-actions[bot]]

👋 Hello @MiNaMisan, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

@MiNaMisan hi there! 👋

Thank you for providing detailed information and visuals. The sudden increase in mAP you observed could be due to several factors:

1. Randomness in Training: Neural network training involves stochastic processes, and sometimes, random initialization or data shuffling can lead to unexpected spikes in performance.

2. Learning Rate and Hyperparameters: As you suspected, the learning rate or other hyperparameters might have played a role. You can experiment with different learning rates or use a learning rate scheduler to see if it stabilizes the training process.

3. Data Augmentation: Variations in data augmentation can sometimes lead to sudden improvements. Ensure that your augmentation settings are consistent across all training runs.

4. Batch Size: Changing the batch size can affect the training dynamics. Larger batch sizes generally provide more stable gradients but may require adjustments in learning rate.

To identify the specific reason, you can:

• Review Training Logs: Check for any anomalies or patterns in the training logs around the epochs where the spike occurred.
• Hyperparameter Tuning: Use tools like Hyperparameter Evolution to optimize your hyperparameters systematically.

For more detailed guidance on achieving the best training results, you can refer to our Tips for Best Training Results page.

Good luck with your training! 🍀 If you have any further questions, feel free to ask.

[MiNaMisan]

@glenn-jocher thanks for replying! I'm trying to solve this problem now by adjusting hyperparameters now.

please allow ask one more question

Review Training Logs: Check for any anomalies or patterns in the training logs around the epochs where the spike occurred. I've used TensorBoard to check for any suspicious parameters during training, but there are only limited parameters I can monitor: train obj/box/cls loss, val obj/box/cls loss, metrics of precision, recall, and mAP0.5, as well as lr0, lr1, and lr2.

If I want to see more parameters, such as total loss or variable updates for each epoch, is this possible? Also, are there any useful tools that can help me specify these parameters?

Thank a lot

[glenn-jocher]

Hi @MiNaMisan,

Great to hear you're diving into hyperparameter adjustments!

Regarding your question about monitoring more parameters in TensorBoard, YOLOv5's integration with TensorBoard primarily focuses on the key metrics you've listed. If you're looking to track additional details like total loss or specific variable updates per epoch, you might need to modify the logging code in the training script.

For a more customized tracking, you might consider using other tools like Weights & Biases, which integrates well with YOLOv5 and allows for more extensive monitoring and visualization options. You can enable Weights & Biases in your training by setting --wandb in your training command, and it will automatically log more metrics and provide a richer interface for analysis.

Hope this helps, and keep up the great work!

[MiNaMisan]

@glenn-jocher Thanks for your advice! Sorry for the delayed response. It appears that adjusting the learning rate has been helpful. I will continue working to resolve this issue, but it may take some time.

[glenn-jocher]

Hi @MiNaMisan,

No worries about the delay! I'm glad to hear that adjusting the learning rate has been helpful. Fine-tuning hyperparameters can indeed be a bit of a journey, but it sounds like you're on the right track.

If you encounter any further issues or need more detailed insights, feel free to share a minimum reproducible code example. This will help us better understand the context and provide more targeted assistance. You can find guidelines for creating a reproducible example here.

Additionally, make sure you're using the latest versions of torch and YOLOv5 from our GitHub repository. Sometimes, updates include important fixes and improvements that could resolve unexpected behaviors.

Keep experimenting, and don't hesitate to reach out if you need more help. The YOLO community and the Ultralytics team are here to support you! 🚀

Best of luck with your training! 🍀